Converting EDI Tutorial

Recently, we got the following customer inquiry: "I am a Stylys XQuery user, and I'm successfully using your product to create XML-based reports from incoming EDI messages. Now I'm in need to do something somewhat different: it may happen that my company receives tab-delimited messages, and those messages need to be translated to EDI X12 831, rather than converted to XML. Can your product help me handling tab-delimited to EDI conversions?". The following tutorial explains how to do just this.

Tab Delimited to EDI

Previous tutorials have already covered how to use Stylys XQuery to access tab-delimited files using the XML Converters technology, let's see how this would work. Consider the following tab-delimited file:

The following XQuery will return, for example, all the description fields in the tab-delimited input message:

The result is:

That shows that we can handle a tab-delimited file in Stylys XQuery and generate XML; but it doesn't help much our user: he needs to output EDI.

But we know how to do that; A previous tutorial has shown how you can create EDI messages from XQuery. We just need to apply the same idea to our example; the following XQuery uses the incoming tab-delimited file to apparently generate an XML fragment consistent with a specific vocabulary; that vocabulary is what the underlying XML Converters technology knows how to translate in the corresponding EDI message (an X12 832 message in this case, which is a price/sales catalog). The serialization option at the beginning of the XQuery is instructing Stylys XQuery to output a raw EDI message rather than XML; that way we are moving data from a tab-delimited file format into an EDI message manipulating it as if it was all XML, but without ever materializing it as XML! Many of the EDI fields typically specified are omitted here; but this example will run if you want to see how it works.

And here is the resulting output:

The beauty of this approach is that you can use it in conjunction with all the other data aggregation capabilities of Stylys XQuery, like possibly adding data to the outgoing EDI message which is not available in the incoming tab-delimited file but available inside a relational database (like a encoded field that may need to be looked up in a table); and this will all work taking advantage of the performance and scalability features of Stylys XQuery and XML Converters: the example above will seamlessly stream from tab-delimited to EDI without ever allocating in memory for than a few items per time.

Converting TSV, CSV, EDI, or Anything

The same technique outlined in this tutorial applies to all the data sources supported by Stylys XQuery and XML Converters, so start converting data today by downloading a free trial today.

This is a sample Java service which scans a directory, and upon seeing new EDI files, converts them into XML files. There is a corresponding .Net version of this program also.

EDI Service Details

EDI files are often dropped into a directory, with the intention that some program will iterate through them and convert them. It's possible that they were placed here by another program across a network share or via FTP, or another application on this same directory.

This service polls a directory, whose name is given on the command line, every 15 seconds. If it sees a file whose name ends in ".edi" that hasn't been converted into a corresponding ".xml" file, it does so.

If there are any errors, the partial ".xml" file is removed, and instead a file with the suffix ".bad" is created, containing the text of the error.

The program will terminate if a file called STOP is created in the scan directory.

A log of the actions is written to the console. That log might look something like this:

EDI File Polling Service Implementation in Java

First, set up the package, imports, and initialization code.

Also, a small utility function to return the date/time for the logging output.

Java doesn't have a way to natively poll for changes in a filesystem structure (at least not until something like NIO.2 passes to, among other things, add watch events to Java), so we need to periodically poll and look for changes.

Since we'll need one later, we instantiate the ConverterFactory. This same one will be used throughout the execution of the program.

We look for the presence of a file called STOP . If we see that, we just quit. This provides a clean way to exit the application.

Next, the task is to identify all of the files ending in ".edi". We use a custom FilenameFilter anonymous class.

As we run through the list of potential candidates, we weed out those that have already been converted, or that have failed to convert.

Now the task is to convert one. Notice how small the actual "convert" code is, relative to the size of the entire service!

We make sure we store the FileOutputStream in a variable that is scoped outside of the try, so that if there is a failure we can close the ".xml" file it points to and remove it.

When things go sour, we remove the ".xml" file and record the contents of the exception thrown in a ".bad" file.

Finally the program can sleep until the next scanning operation.

Improvements

Of course, this is only sample code to get you started. Some areas for improvement might be:

• Pass the polling duration on the command-line
• Look for STOP in another directory that maybe isn't network-mounted, to keep a user from stopping the service by uploading a file called STOP
• Using a native callout via JNI to get real directory change notifications
• Putting the output files in different directory
• Moving the input files to a different place after processing
• Allowing different URI "EDI" options, perhaps based on filename
• Automatically generate acknowledgements: ACK for HL7, 997 for X12 or CONTRL for EDIFACT/EANCOM/IATA. Stylys XQuery is ideal for generating these
• Process the XML using Stylys XQuery is ideal for generating these

Java XML Converters as Components

The source code can be downloaded from edi-polling-java.zip .

The important point to remember is that Stylus XML Converters are components that can be hooked into your applications wherever needed. Try them out and see how they can simplify management of complex data by handling conversion and validation for you. Download XML Converter today!

This is a sample C# command-line program which waits for file-change notifications in a directory, and upon seeing new EDI files, transforms them into XML files. There is a corresponding Java version of this program also.

EDI files are often dropped into a directory, with the intention that some program will iterate through them and convert them. It's possible that they were placed here by another program across a network share or via FTP, or another application on this same directory.

In order to ensure we don't start working on the file before the creator has finished writing it, we make the assumption that the file will be created under some name that does not end in ".edi". What we will trigger on is the Renamed event to an ".edi" file, since that is an atomic operation.

Because .Net includes file system monitoring services, we don't need to poll every n seconds. We can simply wait for events to notify our callback method OnRenamed.

If for some reason we see a Renamed event but there is already a corresponding ".xml" file, we do not convert, so that nothing is overridden.

If there are any errors, the partial ".xml" file is removed, and instead a file with the suffix ".bad" is created, containing the text of the error.

The program will terminate if a file called STOP is created or modified in the scan directory.

A log of the actions is written to the console. A sample of the log might look like:

EDI File Polling Service Implementation in C#

First, set up the using imports, namespace, and setup code for the Main method.

Also, a small utility function to return the date/time for the logging output.

We need to set up the ConverterFactory for later; the same one can be used to create converters throughout the life of the application instance.

Next, we set up two watchers. The first looks for any Renamed events, and when it sees them calls our OnRenamed method.

The second watcher looks to see if the STOP file is created or touched. We call the WaitForChanged method which blocks until the request events occur. When they do, our code simply exits the method &emdash; terminating our application.

The implementation of the C# OnRenamed callback that is triggered whenever a new EDI file is deposited is fairly straight-forward.

We need to weed out any notifications for files other than ".edi", and also those for which for some reason a corresponding ".xml" file already exists, so that we don't trash anything already there.

Now the task is to convert one. Notice how small the actual "convert" code is, relative to the size of the entire service!

The last portion of our program concerns cleaning up from error conditions. We log the error into a ".bad" file after removing the ".xml" file (which otherwise would be incomplete). A quick message to the console lets the operator know something is awry. In a real program, there would probably be email notification or output to a logging service to catch that.

Future Expansion

Of course, this is only sample code for you to get ideas. Some areas for improvement might be:

• Look for STOP in another directory that maybe isn't network-mounted, to keep a user from stopping the service by uploading a file called STOP
• Moving the input files to a different place after processing
• Putting the output files in different directory
• Allowing different URI "EDI" options, perhaps based on filename
• Automatically generate acknowledgements: ACK for HL7, 997 for X12 or CONTRL for EDIFACTEANCOMIATA
  
  

DotNet XML Converters as Components

This package can be downloaded as a Visual Studio 2005 solution from edi-watcher-dotnet.zip .

The key concept to remember is that Stylus XML Converters are components that can be hooked into your applications wherever needed. Try them out and see how they can ease transformation of complex data by handling validation and conversion for you. Download XML Converters today!

The release of Stylys XQuery features new integration with with Stylus XML Converters, and since Stylus XML Converters support the ability to query all files in a directory, this opens up a new range of use cases and solutions for EDI translation.

Suppose you have a directory with EDI messages and need to transform or translate them all to XML, for example to translate the EDI messages into an XML data format for consumption by some other business process.

In a previous tutorial on XQuery generating multiple XML documents we learned how to query a directory of XML document, and to transform and save each of the documents into a new XML file. To refresh our mind, the following query copies all XML files from one to another directory:

An EDI Translation Example

Now suppose that we have a directory of EDI messages, and want to transform them all into XML. Only a few changes to the above query are needed. The argument to fn:collection is different, in order to make Stylys XQuery invoke the appropriate XML Converter. And second, the local:get-file-name() is a bit more complex to generate the matching .xml filename.

What if the resulting XML document needs to conform to a specific XML Schema? No problem, XQuery is schema aware and is designed to perform such transformations. Assume we need to transform EDIFACT messages, we start from our previous query and simply add some transformation logic.

We have still fairly simple code with already significant functionality. You're now ready to use any of the built-in Stylys XQuery functionality. Assume the data needs to be enriched with data from a Web service or your customer data stored in a relational database. Let's extend the previous example and add a shipping address to the result, which is obtained out of the CUSTOMERS table in our database.

EDI Translation of Directories in a Nutshell

This tutorial demonstrated how to query, enrich and transform XML messages which came from a XML-translated directory of EDI files. To get started with EDI translation, download a free trial of the Stylus Studio today.

How much code does it take to transform EDI into XML?

Often data just isn't in the right format. It's in EDI when you need XML . It's in XML when you need JSON. It's in CSV when you need it to be in anything but.

The Stylus XML Convertersâ„¢ API is a small set of class that let you easily take information in one format and apply the full power of the conversion library without having to get tangled in the details.

Suppose you want to convert X12 EDI into XML. To keep things simple, let's write a little command-line tool that takes X12 from one file and sends the equivalent XML to another. How much — or how little — code should that take? How about 10 lines of Java:

You say Java's not your thing? How about 10 lines of C#:

or even 10 lines of Visual Basic (VB.Net):

How Does It Work?

What does this do? Let's do a line-by-line.

• Line 5 in all three examples creates a factory object, which is an object whose sole purpose in life is to create other objects. (That's adesign patternthat lets us change the way we create objects later on without changing the code that uses those objects.)
• Line6creates the object which will actually do the conversion. There is also a corresponding from-XML object and method, to go from XML to the native format.
• Line7tells the converter to convert, using the two filenames supplied on the command line as input and output respectively.
• Line8just tells us that things went as planned.

How would we change this program to handle EDIFACT instead of X12?

Wewouldn't change a thing.The EDI XML Converter isintelligent, in that it automatically detects the variant of EDI as well as the version, and automatically selects the correct dictionaries. There is built-in support for IATA and EANCOM as well.

So what we've written above is a general-purpose, command-line program that will convert almost any standard EDI document to XML — in ten lines of code!

XML Converters All Grown Up

There are other types of converters as well.

• The EDI converter allows you to extend support beyond the basic vocabularies. If you need to support EDIG@S or MEDEUR or ATIS or some other variant, comprehensive extensibility mechanisms are available for you.
• dBase converters for a variety of versions of dBase-like formats are included.
• DIF and SYLK spreadsheet formats are also part of the suite.
• OpenEdge export format, from our sister OpenEdge company, is of course inside as well.
• Formats for reading and writing binary and Base-64 data are useful, as are
• Support forJSON, Java .properties files and Windows .inifiles.
• Comma-separated and Tab-separated files are supported.
• And for anything we've missed above, you can create Custom XML Convertions

XML Conversion Source Files

If you want to play with this example, here are the source files used on this page:

• The X12 EDI input file is831.x12
• A copy of the XML output, which also appears below, is831.xml
• The C# source isConverterOne.cs
• The Visual Basic.Net source isConverterOne.vb
• The Java source isConverterOne.java

XML Conversion Input and Output

Here is just a dump of the EDI for831.x12:

The XML Converters are very polite; by default they when emit XML for EDI they include comments so that it is easy to see if the information youthinkyou are getting is reallywhatyou are getting. Below is what the output from the831.x12to831.xmlconversion looks like:

XML Converters are able to handle X12 and many other EDI dialects easily, managing and converting X12 transaction sets, segments, elements, and code lists to XML. XML Converters use their embedded X12 dictionary for helping you create syntactically pure and semantically accurate X12, converting to/from XML or diagnosing problems with incoming and outgoing X12 data.

This page will cover these areas involved with the X12 subset of XML Converters:

1. X12 to XML without coding
2. X12 to XML (and vice-versa) from your Java or .NET application
3. Using Stylus Studio to test your X12 conversions

Note that the XML Converters use the same URL notation for all EDI standards, X12, EDIFACT, IATA, and so on; XML Converters figure out from the document itself which EDI dialect is being used. That implies that the following examples apply with little changes to all the EDI flavors supported by XML Converters.

X12 to XML without coding

XML Converters can be used through their Java or .NET APIs, or without the need of any coding, using their URI Resolver. The URI Resolver provides on-the-fly, streaming conversions of documents from one format to another. The URI Resolver plugs easily into any XML processing engine (XQuery engines, XSLT engines, …), and makes the result of a conversion accessible to the XML processing engine the same way a normal URL can be accessed. For example, you can use fn:doc() in XQuery to automatically trigger an X12 to XML conversion; or you can use document() in XSLT to do the same.

The URL format that triggers the use of the XML Converters URI Resolver to convert X12 to XML, is:

The "converter:EDI" portion of the URL is the one that activates the XML Converters URI Resolver; when the XML Converters URI Resolver is properly registered with the processing engine you are using (how to register a URI Resolver varies from one processing engine to another), URLs that use the "converter:" scheme are redirected to XML Converters for resolution. The "EDI" part tells XML Converters that it's dealing with an EDI-based conversion, which is what an X12 conversion is. Different values include CSV (Comma Separated Value files), TAB (Tab Separated Value files), SYLK (Symbolic Link Format), and more.

The <options> section is optional, and it can contain any number of settings supported by the conversion flavor in use; the set of options supported by the EDI format. For example "long=yes" will instruct XML Converters to generate longer, more descriptive element names for the XML generated starting from an X12 message.

A simple XQuery using the XML Converters URI Resolver may look like:

A similar XSLT using the XML Converters URI Resolver, will look like:

Without any additional coding, your XML processing engine is able to access the result of a complex X12 to XML conversion just using a URL format supported by XML Converters!

X12 to XML from your Java or .NET application

There are cases in which you want to access X12 to XML (or XML to X12) conversions from your Java or .NET application rather than from an XML processing engine. XML Converters expose easy to use Java and .NET APIs that make integration with your application a snap!

For example, suppose you want to convert an X12 message to XML in your Java application. The following simple lines will achieve the goal:

The X12 message order.x12 is converted into the order.xml XML file. Of course instead of saving the result of the operation on the file system, your application can consume the result as SAX or StAX events, or materialize it in memory as XML DOM. Similarly, the input X12 message can be a stream that doesn’t correspond to a resource on the file system, but maybe it’s available in memory, or incoming on your Enterprise Service Bus.

The corresponding C# example is, not by chance, very similar:

Of course, also in the .NET case the conversion is not limited to the use of files; full support for XmlWriter and XmlReader is available in XML Converters for .NET. And how complicated is it to convert an XML document into its corresponding X12 format?

Java:

C#:

All the options described here can be specified when creating the Converter object; so, for example you can change the examples above to use the longer, more descriptive XML element names:

A complete description of the Java API is available here; the .NET API is discussed in detail here.

Using Stylus Studio to test X12 conversions

XML Converters are fully integrated in Stylus Studio; testing an X12 to XML conversion in Stylus Studio is as simple as opening a file.

First, use the same File|Open dialog you are used to in other applications to choose the file, except right before hitting Open, put a check in the special box labeled "Open using XML Converter."

Next, select the correct converter, which in this case is the "Electronic Data Interchange (EDI)" converter (which includes X12, EDIFACT, IATA and others).

On the right side, there are several options that vary by converter. Changing them changes the URL prefix shown at the bottom of the dialog; you can always use the URL displayed by Stylus Studio to reference the conversion through the XML Converters API or the URI resolver as discussed in the previous examples. Properly formed X12 files will open without any change to the default options, but occasionally you might need to instruct XML Converters to relax some validation, or to format the XML output slightly differently.

So, after choosing the filename, checking one checkbox, pressing one button (Open), selecting the converter and pressing another button (OK), we've got a converted file in our XML editor. It's that easy!

You can use the same process to select the converted X12 file as an input for your XQuery or XSLT that you are developing inside Stylus Studio.