EDI - Simple enough?

A comment by Mark Baker around EDI got me thinking about the current amount of infrastructure we are building on top of the transport. Today millions of business messages are flowing formatted as EDI messages and thousands of business processes are built using the MEP defined by EDI documents.

So what is wrong with just using EDI. Probably a couple of things, at one point the cost of the initial transport hardware and communications infrastructure but this has been solved by the draft AS2 standard which is essentially EDI over http(s) which has reduced the costs dramatically. So much so Walmart has mandated it for all communications.

The amount of work that has gone into the MEP and the definition of the document elements is huge and extremely valuable IP. The only remaining piece is the actual format itself which is very “concise” as it was designed when bandwidth was expensive. The format itself is hard to work with compared with XML due mainly to the lack of tools. If every language had an open source EDI parser (or two) and a transformation tool like XSLT would everyone be using EDI today?

The mechanics of EDI at the MEP level provide a fairly complete set of business interactions but getting into the details of the message and extending the messages is very complex and requires very specific knowledge that is only applicable to EDI - no one has every used EDI as a format for config files or build scripts. XML on the other hand has become the universal data language, because it is so easy to manipulate and mold.

In some ways the discipline that EDI imposed has resulted in a loss of simplicity as instead of a set of well defined MEPs we now have a large number of standards that try and do the same but in fact do not focus on the MEP but rather on making the communication more complex. Where is the WS-850 rather than the generic WS-Metadata. Are we missing the point because it is to easy to avoid it?

Information overload: volume or change

While there has been a dramatic increase in information volume the actual problem for me is the rate of information change. Or maybe I am unable to manage the fundamentals of the information flow, how do pick out the key pieces of information and relate back to my existing knowledge framework.

When the information sources and formats change so rapidly we are unable to categorize them before we move onto the next piece. The whole idea of refereed technical papers was to try and place the information in a broader context and relate it to other similar information. Today's hyperlinks may be technically better but the information schema is not well designed so they do not perform the necessary context creation, or at best it is ad hoc.

Of course Blogs (like this one) add to the problem as we publish more a stream of consciousness (speaking only for myself) rather than a well consider view with information to back it up. Blogs are thoughts in progress and do contain many nuggets but they needed to be forged into coherent thoughts and placed in a larger context.

Is the next great Google type company not something that finds the atoms but something that creates the contextual fabric of knowledge?