Telescopes and views

Jon Udell's essay "The wrong end of the telescope ?" raises some interesting issues and summarizes some thoughts from several sources.

They all go to illustrate how difficult it is to put a simple application into a browser page. A part of the problem is that the browser/http/web server combination was never designed to support the rich applications we are trying to shoe into it. Therefore the issue is not really different ends of the telescope but rather different telescopes. Programming the browser is very different from programming the server and they do not connect very well.

The problem lies in the marked impedance between the browser model and the server programming model. Server side programming is relatively easy - the tools are there and it is up to the practitioner's to use them correctly.

To match the server side interfaces with the client is hard when you go beyond a simple form. Entering the world of multi-page applications becomes challenging to say the least. We have invented all sorts of interesting work arounds/add-ons - cookies, frames, iframes, DHTML etc to try and make it possible to deliver rich applications. Some people have created neat frameworks to help such as this from cross-browser.com but in the end the complexity remains in connecting the back to the front.

A good indication of the overall complexity is the number of attempts to build easier to use frameworks - everyone has made an attempt at creating either a new scripting language, a new template framework, a new application stack, application builder and now rich clients. The
proliferation of these tools is a symptom of the problem- sophisticated web applications are too hard to build. The reason is, I believe, there is a impedance mismatch between the browser and the server - HTTP is not the issue.

Sometimes it feels like we are all using different telescopes to hunt for the most elusive thing of then all - the easy model of building, debugging and maintaining sophisticated and usable internet applications.

EDI - Simple enough?

A comment by Mark Baker around EDI got me thinking about the current amount of infrastructure we are building on top of the transport. Today millions of business messages are flowing formatted as EDI messages and thousands of business processes are built using the MEP defined by EDI documents.

So what is wrong with just using EDI. Probably a couple of things, at one point the cost of the initial transport hardware and communications infrastructure but this has been solved by the draft AS2 standard which is essentially EDI over http(s) which has reduced the costs dramatically. So much so Walmart has mandated it for all communications.

The amount of work that has gone into the MEP and the definition of the document elements is huge and extremely valuable IP. The only remaining piece is the actual format itself which is very “concise” as it was designed when bandwidth was expensive. The format itself is hard to work with compared with XML due mainly to the lack of tools. If every language had an open source EDI parser (or two) and a transformation tool like XSLT would everyone be using EDI today?

The mechanics of EDI at the MEP level provide a fairly complete set of business interactions but getting into the details of the message and extending the messages is very complex and requires very specific knowledge that is only applicable to EDI - no one has every used EDI as a format for config files or build scripts. XML on the other hand has become the universal data language, because it is so easy to manipulate and mold.

In some ways the discipline that EDI imposed has resulted in a loss of simplicity as instead of a set of well defined MEPs we now have a large number of standards that try and do the same but in fact do not focus on the MEP but rather on making the communication more complex. Where is the WS-850 rather than the generic WS-Metadata. Are we missing the point because it is to easy to avoid it?

Information overload: volume or change

While there has been a dramatic increase in information volume the actual problem for me is the rate of information change. Or maybe I am unable to manage the fundamentals of the information flow, how do pick out the key pieces of information and relate back to my existing knowledge framework.

When the information sources and formats change so rapidly we are unable to categorize them before we move onto the next piece. The whole idea of refereed technical papers was to try and place the information in a broader context and relate it to other similar information. Today's hyperlinks may be technically better but the information schema is not well designed so they do not perform the necessary context creation, or at best it is ad hoc.

Of course Blogs (like this one) add to the problem as we publish more a stream of consciousness (speaking only for myself) rather than a well consider view with information to back it up. Blogs are thoughts in progress and do contain many nuggets but they needed to be forged into coherent thoughts and placed in a larger context.

Is the next great Google type company not something that finds the atoms but something that creates the contextual fabric of knowledge?

Open Source to Open Services

Open source has shown that simplicity can be created in the infrastructure but it still misses its potential as code still has to be designed to be installed and maintained on various platforms that are changing at uneven rates. It should rather be offered as an open service that can be combined with other services to create new services that live in the network. I do not want software I want solutions to problems. (I still like software and looking at code but i really do not need it - figuring out how Google Maps works is fun and interesting but I do not need to know how it works to use it. To integrate with it I should need to know very little but integration is still too hard.

Integration is a problem because most software is not designed to be integrated. Just as till the advent of open source most enterprise software was not designed to be easily maintained, built and deployed. A huge effort went into (and still goes into) designing software to be deployed on a large number of target platforms. This effort provides zero value to the consumer of the solution.

If the same effort by smart people went into creating simple open service interfaces as they have done to create simple maintenance and deployment for open source we would have a very rich ecology of shared services and a very different economic model. Open source is trying (and in many cases succeeding) in moving the economic model to a maintenance model - well take the next logical step and move to a complete utility model that measures economic value by ease of integration and SLA (Service Level Agreement).

The economics are on the side of shared infrastructure and software as a service. Moving to open services would dramatically increase the impact of the open source community and move the value to the SLA of the service not the portability of the service.

More simplicity

After my last post on simplicity I received an email from an old Teknekron/Tibco colleague Steve Wilhem who made a couple of great points that should be shared. I have para-phrased his email so any errors in interpretation are mine.

The first was around refactoring and how to achieve simplicity (specifically in a framework) the need to constantly refactor, do not expect to get it right the first time but keep evolving it. The other was more subtle and I thought most important. Do not try and solve all possible problems - leave the system open enough such that the last ten percent or so can be done by the user of the system. I think this is key as this is typically where we add complexity - trying to solve all the weird edge conditions we can possibly dream up. Let's assume our users are smart and give them the hooks to solve the problem in the way they see fit. Moves any complexity to specific applications rather than the infrastructure where it does not belong. Thanks Steve great suggestions.

Simplicity Rules!

Sam Ruby recently posted a great presentation on amongst other things simplicity. Adam Bosworth posted a long article a while ago in a similar vein. It seems that the importance of simplicity is being recognized.

I am not sure that it is simplicity of implementation as in being not sophisticated that is being suggested, rather it is elegance or simplicity of the solution. The original Mac was a complex piece of engineering as was the original Palm but both had simplicity of use.

Open source drives simplicity by rubbing off parts of a design that have unnecessary complexity. Open source projects that have community value tend to get simpler as in easy to build test and deploy because no one wants to go through complex steps again and again using up their most valuable commodity - personal time. To take an example from Sam's talk - try installing and running WebSphere versus JBoss - last time I tried, I gave up on WebSphere after several hours while I had JBoss downloaded and running in less than 30 minutes. (This is not a recommendation that programming J2EE is a model of simplicity ;-)).

Rapid evolution towards simplicity is usually the result of several smart people driving a solution to a shared problem. Open source by its nature attracts more people and as it is a meritocracy it gets rid of over complex solutions pretty quickly. Having development teams organized in a similar fashion grinds out clean solutions faster too.

Making simplicity a goal in integration is key to success - it can be an aspirational goal i.e. integration may never but point and click but unless the goal is simplicity rather than solving many imaginary problems then the possibility to asymptotically approach it will never happen. So setting simplicity of solution is always the right goal.

Deploying hosted applications - boxes are bad

I have used hosting providers for many years now and before that worked with rooms full of computers to serve various customer applications. This is one of the first things where I saw the huge value in outsourcing and the value that shared infra-structure brought to customers, but I want a lot more than is being provided today.

We have come a long way in managing the infrastructure for hosting applications. However the hosting model is still tied to the model of boxed cpu's. Blades are great and racking and stacking has been a great way to go for creating cheap scalable architectures. We are however a long way from the ideal case. The ideal case is where I agree with the hosting provider the high level architecture and then push software to the specific part of the system architecture and pay for an SLA and usage. Both SLA and usage are complex numbers to work out but the degrees of freedom they introduce should make it easy to develop innovative and fair pricing models.

We are a long way from this as the software tools we use to develop are also locked into the box model. This is not surprising as commercial tools are priced by the box. However open source tools do not have this limitation. Another way to look at this is from a virtual machine - why should the JVM or CLR be tied to a box should it not manage resources for the application and demand more resources as demand increases?

Global computing projects have been leveraging this for years but each application has been custom designed is it not time for this to become a standard part of the network?

Tools and Intermediaries

When I think of intermediaries I typically think of additional tools that I have in my tool box and can use. They are something I can use within my processes but are run and managed by someone else - in other words shared infrastructure.

A simple example of an intermediary would be a data transformation service. This service would accept any message in a specific wrapper (SOAP or REST) and a URL pointing to a definition of the transformation instructions and then return the result. Other services could be shipping calculators, tax calculators etc.

Whether the service returns the message to the sender or forwards it to another destination seems to determine whether the service is defined as an intermediate service or an end point. However the service should not really care, where the message goes next, it should be a capability of the service (does it support forwarding) and a decision made by the application developer - where do I want it to go.

Discovery of these shared services is a challenge today as I typically need a complete ecology working together before they are truly useful. The begining of an ecology does seem to require a common set of semantics and a way of transforming the semantics. This implies a common schema for the ecology and transformation services to move in and out of the common semantics. For the ecology to exist there must be sufficient agreement between all parties of common semantics. A good example of ommon semantics is in the retail industry where ARTS that has defined a set of XML Schemas under the heading of IXRetail. This provides the common semantics that enable parts of the ecology to communicate.

Once the semantics are defined and available to the community shared services can be created but how individual services interact and work together is still an issue as there is no really good way to build collaboration between service providers in the loosely coupled framework that is needed to foster collaboration and independant development.

To make interacting intermediaries a viable source of tools the ability to boot strap an ecology needs to be available - anyone got ideas?