Towards the holy grail of integration

Where the machine does most of the interface and field mapping work. SOA promises to help with this, but it depends on the old chestnut of trying to "agree on a schema" which is a barrier to flexibility of semantic expression and denies potential interaction that keeps humans in the loop. Think of how angry forms make most people feel. I am increasingly beginning to believe that true automated integration is an AI problem and probably its killer app. A job for Google. They may even call it something like Ploogle-and-play.

More on COTS vs custom software

Lausanne Flon Copyright 2002-2008 A. Barake

It has been my experience that integration and associated maintenance of commercial off-the-shelf software (COTS) is an expensive and difficult process due to a few factors:

• The "normal" upgrade cycle for vendors is often strategic and is driven by internal rather than external compatibility – upgrades also tend to break interfaces regardless of intent, it just does not help that the intent to play well with others is often at odds with large company policies (i.e. marketing strategy), especially when their suite of products covers many functional domains
• Integration requires glue, and glue can either be COTS or home grown. For COTS, see above objection, and add cost. Home grown defeats the argument of using COTS in the first place Q.E.D.

Now, if we look at home-grown software, we have a few things to consider:

• Maintenance is dependent on knowledge, but unless you are writing in APL  (and even then), the pool of developers with expertise usually exists; reading code and fixing it is what they are hired for and have some training in.
• Developers in my experience much prefer to work on code where the source is available than integrate stuff using configuration and proprietary tools where the options are limited and the dependency on documentation and vendor support is limiting
• Code is now pretty much commoditized except in specialized areas (telephony switching, military stuff, embedded stuff) and even then… there are only a handful of approaches that people use to write and maintain code and most good developers can immerse themselves in a code base within days of weeks and make it their own.
• Renewal is key, and owning a system and renewing it by adding features is easier with home grown that has few integration points with COTS than with disparate COTS on different version cycles with glue in between.

The standards efforts are a sort of socialist (Stalinist really) approach to trying to control the ecological process described above, but is thwarted by vendor power struggles and the usual embrace and extend strategies, coupled with purchasing power in the hands of non-detail oriented people (i.e. not tecchies).

UNIX and other open system approaches have and are trying to tackle the problem through factoring out of common functional requirements into modular technical modules and protocols, and have had some success in advancing the state of the art – the Internet and its associated protocols and services are a direct result. The OSI stack is a manifestation of this approach and has become a pattern of understanding outside that world.

One approach is to layer the commercial stuff and isolate it from other commercial stuff through standard gateway mechanisms, off the shelf if you like, and these include queues, UNIX gateways and subsystems that use HTTP, SMTP and LDAP, xDBC as well as XML protocols where practical and where translation mechanisms are easy to obtain (message brokers, AJAX, browsers, SAML…) At another level we should be writing our own application code to use these underlying systems and the glue should be at the application level – i.e. SOA – services are COTS with standardized interfaces, applications are not.

Les mots

A Baja beach, bay side, Copyright 1988-2008 A. Barake

I was asked by my son about the meaning of the word "load". The context was "load the car". As I explained, by giving an example ("load the bags in the car"), I realized that the example provided an instance that could be generalized, but that was inherently ambiguous outside the continuity of experience. One has to have loaded things to understand that bags are just things, and we can extend the concept, and then we can even go further and talk about "a load" as a thing that is "loaded", without ending up in an infinite recursion.

You know where this is going...

Machines need to have exact mappings of symbols to "actions". Actions are just other symbol manipulations. So we have mappings and more mappings and rules and context and all the fodder of Minsky-type AI. Not good enough it seems. Gelernter and others have realized that one must be embedded in experience to have "knowledge". Husserl and Heidegger said it much earlier, but disciplines rarely cross. So we are realizing now that cognition is a sort of misnomer, we need a word to talk about "knowing", or better still "questioning" to get to a model of reality that is useful.

All this can lead to better and more flexible interface design for a start. Imagine a handshake that allows systems to agree on field semantics and field syntax for data exchange without all that WSDL baggage. Give me a couple of ports and go for it. Virus and spyware writers are now writing the primordial soup that will lead to these higher organisms one day.

More on automated interfaces

• UNIX philosophy of universal read, write, create (sic), seek verbs is akin to financial transactions, where bytes are analogous to money
• HTTP extended this with POST and made it stateless - so it becomes a client-server protocol (a good alternative to X-Windows by the way)
• The universal currency is data, and it can contain anything, it can buy anything. Putting meaning to it is really the difficult part, and the notion of objects is really an attempt at creating a philosophy of data
• Right now we have an inflation of data, too much currency, and its value is being balkanized, some currencies are more precious than others, security related ones, and maybe some video or music bytes. Copyright is an attempt at increasing the value of bytes through currency control.
• So if an object defines the methods we can use to manipulate it, then one way to have objects communicate more easily (ideally automatically) is to constrain what objects can do to a machine manageable set. This may not be much of a constraint, only one that promotes efficiency through consistency.
• Or we can take the idea behind types to heart and have a huge catalog of types and ensure that they can play together - real types which map to useful objects
• For example, a user at any terminal could drop select from a list of things like ADDRESS CHANGE, VALIDATE ID, SUBMIT CLAIM, BUY, COMPLAIN, CHANGE FIELD, etc
• This is the "naked object" philosophy taken one step further. No need for a graphical interface, just a nice "no error" interface - where all choices that work are shown. The "trie" of possibilities would reduce itself as transactions are chosen to interact, limiting the choices the more you decide
• In short, modeling. Can this work in an ad-hoc world? Maybe the modeling should be automated through affinity and tuple-space approaches to ad-hoc property lists
• This, coupled with a hierarchy of state machines (like game play AI) would go a long way to creating automated interface coupling.

That SOA thing

Here is the definition from an authoritative vendor-neutral body:

The World Wide Web Consortium’s glossary defines Service Oriented Architecture (SOA) as

"A set of components which can be invoked, and whose interface descriptions can be published and discovered"

Note that the definition does not talk about the type of transport, it is all about interface descriptions – Web services are therefore a subset.

As in all such interface standardization efforts discovering the service (semantics) is key.

How to make apps talk to each other meaningfully (semantics and syntax) without a human in the loop is a problem old as operating systems.

This has been and remains the “holy grail” of integration. Getting apps to talk across their boundaries has been a staple of data processing since I can remember.

We have had shared memory, files, networking, RPC, CORBA, and the now the latest idea is to re-use the Web servers connected to apps as Web services by parsing the HTML or XML to understand the context of the data across HTTP [We seem to be adding layer after layer of code to do this for some reason. Maybe with the hope of adding some sort of “intelligence”, maybe not…]. When we only use TCP/IP interfaces instead we generalize and call it SOA. This tells us that TCP/IP is now a commodity layer. It is all about commoditization moving up the stack.

In all cases we need to make the plug for the socket and make sure the voltages are OK; i.e. the syntax and semantics have to work across the boundary.

The Web has given us a new way to think about this.

Links – URI and URL’s

SOA is the idea that an app can “choose, click on, and follow a link” by itself to get a job done – and sometimes fill forms too.

There are 4 major verbs we can apply to URI’s and URL’s: GET (click), PUT (put up a resource), POST (fill-in a form mostly), and DELETE (get rid of a resource), plus a few more: This is HTTP. It is meant to stateless (except for cookies and URI-held sessions ….) because the state is supposed to be held by the user – and when the client becomes an app, that can complicate things greatly – all of a sudden you have servers talking to each other with state.

Still, we have not progressed to semantics at the app layer – this is transport and syntax.

We have always depended on humans (programmers, analysts and users). We have been doing this – imperfectly – first through command line, shell scripts, GUIs, and now Web pages and forms.

When you cut and paste, when you embed a table or a picture into a document and when you save a file to be retrieved by another app you are doing inter process communications. Windows and GUI’s allow us to do this more easily, with visuals– it may be their most obvious value proposition – the basis of the MS empire. The Web is similar, we press on links to hop among and between servers and apps.

Computers, on the other hand need to be rote taught. They can crawl web sites, but they have difficulty making them inter operate without humans writing the glue. Web 2.0 is about doing this more easily – “mashing up” service interfaces using XML and HTTP and the smarts of a browser scripting language – ECMAScript (JavaScript)

SOA ideals say that we can publish somewhat constrained specs that will allow machines to automatically selects forms and links and act on them, even when that they may never have seen before.

The more realistic approach is to say that we will need humans to guide the machines, but that the molding of sockets and plugs will get less onerous – we are attempting to standardize the interface syntax and restrict the semantics to a published, well managed, machine-discoverable set. In the interim, it helps us integrate stuff with less work.