A Tale of Two Realities

I have just begun a new job. No hiatus between this new one and the old. One day I am in hell, the next in paradise.

The one I left was as a software manager in a large organization, and the current one is as a software lead in a very small organization.

Other objective differences: the previous job was bureaucratic, there were many processes and constraints and a network of communication and decision paths, and the systems my team built had to reflect the complex interrelated and often historically constrained requirements.

The current one is technical, working within a small team and with very precise mostly self-defined scientific requirements, adjusted to meet specific customer requests. The software has to run precision equipment to capture and calculate results within extremely tight tolerances, both in time and numerical accuracy. The only measure of quality is repeatable precision. The culture is one of technology and science.

In the new job, I report to people who have the same engineering training as I do, who have more experience in the R&D business than I do, who I can ask questions of without risk of offending or destabilizing them emotionally. Peers and superiors in all senses.

In my previous job, I reported to people who asked me to explain what I did "in simple terms" and who liked to hear themselves talk so that they could pretend to make sense of their lumbering thoughts in public forums, often referred to as "meetings" but which felt like beatings. Dissent could be career limiting.

The analogy I would like to give to dispel any conclusion that I may be a pretentious ass is as follows:

Suppose you are a medical practitioner, a physician, and you see patients in a clinical context. You have to diagnose and prescribe treatment as part of your day to day routine.

Also, suppose that you are "managed" by someone who had not undergone the same level of training as you, say a registered nurse, who knew the lingo, understood the context, but did not feel confident enough to make life or death decisions affecting patient health, but felt competent enough or was somehow appointed to tell you, the physician, how to run your practice.

For example, "please use language that is easier to understand in your charts". Or  "make sure your handwriting is legible for the pharmacist", or " you have not seen your quota of patients today, why did you spend 6 minutes more than average with Mrs. Smith today?".

Could you practice under such circumstances? How long would you last?

I lasted 5 years in the analogous IT context. My manager was a technologist, previously a school teacher who joined th IT boom of the nineties and hung on. She did a bit of COBOL programming, hated it and went into management because she could talk better than she could listen or understand.

She is proud to claim that she does not understand the need for system architecture versus program design, her eyes glaze over when there is talk of latency, language choice, servers, data flow analysis, code profiling, and factoring and she had no patience for options analysis or proof-of-concepts. She will not deign to read code. She has trouble with email client software and spreadsheets.

She wants simple, clear explanations and plans, and wants to set deadlines before beginning design, because iterative work can only lead to grief. She always used waterfall approaches in COBOL, and they worked fine for her.

Her weakness and fear prevent her from adding value, from making decisions. She was a pass-through for her manager's decisions to me and my peers. No value added, but lots of aggravation and delay. Never understood or wanted the concept of situational management, which is probably the most effective way of managing people, so effective that there have been lawsuits about who has a right to claim authorship and teach it. She is a scared rabbit, hanging on until she can retire and cash in.

Others in that environment have said that we, the technical folk who did the work, should not "speak Neanderthal" when in the presence of senior executives, by which it was meant that we should use non-IT terms of less than 3 syllables. The executives in question were the CIO and her direct reports.

But all this is now in my past, one-day-old stuff. I am now in another world, where the conversations around me centre on measurement tolerances, ADC resolutions, real-time interrupt-driven code, clock speed and sampling rates, hierarchical state machine approaches, language selection and compiler efficiencies. I have scope probes and micro-controllers on benches beside me, no fuzzy walled grey cubicle partitions in sight, and a project to deliver with people I can talk to who want me to talk dirty.

I am in heaven.

Cognitive Philosophy

There seems to be a movement afoot - long overdue - to embody the foundations of intelligence and philosophy. It is partially old news, the heroic AI of the 50's, 60's and 70's has been discounted despite our benefiting from the results of that research in today's mainstream technology: neural nets, machine speech recognition, handwriting recognition, natural language translation, cars that drive themselves, and all kind of other low-level intelligent (unconscious?) activities simulated by programs.

I think that it is the Minsky flavour of AI (now re branded cognitive science), the one grounded in logic, seems to have fallen out of favour -since the fruits of that work did not rise up to the expectation of the "I" in AI; an AI based only on logic, a formal game with symbols that pretended to be words. How can that be smart without the observer having to make all kinds of allowances and assumptions?

This stuff was subject to sometimes acrimonious debate for a while, until technology and big data made many of the arguments moot.

The good news is that there is now a strong movement towards the recognition of the role of the perceptual environment and its embodiment in animals like us as part of the equation.

This realization may be due to the Web, indirectly, since it should be now be obvious that the seeming "intelligence" of the Google search would not exist without the "environment" of all the Web sites, which are of course interactions with people's brains (in fact there was a very simple AI program that used "Google distance" as a way to measure metaphorical strength between concepts. For example "Castro" and "cigar" scored pretty high on that scale).

So now there is a movement, a possibly serendipitous program of research coming together to ground AI in the real-world of experience, and some of the people that I think are leading this effort align themselves along a somewhat elegant but odd symmetry of ideas.

Let me explain.

There is David Gelernter, the computer scientist and columnist who wrote "The Muse in the Machine" and "Mirror worlds". These books made me want to go back and look at AI again. In fact I ran back. His arguments are so cogent, so seemingly simple and common-sensical that I almost missed their radical novelty. He states that data and the links between data is the key to intelligence. These links are often emotional gradients, in the sense that logic is not always what binds ideas; it is similar emotional affect!

The brain associates memories based on the closeness of the emotions they evoke, the sensory input parallels that occurred as the memory is created. The whole model, not just the idea...how you felt, how cold or hot or tired or dizzy you were. This links data to environment so elegantly that it is a mystery to me why no-one has picked up on this earlier. He breaks the syntax/semantics dichotomy very deeply. Also, Gelernter has very conservative political views - right of centre. I am noting this now so as to make my strange symmetry argument a bit further down. Bear with me.

There is George Lakoff, the cognitive linguist. His work with Johnson on embodied philosophy and with Nunez on the origins of math, takes the empirical evidence around the links between our brain wiring and the perception of the world and uses it as a foundation for philosophy.

Basic brain mechanisms are considered to be foundational and ideas grow as hierarchies of metaphor (i.e. links).

Metaphor becomes a key mechanism for cognition. Similarly to Gelernter,  Lakoff et al propound that metaphors are based on evolutionary and environmental constraints, not some platonic logic ideal. No such thing as the separation of  mind from the world of things. These guys are existentialist scientists. Finally! The rampant positivism and Platonism of the English speaking world, the Whitehead/Russell school, is giving ground. Merleau-Ponty and Husserl are now more visible in the world of scientific cognition.

Lakoff is an ex-student of Chomsky. He seems to disagree with the flavour of Chomsky' thesis of hard-wired syntax in the brain. He believes syntax and semantics to be more of a spectrum of complexity in cognitive biology than a differentiator between humans and other animals. Also he is a non-neo-con, a strong and vocal opponent of  right wing politics as it is played in the US today, and although I hesitate to slot him with Chomsky on the political front, they are on the same side of a hypothetical centre line.

So we have right and a left wing thinkers on cognition who fundamentally disagree in the political realm, the realm of people relations, nations, influence, and power. That is a healthy and good thing. This lends support to the notion that there is a grounding in "objective" reality for this thinking on cognition, it is much more than opinion.  (A slim argument, I know, but hey, this is my blog, I can say what I want.)

Aside from the political thing, I want to continue to talk about is the cognition thing. There is another thinker/scientist that seems to be breaking new ground along those same lines.

He is a neuro-scientist, one who gets his hands dirty opening up craniums, working with brain surgeons. William H. Calvin. I read two books of his: "Conversations with Neil's Brain" and "How Brains Think, Evolving Intelligence Then and Now". His ideas centre around the notion that cognition is an initially subconscious Darwinian process among neural bundles that remember perceptions, integrate them and ultimately model the environment perceived by the body, and that the successful "species" of memories/emotions (let's call them cognitions) are the ones that we call conscious, the ones that rise to the top of the noise of cognition, to differentiate themselves as our inner voices.

His research fundamentally agrees with the models put forth by Gelernter and Lakoff. Like Lakoff, he believes that most cognition in the brain is fomented through unconscious processes. Like Gelernter he finds that memories are linked through non-logical processes. He shares with Lakoff a deep knowledge of Chomsky's ideas on innate language ability and like Lakoff disputes the simplicity of these models, without denying the human brain's ability to parse language. He also talks of metaphor, not surprisingly, since all this is supported by empirical evidence at the chemical and behavioural level.

So I am very happy that this stuff is happening. I tended to get hot under the collar when I read "classic" texts by Minsky or Pinker on intelligence. They seemed simplistic and smug. I was always perplexed by the lack of attention given to Grey Walter's ideas which preceded but anticipated Gelernter's. I wanted 'The Muse in the Machine" to be discussed more, but to my chagrin, Penrose's strange books on quantum brain hypotheses got more press.

Let's get back to empirical research, let's continue the program of work that will ultimately bring philosophy and psychology back into science.The foundational thinkers list above should also include Turing. Even Penrose mentions his "other work". He did mathematical research on molecular biology (plants mostly) and how stem cells can become complex structures. His mathematically-based hypotheses on these matters were validated experimentally just this year in the UK. Ultimately this theoretical work may prove to explain some of the brain mechanisms underlying the work Calvin and others are doing.

There are conceptual layers to all this analysis, from the bottom-up: the molecular, the cellular, the systems (bundles of neurons evolving), and the cognitive and behavioral, leading to metaphor, logic and science, including physics, on to philosophy, morality  and politics. The gaps needs to be filled so that we have a continuum of understanding. A New Science in the Vico sense. Something we made and understood that we made, but that is grounded in experience.

(I just found an earlier post that was wishing this stuff would happen, and so here we are.)

Towards the holy grail of integration

Where the machine does most of the interface and field mapping work. SOA promises to help with this, but it depends on the old chestnut of trying to "agree on a schema" which is a barrier to flexibility of semantic expression and denies potential interaction that keeps humans in the loop. Think of how angry forms make most people feel. I am increasingly beginning to believe that true automated integration is an AI problem and probably its killer app. A job for Google. They may even call it something like Ploogle-and-play.

Software engineering and hazing

Maybe the reason engineering faculties used to have such painful initiation rituals was that they (unconsciously I'm sure) wanted to drum into you that there was a culture here, you had to adhere to the body of knowledge that existed, that was developed, and that you could not "come in arrogant". This is different from computer science, where everyone is de facto arrogant and knows everything. And when you get old and wizened in the IT world, you try to maintain your influence by going into management, and the continual flow of new technologies keeps coming, almost as if to prevent experience with the details from continuing to grow, to maintain the arrogance of the new, a bubble effect really.

Ultimately it is only experience that counts :

 what you say I forget, 

what I say, I remember, 

what I do, I understand.

More on COTS vs custom software

Lausanne Flon Copyright 2002-2008 A. Barake

It has been my experience that integration and associated maintenance of commercial off-the-shelf software (COTS) is an expensive and difficult process due to a few factors:

• The "normal" upgrade cycle for vendors is often strategic and is driven by internal rather than external compatibility – upgrades also tend to break interfaces regardless of intent, it just does not help that the intent to play well with others is often at odds with large company policies (i.e. marketing strategy), especially when their suite of products covers many functional domains
• Integration requires glue, and glue can either be COTS or home grown. For COTS, see above objection, and add cost. Home grown defeats the argument of using COTS in the first place Q.E.D.

Now, if we look at home-grown software, we have a few things to consider:

• Maintenance is dependent on knowledge, but unless you are writing in APL  (and even then), the pool of developers with expertise usually exists; reading code and fixing it is what they are hired for and have some training in.
• Developers in my experience much prefer to work on code where the source is available than integrate stuff using configuration and proprietary tools where the options are limited and the dependency on documentation and vendor support is limiting
• Code is now pretty much commoditized except in specialized areas (telephony switching, military stuff, embedded stuff) and even then… there are only a handful of approaches that people use to write and maintain code and most good developers can immerse themselves in a code base within days of weeks and make it their own.
• Renewal is key, and owning a system and renewing it by adding features is easier with home grown that has few integration points with COTS than with disparate COTS on different version cycles with glue in between.

The standards efforts are a sort of socialist (Stalinist really) approach to trying to control the ecological process described above, but is thwarted by vendor power struggles and the usual embrace and extend strategies, coupled with purchasing power in the hands of non-detail oriented people (i.e. not tecchies).

UNIX and other open system approaches have and are trying to tackle the problem through factoring out of common functional requirements into modular technical modules and protocols, and have had some success in advancing the state of the art – the Internet and its associated protocols and services are a direct result. The OSI stack is a manifestation of this approach and has become a pattern of understanding outside that world.

One approach is to layer the commercial stuff and isolate it from other commercial stuff through standard gateway mechanisms, off the shelf if you like, and these include queues, UNIX gateways and subsystems that use HTTP, SMTP and LDAP, xDBC as well as XML protocols where practical and where translation mechanisms are easy to obtain (message brokers, AJAX, browsers, SAML…) At another level we should be writing our own application code to use these underlying systems and the glue should be at the application level – i.e. SOA – services are COTS with standardized interfaces, applications are not.

Les mots

A Baja beach, bay side, Copyright 1988-2008 A. Barake

I was asked by my son about the meaning of the word "load". The context was "load the car". As I explained, by giving an example ("load the bags in the car"), I realized that the example provided an instance that could be generalized, but that was inherently ambiguous outside the continuity of experience. One has to have loaded things to understand that bags are just things, and we can extend the concept, and then we can even go further and talk about "a load" as a thing that is "loaded", without ending up in an infinite recursion.

You know where this is going...

Machines need to have exact mappings of symbols to "actions". Actions are just other symbol manipulations. So we have mappings and more mappings and rules and context and all the fodder of Minsky-type AI. Not good enough it seems. Gelernter and others have realized that one must be embedded in experience to have "knowledge". Husserl and Heidegger said it much earlier, but disciplines rarely cross. So we are realizing now that cognition is a sort of misnomer, we need a word to talk about "knowing", or better still "questioning" to get to a model of reality that is useful.

All this can lead to better and more flexible interface design for a start. Imagine a handshake that allows systems to agree on field semantics and field syntax for data exchange without all that WSDL baggage. Give me a couple of ports and go for it. Virus and spyware writers are now writing the primordial soup that will lead to these higher organisms one day.

Intel and MS

The Concorde parked on Manhattan's Intrepid Air, Space and Sea Museum.

I remember attending another launch, back in the early 1980's - it may have been 1983 - when Windows was just becoming real. It was in Seattle, and Bill Gates was talking to a relatively small audience. Many of the cooler attendees were typing on Tandy Model 100's and making annoying key clicks.

Bill said something that was surprising to me at the time, that he predicted that Intel and Microsoft would become the dominant presences on the desktop. You have to remember that back then there were lots of contenders, including the new Mac, the really advanced Amiga and many other smaller players, and Bill's statement seemed preposterous to me, since his DOS was so primitive and Intel's 8088 was nasty to program, slower than the 6502 (except in clock rate) and not at all like the PDP-11 (which was something the Motorola 6800 and later 68000's were aspiring to). Some of us were even aware of Unix and its possibilities on the desktop.

He was right of course, but I think that he made the prediction come true, rather than actually saying something that made sense. His tactics and acumen forced the issue, and the contenders mostly disappeared, except for Apple. Despite being technologically inferior both the CPU and the UI he promoted became dominant. He saw that business decisions were not made by techies, even in the emerging micro world, but by risk-averse non-techies who prefer a brand name to any technological advantage.

I guess that is why he then started to draw huge audiences.

The Network is NOT the Computer?

A hurdy-gurdie at the Victoria and Albert Museum in London.

I’ve recently attended Microsoft's launch of their 2008 suite of products, including Visual Studio. It was held in a movie theatre where there were two presentation streams, one for IT Professionals and one for Developers. I attended the developer stream. Very little PowerPoint and lots of live demos with code snippets being included in the examples. Visual Studio now supports Javascript as a first class citizen. This drew applause. I guess it was a sorely needed gap in IDE’s. I wonder if Javascript will become (more) fragmented as a result.

The integration of desktop presentation with the network, with attendant DRM-enforcing security as well as extensions to browser functionality is overwhelming and integration of desktop with network apps is almost complete; they are ready to link their OS and .Net environments to Internet content in a seamless way - the network is NOT the computer was the loud subtext. In fact they have made it possible to decouple the presentation layer of an application from the server side so that it works either through a client-server virtual desktop (think Citrix-like), through a regular-looking window, or through a browser.

This is resistance to the Google model, where the client is as thin as possible and standards are used as they should be. The gamble with the MS approach is that users will be sufficiently attracted to the extra features of a tighter integration to the desktop to pay money for it. Makes sense given the business model.

There were also a few intro videos at the start of the various sessions, and the most humorous included a therapist and his patient discussing the relationship between a developer, his machine, the tools and the operating system. Here, the unstated subtext was that colourful computer cases, slick hardware design etc were no match for technical flexibility and ego-boosting developer learning curves (a jibe at Apple I guess). Shakespeare wrote that wisdom comes alone through suffering, which may explain why hazing works to bind a group, and may explain why people who adopt product lines like IBM’s and Microsoft’s end up defending them so strongly. I think it is a case of confusing the side effect with the cause.

More on automated interfaces

• UNIX philosophy of universal read, write, create (sic), seek verbs is akin to financial transactions, where bytes are analogous to money
• HTTP extended this with POST and made it stateless - so it becomes a client-server protocol (a good alternative to X-Windows by the way)
• The universal currency is data, and it can contain anything, it can buy anything. Putting meaning to it is really the difficult part, and the notion of objects is really an attempt at creating a philosophy of data
• Right now we have an inflation of data, too much currency, and its value is being balkanized, some currencies are more precious than others, security related ones, and maybe some video or music bytes. Copyright is an attempt at increasing the value of bytes through currency control.
• So if an object defines the methods we can use to manipulate it, then one way to have objects communicate more easily (ideally automatically) is to constrain what objects can do to a machine manageable set. This may not be much of a constraint, only one that promotes efficiency through consistency.
• Or we can take the idea behind types to heart and have a huge catalog of types and ensure that they can play together - real types which map to useful objects
• For example, a user at any terminal could drop select from a list of things like ADDRESS CHANGE, VALIDATE ID, SUBMIT CLAIM, BUY, COMPLAIN, CHANGE FIELD, etc
• This is the "naked object" philosophy taken one step further. No need for a graphical interface, just a nice "no error" interface - where all choices that work are shown. The "trie" of possibilities would reduce itself as transactions are chosen to interact, limiting the choices the more you decide
• In short, modeling. Can this work in an ad-hoc world? Maybe the modeling should be automated through affinity and tuple-space approaches to ad-hoc property lists
• This, coupled with a hierarchy of state machines (like game play AI) would go a long way to creating automated interface coupling.

That SOA thing

Here is the definition from an authoritative vendor-neutral body:

The World Wide Web Consortium’s glossary defines Service Oriented Architecture (SOA) as

"A set of components which can be invoked, and whose interface descriptions can be published and discovered"

Note that the definition does not talk about the type of transport, it is all about interface descriptions – Web services are therefore a subset.

As in all such interface standardization efforts discovering the service (semantics) is key.

How to make apps talk to each other meaningfully (semantics and syntax) without a human in the loop is a problem old as operating systems.

This has been and remains the “holy grail” of integration. Getting apps to talk across their boundaries has been a staple of data processing since I can remember.

We have had shared memory, files, networking, RPC, CORBA, and the now the latest idea is to re-use the Web servers connected to apps as Web services by parsing the HTML or XML to understand the context of the data across HTTP [We seem to be adding layer after layer of code to do this for some reason. Maybe with the hope of adding some sort of “intelligence”, maybe not…]. When we only use TCP/IP interfaces instead we generalize and call it SOA. This tells us that TCP/IP is now a commodity layer. It is all about commoditization moving up the stack.

In all cases we need to make the plug for the socket and make sure the voltages are OK; i.e. the syntax and semantics have to work across the boundary.

The Web has given us a new way to think about this.

Links – URI and URL’s

SOA is the idea that an app can “choose, click on, and follow a link” by itself to get a job done – and sometimes fill forms too.

There are 4 major verbs we can apply to URI’s and URL’s: GET (click), PUT (put up a resource), POST (fill-in a form mostly), and DELETE (get rid of a resource), plus a few more: This is HTTP. It is meant to stateless (except for cookies and URI-held sessions ….) because the state is supposed to be held by the user – and when the client becomes an app, that can complicate things greatly – all of a sudden you have servers talking to each other with state.

Still, we have not progressed to semantics at the app layer – this is transport and syntax.

We have always depended on humans (programmers, analysts and users). We have been doing this – imperfectly – first through command line, shell scripts, GUIs, and now Web pages and forms.

When you cut and paste, when you embed a table or a picture into a document and when you save a file to be retrieved by another app you are doing inter process communications. Windows and GUI’s allow us to do this more easily, with visuals– it may be their most obvious value proposition – the basis of the MS empire. The Web is similar, we press on links to hop among and between servers and apps.

Computers, on the other hand need to be rote taught. They can crawl web sites, but they have difficulty making them inter operate without humans writing the glue. Web 2.0 is about doing this more easily – “mashing up” service interfaces using XML and HTTP and the smarts of a browser scripting language – ECMAScript (JavaScript)

SOA ideals say that we can publish somewhat constrained specs that will allow machines to automatically selects forms and links and act on them, even when that they may never have seen before.

The more realistic approach is to say that we will need humans to guide the machines, but that the molding of sockets and plugs will get less onerous – we are attempting to standardize the interface syntax and restrict the semantics to a published, well managed, machine-discoverable set. In the interim, it helps us integrate stuff with less work.

Dysfunctional requirements and IT ideology

From my flickr

There are 3 types of requirements : functional, non-functional and dysfunctional (the latter are usually not documented).

The dysfunctional ones represent the distance between the proposed system and the organizational structure.

IM/IT systems must align against the business organization (people and processes) to work well. Management must decide who must budge before a new system is put in place in either camp. When deciding whether to build or buy a system, this effect must be considered. Neglecting to factor-in the cost of the organizational change can cancel the perceived economic benefit of using commercial off-the-shelf software (COTS).

Organizations usually have to change to fit the COTS assumptions, especially for large enterprises with large systems such as enterprise resource planning (ERP). This can also apply to core business systems, since they differentiate the company from others. Think of mobile phone billing for example.

If COTS is chosen (for ERP for example), it is likely that the finance and personnel departments will have to change their processes significantly. There are many case studies that show that CEO support is required for such migrations to succeed, since it becomes a governance issue. It has become the common wisdom now.

Replacing custom-built software with COTS is a form of devolution. Size matters in this kind of decision because the “cost of re-organizing versus the benefit of saving on development costs” equation must be juggled. The bigger the org the more the cost of adopting COTS since the org usually has to change. The converse holds as well.

So where does the argument that COTS is preferable come from? Why does upper management often fear custom work? Why do they see it a a dependence on their technical staff, and why should this be preferable to a dependence on vendors? One possibility is that the pattern emerged from the computer hardware world? COTS for hardware usually makes sense for non-manufacturing sectors since the costs of custom hardware are prohibitive and it is generally ridiculous to suggest that one should build computers from scratch. Again, there are notable exceptions, Google apparently configures boxes quite heavily to suit their needs.

The COTS for software approach is not so obvious with the advent of open source and the devolution from mainframes towards lower cost systems, the proliferation of high-level development tools and the commoditization of the software stack. For example networking and OS are now mostly standardized, and much of the higher level layers are becoming standard - Mail/Web, database and application servers. Even development environment possibilities. The database-to-presentation layers are now the target of service-oriented architecture (SOA) driven commoditization, but the vendors are all looking for their own lock-in (i.e. avoiding commoditization) while claiming interoperability to get the sale.

For example vendor A’s ESB makes integration with vendor B’s apps much more difficult than with vendor A’s. Interfaces are easy to import but hard to export. Such pitfalls can be avoided by a vendor diversification strategy across the application layer that is coupled with good cost control and use of custom solutions where appropriate. Thankfully ESB’s are not a prerequisite for SOA, but this is not generally well understood.

So IT planning strategies need to revisit the seemingly popular notion that custom is always a liability. I think that if cost control is the business objective, then it should be decoupled from the “customization vs. COTS” argument. There is no direct mapping. The organizational size attribute has to be factored in, amongst other things.

Another ideology-prone minefield is standardization. Standards should really be orthogonal to this discussion, but are related in strange ways. Standards are sometimes perceived as a factoring exercise to reduce the proliferation of types of solutions and products in an IT department. This can work very well with hardware. Where standards are problematic is when they prescribe major system software COTS without taking into account organization structure (again). Why prescribe COTS through standards? Well COTS is good, right? And less COTS variety should be even better. Hummm.

Vendors claim and often try to provide the flexibility in their products to allow (some) business process mapping and legacy integration but usually at the cost of configuration complexity or alternately through costly professional service customization.

The benefits of these approaches over custom development need to be weighed carefully; there are plenty of case studies that show success with either approach and the cost-benefit is not at all straightforward, especially when you consider upgrade cycles and other dependencies.

If product/portfolio standards are adopted by an organization, then it must also adopt compatible organizational and operational standards, since software is really an extension of organizational processes and COTS will force an organization into a process relationship with the vendor. It is a form of outsourcing.

This can ultimately lead to having the hired-guns, the management change consultants and the vendor IT/IM professional services too tightly coupled, resulting in them having a profound influence on organization structure and operations. Are organizations prepared for this kind of loss of control in exchange for perceived but generally illusory cost benefits?

I think that one of the sub texts of the discussion is at root ideological - outsource development versus do it in-house. Yes, IT departments do have much influence and are complex things to manage, and require expertise, but control cannot always be gained through outsourcing or by buying COTS, unless you are very small or have very simple needs. Horror stories abound. Control and accountability concerns need to be well understood and documented and the business objective must be stated explicitly and decoupled from the implementation. There are no silver bullets, only smart management.

Vico and iterative design - History as a spiral

Giambattista Vico was an Italian 17th century academic who wrote about history scientifically. His thesis is that history is a helix, and that the we can know history better than we can know nature because we make history. It is the New Science.

History is iterative design, in short. We are creating our cultures through spiral cycles of successive approximations. There is no overarching strategy, only tentative approaches, herding of instinct by the powerful, and some imperfect knowledge of the past that gets better as we approach the present.

Iterative design is the norm. Anything else is just so much hubris.

Waterfalls mostly create entropy.

The Art of (tech) War

(photo copyright 2007, A. Barake)

Where I talk about more-or-less obvious but infrequently discussed strategies by Sun, IBM, Microsoft et al.

I think it was in during the last century, with the advent of the hardware/software dichotomy that this type of strategy evolved. IBM may have been the first to use it successfully. A prerequisite is a pair of complementary things to sell, with at least one that you control completely, through secrecy, patents or a huge market lead. Let me describe it.

Suppose you sell hardware, and you make and sell software for it too. Your competitors also sell hardware and software of course. Your hardware is different and possibly proprietary or difficult to clone or to re-brand. So the strategy is to make software that runs on both your own and your competitors' box. You under-price or give away the software, and call it bundling.

Better still, you give away stuff that is almost or as good as your competitor’s, and you value-add and sell a version that runs only on your platform.

Sounds familiar?

IBM with Java and Linux
Sun with Solaris
And both are also attacking Microsoft with Eclipse and OpenOffice respectively.

Yet Microsoft thrives.
Explain in 2 pages or less....

Microsoft fights back using another strategy; they sell software that runs mostly on smaller cheaper platforms. They have an arrangement of co-dependency, as their releases grow and become more bloated, they require bigger and bigger Intel horsepower, until that low-cost hardware begins to overtake the computing power of big boys. This is an attrition strategy; much vaunted in management theory courses.

Microsoft's core business is software, so it is the protected commodity. The hardware is Intel, but multi-vendor and commoditized so competition drives the price down below the proprietary guys’ stuff. They are pursuing a software-first strategy and the other guys are pursuing a hardware-first strategy. Are you still with me?

It is with reason that IBM fears Microsoft more than it fears Sun or HP.

IBM and HP also sell Intel servers, but they are branded, not commodities. Both try to mitigate the hardware-first exposure by also making money on professional services (which can sometimes be an adjunct to sale - nudge nudge), server software and very big boxes. Forced upgrades due to byzantine dependencies among their software offerings help too, and the upgrade cycle is one way to drive this.

IBM has little software presence on the desktop except with IDE’s and I’m not sure they sell workstations since the Lenovo deal. In other words, they threw in the towel for now.

On the Internet side:
The classic version of this story is, of course, Microsoft vs Netscape, where hardware becomes system software and software becomes the browser. We know who won.

More subtle, as chronicled by others before me, Google learned a thing or two from this. They give away email, blogging, API’s, server side apps - to counter Yahoo’s offerings in that space. Free and easy Web access for the masses, no emphasis on paid subscription modes, whereas Yahoo does act as an ISP. Google’s core business depends on search and on page views, all the rest of their offerings simply draw customers into their portals while at the same time undermining Yahoo’s non-search businesses. They are not evil but almost...they followed in the footsteps of the well-tried IBM and MS strategies.

Yahoo - has no strategy that I can tell. They are like Sun. Java was a gift to the community that hurt Sun, since it is platform independent. I guess they thought that they could make an OS/hardware combo that ran the JVM better than the other guys, but IBM with its marketing (and technical) clout claims that that is not the case. In a way, it is as if Microsoft had written Linux. That is what happens to a nice earnest tech company that swims amongst the sharks.

Is their hope for the "good tech" to overcome the "mediocre tech" with good strategy?
My proposal would be combine the Apple, Sun and Yahoo brands as the high-end stuff of computing. The champagne. Then market it to consumers and businesses as the cream of the tech-savvy crop. Apple and Sun are an almost perfect match, and it would get Apple into that server market they so covet. R&D would be a plus rather than a risk, since the audience would want to take those risks, it would be cool, fodder for the early adopters. Yahoo would provide even more channels for Apple’s media side

There. Free strategy. Why don’t I command the multi-million dollar bonus?

I know many tech types who can't bring themselves to believe that that kind of “deep thinking” goes on; but this is exactly what the people who took biz courses when you were toiling to understand complex stuff end up doing, and they make the big bucks. Hustlers, thugs, ruthless opportunists... having the only kind of fun they can, since they can’t make things, are too competitive to collaborate with anyone, and certainly can’t code. I think they are missing out, because in the end, what's left is an abstract and hollow victory, scorched earth, and lots of resentment against inferior technologies. Real satisfaction can only come by making things better, and not just in the monetary sense as noted here. The alternative is growing entropy until the next cleansing crisis.

P.S.
This just in.

Logic of opposites

(Picture of Lausanne Park copyright A. Barake)

I recently read a blog quoting E. Wiesel, about how the opposite of love is not hate but indifference. ( I thought it was Dave Pollard, but I can't find the link). In any case, the point is that attention is the commodity being given or withheld. I am severely summarizing, but I tripped on the word "opposite". How do we define it in such a context? Why can't hate be the opposite of love if intimacy or something else defined as the desired commodity?

Math and logic try to resolve these ambiguities by making distinctions between converse, complement, and other technical relationships, that have something to do with oppositions. The usefulness of these constructs depends on their generality and how they fit within the meta-world of math operators and relationships - set theory etc. Why this is satisfying is a mystery. It may be that our minds are happy to tie these loose ends together in a big coherent construct, or it maybe that our world is actually organized in this way, at least physically, but I am not sure.

I think that what the crisis of mathematics that occurred over 100 years ago shows is that as culture and knowledge and philosophy mature, fragmentation inevitably takes hold. Size matters in that holding something too big together takes exponentially increasing effort, and this applies to things both physical like boats and civilizations or entirely platonic - like mathematics.

Words like "opposite" and "similar" can be so misused in important contexts. Mathematics tries to bring precision to them by limiting the context, and some people try to bridge that limited context world with the real one via ideas like AI, but I am not sure that there is a mapping. the computer is a limited for now in its interactions with our reality.

Also, logical operations depend on predicates, and transitivity, and operations that are repeatable and consistent, which means that the environment in which they operate must be stable. This is not a good environmental expectation for non-mathematical reality.

I was attracted to mathematics and computer programming because it gave me this sense of control, but now I realize that as in programming, the real-world intrudes and must really be considered first.