I have just begun a new job. No hiatus between this new one and the old. One day I am in hell, the next in paradise.
The one I left was as a software manager in a large organization, and the current one is as a software lead in a very small organization.
Other objective differences: the previous job was bureaucratic, there were many processes and constraints and a network of communication and decision paths, and the systems my team built had to reflect the complex interrelated and often historically constrained requirements.
The current one is technical, working within a small team and with very precise mostly self-defined scientific requirements, adjusted to meet specific customer requests. The software has to run precision equipment to capture and calculate results within extremely tight tolerances, both in time and numerical accuracy. The only measure of quality is repeatable precision. The culture is one of technology and science.
In the new job, I report to people who have the same engineering training as I do, who have more experience in the R&D business than I do, who I can ask questions of without risk of offending or destabilizing them emotionally. Peers and superiors in all senses.
In my previous job, I reported to people who asked me to explain what I did "in simple terms" and who liked to hear themselves talk so that they could pretend to make sense of their lumbering thoughts in public forums, often referred to as "meetings" but which felt like beatings. Dissent could be career limiting.
The analogy I would like to give to dispel any conclusion that I may be a pretentious ass is as follows:
Suppose you are a medical practitioner, a physician, and you see patients in a clinical context. You have to diagnose and prescribe treatment as part of your day to day routine.
Also, suppose that you are "managed" by someone who had not undergone the same level of training as you, say a registered nurse, who knew the lingo, understood the context, but did not feel confident enough to make life or death decisions affecting patient health, but felt competent enough or was somehow appointed to tell you, the physician, how to run your practice.
For example, "please use language that is easier to understand in your charts". Or "make sure your handwriting is legible for the pharmacist", or " you have not seen your quota of patients today, why did you spend 6 minutes more than average with Mrs. Smith today?".
Could you practice under such circumstances? How long would you last?
I lasted 5 years in the analogous IT context. My manager was a technologist, previously a school teacher who joined th IT boom of the nineties and hung on. She did a bit of COBOL programming, hated it and went into management because she could talk better than she could listen or understand.
She is proud to claim that she does not understand the need for system architecture versus program design, her eyes glaze over when there is talk of latency, language choice, servers, data flow analysis, code profiling, and factoring and she had no patience for options analysis or proof-of-concepts. She will not deign to read code. She has trouble with email client software and spreadsheets.
She wants simple, clear explanations and plans, and wants to set deadlines before beginning design, because iterative work can only lead to grief. She always used waterfall approaches in COBOL, and they worked fine for her.
Her weakness and fear prevent her from adding value, from making decisions. She was a pass-through for her manager's decisions to me and my peers. No value added, but lots of aggravation and delay. Never understood or wanted the concept of situational management, which is probably the most effective way of managing people, so effective that there have been lawsuits about who has a right to claim authorship and teach it. She is a scared rabbit, hanging on until she can retire and cash in.
Others in that environment have said that we, the technical folk who did the work, should not "speak Neanderthal" when in the presence of senior executives, by which it was meant that we should use non-IT terms of less than 3 syllables. The executives in question were the CIO and her direct reports.
But all this is now in my past, one-day-old stuff. I am now in another world, where the conversations around me centre on measurement tolerances, ADC resolutions, real-time interrupt-driven code, clock speed and sampling rates, hierarchical state machine approaches, language selection and compiler efficiencies. I have scope probes and micro-controllers on benches beside me, no fuzzy walled grey cubicle partitions in sight, and a project to deliver with people I can talk to who want me to talk dirty.
I am in heaven.
IM/IT systems must align against the business organization (people and processes) to work well. Management must decide who must budge before a new system is put in place in either camp. When deciding whether to build or buy a system, this effect must be considered. Neglecting to factor-in the cost of the organizational change can cancel the perceived economic benefit of using commercial off-the-shelf software (COTS).
Organizations usually have to change to fit the COTS assumptions, especially for large enterprises with large systems such as enterprise resource planning (ERP). This can also apply to core business systems, since they differentiate the company from others. Think of mobile phone billing for example.
If COTS is chosen (for ERP for example), it is likely that the finance and personnel departments will have to change their processes significantly. There are many case studies that show that CEO support is required for such migrations to succeed, since it becomes a governance issue. It has become the common wisdom now.
Replacing custom-built software with COTS is a form of devolution. Size matters in this kind of decision because the “cost of re-organizing versus the benefit of saving on development costs” equation must be juggled. The bigger the org the more the cost of adopting COTS since the org usually has to change. The converse holds as well.
So where does the argument that COTS is preferable come from? Why does upper management often fear custom work? Why do they see it a a dependence on their technical staff, and why should this be preferable to a dependence on vendors? One possibility is that the pattern emerged from the computer hardware world? COTS for hardware usually makes sense for non-manufacturing sectors since the costs of custom hardware are prohibitive and it is generally ridiculous to suggest that one should build computers from scratch. Again, there are notable exceptions, Google apparently configures boxes quite heavily to suit their needs.
The COTS for software approach is not so obvious with the advent of open source and the devolution from mainframes towards lower cost systems, the proliferation of high-level development tools and the commoditization of the software stack. For example networking and OS are now mostly standardized, and much of the higher level layers are becoming standard - Mail/Web, database and application servers. Even development environment possibilities. The database-to-presentation layers are now the target of service-oriented architecture (SOA) driven commoditization, but the vendors are all looking for their own lock-in (i.e. avoiding commoditization) while claiming interoperability to get the sale.
For example vendor A’s ESB makes integration with vendor B’s apps much more difficult than with vendor A’s. Interfaces are easy to import but hard to export. Such pitfalls can be avoided by a vendor diversification strategy across the application layer that is coupled with good cost control and use of custom solutions where appropriate. Thankfully ESB’s are not a prerequisite for SOA, but this is not generally well understood.
So IT planning strategies need to revisit the seemingly popular notion that custom is always a liability. I think that if cost control is the business objective, then it should be decoupled from the “customization vs. COTS” argument. There is no direct mapping. The organizational size attribute has to be factored in, amongst other things.
Another ideology-prone minefield is standardization. Standards should really be orthogonal to this discussion, but are related in strange ways. Standards are sometimes perceived as a factoring exercise to reduce the proliferation of types of solutions and products in an IT department. This can work very well with hardware. Where standards are problematic is when they prescribe major system software COTS without taking into account organization structure (again). Why prescribe COTS through standards? Well COTS is good, right? And less COTS variety should be even better. Hummm.
Vendors claim and often try to provide the flexibility in their products to allow (some) business process mapping and legacy integration but usually at the cost of configuration complexity or alternately through costly professional service customization.
The benefits of these approaches over custom development need to be weighed carefully; there are plenty of case studies that show success with either approach and the cost-benefit is not at all straightforward, especially when you consider upgrade cycles and other dependencies.
If product/portfolio standards are adopted by an organization, then it must also adopt compatible organizational and operational standards, since software is really an extension of organizational processes and COTS will force an organization into a process relationship with the vendor. It is a form of outsourcing.
This can ultimately lead to having the hired-guns, the management change consultants and the vendor IT/IM professional services too tightly coupled, resulting in them having a profound influence on organization structure and operations. Are organizations prepared for this kind of loss of control in exchange for perceived but generally illusory cost benefits?
I think that one of the sub texts of the discussion is at root ideological - outsource development versus do it in-house. Yes, IT departments do have much influence and are complex things to manage, and require expertise, but control cannot always be gained through outsourcing or by buying COTS, unless you are very small or have very simple needs. Horror stories abound. Control and accountability concerns need to be well understood and documented and the business objective must be stated explicitly and decoupled from the implementation. There are no silver bullets, only smart management.