Offshoring and Specification - Improving the Global Delivery Model

Offshoring has been something with us for at least 10 years. Initially much of it was targeted at testing. There has been a lot written about offshoring and on the role specification plays in achieving a successful offshoring delivery model. So it is to this relationship between an engine that has a global delivery capability (and Cognizant is a fine example of this) and how to make it even better that I blog.

Offshoring is often done on fixed price terms. The financial size of the project and the way in which the contract and subsequent remuneration is managed relies heavily on specifications, governance of the delivery and change management. To date most offshoring companies have similar, although with subtle differences, models for doing this.

The advent of UML as a language for modeling and specifications has certainly helped in understanding what should be delivered. The use of programme management and enterprise architecture functions to govern delivery with clients is a common feature. Many offshoring companies and indeed those that were local Systems Integrators have to a large extent pioneered the roles that we play; from the programme management roles to the enterprise architect roles. Governance and supporting frameworks have been developed to help the process and have been used to great effect. TOGAF certified architects, MSP certified programme managers are in demand. Of course this does not mean TOGAF and MSP are the only games in town, I only mention them because they are well known.

The fundamental cost in offshoring the Software Development Life Cycle (SDLC) is one of people. And because people are involved the removal of ambiguity in specifications where possible is a critical success factor it both protecting the margins of offshoring companies as well as protecting the delivery of solutions to customers.

Not much has changes in terms of the specifications and the methods to create specifications nor in the way in which specification are used to guide the SDLC. Typically success is characterised by a model in which an Enterprise Architect may work alongside Business Analysts onshore to frame a solution and the Programme Manager liaises with the customer and Enterprise Architect to ensure a suitable roadmap for phased delivery of the solution. The same two roles work together on the contract details which reflect the roadmap. The programme (that is the collection of projects needed to deliver the solution) kicks off and the Enterprise Architect and Programme Manager (often with similar roles from the client) ensure the programme is well governed meeting all the targets and adhering to the standards of communication and delivery needed as well as managing changes.

Within the process that is enacted the specification of the solution becomes a key part for both governance and change management. The specification is the document (or documents) that detail what needs to be built in order to deliver the solution. Of course the entire process is larger than just this piece but for the most part the rest of the process deals with change management and seeks to ensure alignment of the solution to the customers needs throughout the life of the programme - hence change management because what we think is correct never is what is needed over the life of a programme; things change.

The specifications delivered to an offshore engine need to make clear what functions need to be written, what inputs they might have and what outputs they might deliver. The specification needs to make clear the order that functions can be used. I have not mentioned data explicitly because it has no meaning without the functions. A piece of data is just that until we decide to do something with it. What we do with it is essentially a function.

The challenge to creating and delivering good specifications is to ensure that they are internally consistent, that they are externally complementary - by which we mean that two pieces of software will work together or interoperate, that they are aligned to the business goals of the customer.

The way in which we meet the first challenge - internally consistent - is largely through governance and within it review. They way we meet the second challenge is the same. The way we meet the third challenge is through transparency with the customer of the SDLC and the solutioning process and of course through continued governance.

Of course when things change we need to assess the impact of that change to projects already in flight. Again this is done through review under governance.

The pattern here is that the key challenges are met with people based processes. And the problem with that is that it is ambiguous. The specification are often ambiguous internally and even more so externally. Not because people are not trying by because the level of abstraction for specifications is not amenable to adequately describe what is needed.

If we use UML State Machines and WSDL contracts as an example. We might have 100 services to create. They all have WSDL contracts. Some of the services will be stateless (typically data look ups, technical services and so on), some will be stateful (this doesn't mean they hold state within the interface but it may well mean that they have knowledge of one or more business transactions and so state relative to the business transaction becomes very important). If 20 of these services need to collaborate then the UML State Machines need to reflect that collaboration. The way we ensure it works is to check each State Machine against what we think are the related State Machines. As an example look at the two below, they should be complementary but are they?





Life got a little better when BPMN came along. It's level of abstraction is higher. But the problem with this is that it very quickly becomes unmangeable and unintelligable because the complexity is delivered as links across services (the roles or actors). Imagine a BPMN diagram for a seemingly straight forward problem. I show one below and you see immediately that the complexity makes it unreadable (and yes I know it is small but the point is that the picture is clearly complex).



So how do we do it better? How can we deliver better specifications and so improve the global delivery model and leverage offshoring at an industrial scale without needed so many heavy people based processes? In short how can we automate some of this and use computational power to make our lives easier?

The answer is to use testable architecture. In my next blog I shall show how it works and how it can be applied and the benefits that ensue. I shall leave you now with a simple business view of testable architecture for a simple example that was generated from a language that is unambiguous and really does specify at the right level of abstraction:

Some fun stuff

I lived and worked in the US from November 1985 to August 1987. During that time I probably learned more about IT than at any other time. I worked on some cool projects met some cool people and had a great time.

I spent many a Saturday night on the lower east side of Manhattan and small bar were I listen and generally lost myself to "Joey Miserable and Worms". See the link to the left.

Normally I blog on tech stuff but this time, for fun and in memory and affection of those times I leave you Joey Miserable and the worms and Worm Opus.

Enterprise Transformation

Overview
Transformation seems to be a bit of buzz word these days. I have seen an increase in bids that involve or are centred around transformation. There is much written about transformation that is available on the web and most of it is in the same vein, namely that IT is an enabler and that integration is costly and that SOA provides some way to reduce this cost and so increase the agility of the underlying IT assets. What I want to do is provide more detail on why this is so.

Enterprise Transformation and IT
So what is transformation in an enterprise and IT context.

Transformation at an enterprise level is about process, people and organisational structure. IT is only an enabler, its role is to automate where possible and assist where full automation is not possible.

For transformation to succeed it must have a business context. When we transform we know where we start from and where we need to go to. Transformation is better achieved and returns greater value if it ties return on investment along the way and becomes a programme of many projects that implement the transformation such that it is aligned to business imperatives. If there are no business imperatives that can be articulated then there is no reason to transform.

Enterprises engage on transformational programmes of change because they want to improve their own efficiency and because they wish to re-align their business with, what they see as, future market conditions. Of course the market changes and so the ability to continually adapt to change becomes ever more important to the place an enterprise needs to go. Equally an enterprise notion of efficiency also changes over time. Some processes that previously would be in-house become commodities and can be outsourced. But to achieve this the enterprise needs a framework in which the processes and the IT landscape sit that can support major adaptation.

The people in this transformation, as they have always been, are very flexible. Transformation can be painful but in the grand scheme of things people adapt and perform in line with the business imperatives that drive the transformation. The inhibitor to transformation is more often than not IT. Computer systems and software are not good at adapting to new situations and this is the bigger challenge to any transformation.

At an enterprise level, transformation change requires re-alignment of process, people, organisational structure and supporting IT to business imperatives. It isn’t just about IT, it isn’t just about people, it isn’t just about process or organisational structure. It is about business imperatives, business context and the way in which business goals can be achieved. Processes are simply a way of ensuring that all of the moving parts (people and IT) do what they need to do at the right time. People and organisational structure is about authority and roles to achieve business goals. IT is simply a way to automate and assist people through encoding of some or all of the processes to ensure greater efficiency. Given it is all about business goals and involves all of these moving parts it should come as little surprise that this is what Enterprise Architects do. They don’t do technology, they do transformation.

I've worked on process modelling since the late 1980's. First with Sema Groupe and then with Object Design for IBM on Fidelio (the code name for Flowmark which is now part of Websphere). It because apparent then that business processes would be very important to enterprises as IT became more prevalent in our society. The downside was always the integration of components and not the process topology. Designing a process as a model is the easy bit but enacting it over disparate resources is always the more costly exercise. Those of us that worked on process modelling back in those days knew that integration and reducing its footprint in the equation would always be the holy grail. If we could make the cost of integration low enough and the time taken to integrate short enough then IT would no longer be the impediment.

The role of the Enterprise Architect
Good enterprise architects have an understanding of the technology drivers that enable transformation to be effective and at the same time have an understanding of organisational structure and roles that need to be in place to enact transformation. They don’t have all of the answers but they think at a similar level to a CXO – which is why they often face off to them. They facilitate the reshaping of an organisation, an enterprise if you will, with the CXO driving and the enterprise architect providing grounded forward thinking advice and guidance on how transformation can be supported by IT in an holistic way commensurate with the business goals.

The role of Models, SOA, BPM and Business Rules
Having a good understanding of IT and IT trends in the future is one of the key skills that good enterprise architects bring to the table. The advent of SOA, the resurgence of BPM and the newer resurgence in Business Rules make transformational change much easier to achieve and much easier to drive. SOA cuts the cost of integration to more manageable levels through the use of standards and standards compliant tooling. BPM provides a framework for process encoding that provides the structural clarity of how an enterprise will work (across the IT and people divide). And Business Rules provide the controls for steering the enterprise ship as it sails towards it’s archipelago of business goals.

But of course none of this is any good unless we can capture where we are and where we need to go to. Having a profound understanding of an AS-IS landscape – by which I mean the process, people, organisation structure and IT – is a base for transformation. Such an understanding needs to minimise ambiguity and maximise clarity. Good models become essential. Good models that can be tested become highly valuable and good models that can be reasoned over become a differentiator.

Having that AS-IS state allows the CXO team to better understand where they are and so construct similar models that describe where they want to go.

The journey to a TO-BE landscape cannot always be determined in advance, it is a journey. To be able to flex and adapt the enterprise without being hide-bound by IT enables the CXO to try things out rapidly. In much the same way as we use agile methods to develop solutions in the face of inexact requirements and in much the same way we might use CMMI to drive iterative improvements we can apply the same techniques to the enterprise as a whole and so continually adapt it to improve efficiency and meet business goals.

Where to start
The first step of this journey is to move towards service enabling the enterprise. Capturing the key processes, automating where we can picking low hanging fruit to turn an AS-IS landscape into a more agile AS-IS landscape. Picking the low hanging fruit can often help to pay for the enablement over time. When enough enablement has been done the transformational journey can begin in earnest because at that point enough of the enterprise is flexible and agile enough to be flexed.

A danger in picking low hanging fruit is to silo enterprise transformation thinking. Looking at one process is never sufficient as it does not engender any synergy. So a single process view does little to help transformation. Rather a more global approach is needed. Capturing the AS-IS and then transforming to a TO-BE state over time requires a global model approach which is neutral to the technology and crosses process boundaries. Testable architecture concepts help to capture a global model by documenting an AS-IS (and enabling a journey to a TO-BE state) to be written formally as a collaboration in which processes may collaborate at the top, middle or bottom of the process graphs. This helps to rationalise the IT real-estate and encourages synergy.

Conclusions
Consider the humble Ford car manufacturing plant and the American football team. In the case of the former they can re-purpose a plant to manufacture on car type or another in 24 hours. They have delivered adaptability for their business. In the case of the later they coach has a play book and a set of behaviours (the players). The goal is to win the game and the objective is to make a first down each time or stop the other team or take a field goal. The coach matches behaviours to plays at each stage of the game demonstrating adaptability.

Thus for me the key to enterprise transformation is to free the enterprise from the moribund pace of change of IT through service enablement, focussing on early ROI to justify the enablement and with a longer term vision of CXO’s to have a flexible and adaptable enterprise for the future. This is why global models are important (and so testable architecture given first through CDL tools). This is why SOA is important. This is why BPM is important and this is why Business Rules are important. They all help in enabling adaptability at differing levels of a enterprise, from the top, in the middle and at the bottom.

And finally, the guardians of technology enablement are not the vendors – they have little insight as to how transformations are performed – it rests with the systems integrators leading the industrialisation of IT.


References:

IT for Enterprise Agility

SOA and Large Scale and Complex Enterprise Transformation

BPM is Not the Same as BPR

Building the Agile Enterprise

Finding the right services to a problem

Almost certainly the will be the start of a stream of blog entires. I am still mulling over the plethora of techniques we are told about and still trying to see if any of them really help or if it is a case of good design principles rehashed yet again. Perhaps if I come up with something concrete we will call it YASIM (Yet Another Service IdentificationMethodology). Anyway here goes part 1.

One of the most difficult things to do in SOA-based projects is finding the right services. There is much fear uncertainly and doubt about this. To meet the challenge many vendors and integrators are seeking to create methodologies to support the process. Consequently there are lots of methodologies about logical designs and how to derive services from them. Some use data flow techniques other use process modeling techniques and all have some taxonomy of services that classify services (albeit with different names) as "business", "data", "technical" and "utility".

The data flow and modeling techniques are intended to tease out services that are business aligned by looking at the data (data flow) or the process. Services are derived from such views and then classified.

Data modeling apart - largely because it is not the way in which the prevailing winds blow - relies on some notion of activity grounded to a role. BPMN is a typical graphical language that is used. This technique is fine for the most part although in the case of BPMN scalability is an issue because it put roles at the front and obscures the process as a result, but nonetheless it works for the most part in deriving services.

Of course deriving what services are needed is only one step in the process of delivery. What is needed in support of this is to identify services that might fulfill need. In a greenfield site this is not a problem but as SOA adoption grows one of the benefits is supposed to be reuse. So how do we ensure reuse and what does it mean?

Finding a service that has both the correct method signatures and the correct behavior, the keystone of reuse, is for the most part an aspiration. It would be wonderful if we could ask a service repository for a service with methods "foo(FooXML)" and "bar(BarXML)" in which "foo" is always called before "bar" or in which order is irrelevant and for the repository to tell us what matches our query.

The reason why we cannot do this sort of search is in part because the behavior of a service is not captured in the repository and because the architecture description (data flow model or process model) does not support any formal linkage between it and the service descriptions in the repository. So we are left to figure it out and the best we can do is find that a service has a "foo" that takes a "FooXML" and a "bar" that takes a "BarXML" as input (similarly for the outputs).

I was at a recent Scribble meeting at Imperial College and I saw the future is now. The Pi4 Technologies Foundation along with Imperial College and Queen Mary College and a few others have been working on a language called Scribble. It is "son of" WS-CDL. A much cleaner curly brace notation for describing teh dynamic behavior of an architecture. Most importantly it has a behavioral type system. The demonstration showed that an encoding of WS-CDL can be rendered into Scribble and then the type system can be used to check for behavioral conformance of JBossESB actions. If you change the WS-CDL having generated the JBossESB (directly or by hand) you can see areas of conformance and non-conformance. Finally we have the linkage between the intent of the architect and the implementation contract of a service in a true SOA platform.

I am looking forward to playing with this when released as part of project Overlord and I expect it make reuse the norm rather than the exception and to do so at a fraction of the cost.

Mulling over architecture

I've been mulling in no specific direction today. Letting the wind take me to wherever it wishes and that often leads me to think more clearly.

I was talking to my dad over the weekend. He was involved in avionics most of his working life and was the examiner of the institute of Quality Assurance in the UK. We were talking about the Farnborough Airshow - which I attended on Friday - and about fly by wire. The European Fighter Aircraft (Typhoon) is a fly by wire as is the A 380. I don't know about anyone else but the standard of software development would make me concerned if I had to fly in one of those. I try to forget it all when I board an aircraft so I don't worry.

It turns out that much of what they do to increase the reliability of fly-by-wire is to use redundancy. Triple systems is one approach that is used. The components are developed independently and then monitored. When they diverge majority voting is used to determine the likely correct behavior. It is statistical because there is no guarantee that the behavior of the two that agree is correct. It is simply the case that two agreeing is likely to lead to correctness.

In high availability system of this nature they use multiple compliers too and they might choose to use different chip sets.

Given my stance on top-down it is heartening to see that project definition is what often guides these systems. No code is cut until after project definition which in turn provides the specification of the system. So what happens when they change things? They negotiate the change and provide a plan for the introduction of the change. It cannot be done on the fly because the complexity of the change often mitigates against a more agile approach. So they simulate they test and so on. What they have is very good governance processes which document the change, the impact and the resulting tests prior to productionisation. All similar to the approaches we use in commerce oriented solutions without simulation and without testing the architecture in any way.

The role of project definition is to provide the requirements against which testing can be measured and against which simulation can occur. The simulation provides a first step towards testable architecture ensuring that the overal design is commensurate with the requirements.

As an aside High Integrity Software development became a real vogue in the 1980's. One of the earliest proponents of formal methods which have underpinned High Integrity Software has been Tony Hoare (Elliot Brothers 1960 until 1968) who oddly enough was at Elliot Brothers during my dad's tenure (John Talbot 1961 until 1965 and then again 1968 until retirement in the 1990's).

So what does this mean to the world of software and commerce. Formal methods are valuable but not a panacea. They need to be introduced by stealth with their benefits laid out. They need to be employed early. As Anthony Hall elaborates:

"It is well known that the early activities in the lifecycle are the most important. According to the 1995 Standish Chaos report , half of all project failures were because of requirements problems. It follows that the most effective use of formal methods is at these early stages: requirements analysis, specification, high-level design. For example it is effective to write a specification formally rather than to write an informal specification then translate it .It is effective to analyse the formal specification as early as possible to detect inconsistency and incompleteness. Similarly, defining an architecture formally means that you can check early on that it satisfies key [functional and non-functional] requirements such as [message order and content], security [and performance SLA's]."

Formal method have been used in avionics for some time. Hence Tony Hoare's involvement in Elliot Brothers. They are for he most part hidden and become a function of tools that are used for design. The use of the Z notation in the early 1980's found popular acclaim in Ada based systems. The problem was it's cryptic nature and therefore lack of skills in using it. Which is why stealth is a good thing. Tie it up in a tool and make it easy (just like type systems in programming languages).

We have a much clearer formal understanding today of distributed computing through the work of Tony Hoare and Robin Milner. What is needed are tools to help us define architectures that remove the ambiguity of human translation and provide a mechanism for the analysis that is needed, henace the cry for formalism. The pi4soa tool suite is but a start. It can become more refined and integrated with other tools (such as Archimate and those specific tools that support Archimate). Architecture tooling and tooling for design is not the most popular of directions because of the lack of runtime scale for remuneration. But they are much needed as in the end they will enable solutions to be build faster, with higher quality and at lower cost whilst remaining suitable agile and aligned to the business and this is what formalism (suitably hidden) can provide.

The Industrialisation of IT

Possible the most important invention that gave rise to the industrial revolution was the micrometer. The inventor of the micrometer was William Gascoigne in the 17th century. It was directly responsible for the engineering discipline in constructing the steam engine and in constructing the Enfield rifle that was used during the civil war in the United States.

What the micrometer did was remove ambiguity. It gave rise to a language of design that enabled precision engineering which by extension gave rise to industrialisation with bullets being made in one place and gun barrels in another.

What has this got to do with CDL?

CDL is possibly as important in the industrialisation of IT. It gives us a language of precision of a system of services which in turn ensures that services are precise (by design) and so interoperate properly - just as the micrometer did for Enfield and Stevensons Rocket.

In classic engineering today simulation is used along with some formal mathematics to test out a design to ensure that it will work. In CDL the same principle is used to simulate and to test so that before a line of code is cut the CDL description is shown to be valid against the requirements and to be correct in computational terms (i.e. free from live locks, dead locks and race conditions).

Testable architecture along with a language of discourse that is precise removes the ambiguity between implementation and requirements. It enables industrialisation and facilitates off shoring of implementation in the same way that Enfield used the micrometer and the precision it gave to design to manufacture solutions in different locations and yet ensure that things work when they are put together.

Response to Luis's comments in my last blog entry

Firstly thank you for the comment Luis.

It has been a little while since I last blogged. In that time the latest Pi4SOA release came out. Some of it's capabilities are listed below in the narrative.

To get the latest release follow the instructions at the end.

On to my response to Luis's comments.

Luis wrote:

Very good article indeed. It clearly describes the main doubts most people have regarding the real added value of CDL and how to addressed it. It is clear vendors are more worried about short term ROI other than best practices and real heterogeneous enterprise architecture; evidence of this is how vendors sell “SOA Suites” as a product / package and not as an architecture approach or practice as it should be (this is probably one of the reasons why many SOA projects end up in DOA –dead on arrival– instead of addressing the real business needs and technological value)

One suggestion; it would be nice to mention how and when CDL can be plug together with other technologies such as BPMN, SCA and BPEL on the different stages of a SOA project and the roll it has one the solution is in place (e.g. in the governance).


Here is my response:

The Pi4SOA tools suite currently supports BPMN. It is simply and export format in which CDL is exported to BPMN. However CDL is richer that BPMN so it is not free from loss of the expressions in CDL. However it may serve the role of review collateral.

BPEL is also supported. You can generate BPEL just as you can generate Java. The downside is that BPEL is more restrictive than CDL because CDL is wider than classic web services and provides message exchange patterns that BPEL cannot handle but which are applicable to SOA (i.e. first class notifications). The BPEL generation is still prototypical as it does not garner much demand. However I think that Overlord will change that.

As to SCA, no current bindings exist to SCA but I do not think that SCA in and of itself is really a target for CDL as it fits two level below rather than one level below. One might of course make the association of SCA to ESB in which case the latest release supports JBossESB out of the box. The problem with ESB's is that there is not real standard. Rather an ESB is a collection of integrated components (Message Bus, Orchestration, Registry, Business Rules and adapters).

Governance is all about control and traceability. This is exactly what CDL provides as an SOA blueprint for a dynamic model. It is the language of architecture in which Java, C# and BPEL provide business logic inside services and in which CDL describes the collaborative behavior of the services as peers. It provides traceability back to requirements (example messages and sequence diagrams) today and in the future will also deal with the non-functionals (SLA's WS-Policy statements and so on) and it provides total coherence to implementation through Java generation, UML (XMI model) generation, and BPEL generation as well as providing runtime checking of behavior against the description. So it provides top to bottom alignment from requirements to implementation. In this way it provides a governance platform that can help to manage the complexity of change over a large real-estate of services. Which is why Cognizant and Redhat are interested in CDL.

Downstream we might try to join it up with ArchiMate, which will provide the route to requirements from the business. But that is some time away and work has yet to start. If anyone is interested then contact me at at the Foundation (see below).

To use the latest release follow the steps below:

1. Go to www.pi4soa.org
2. Select "Download->Browse All Files
3. Press the green "Download" button
4. Select the appropriate platform and download (See below)

pi4soa2.0.0.CR1-eclipse3.3.2-linux.tar.gz
pi4soa2.0.0.CR1-eclipse3.3.2-macosx.zip
pi4soa2.0.0.CR1-eclipse3.3.2-win32.zip
pi4soa2.0.0.CR1-Release-Note.pdf
pi4soa2.0.0.CR1-withsrc.zip
pi4soa2.0.0.CR1.zip
pi4soa.sar

I also have lots of example which I am happy to share. To get them you will need to email me at pi4tech (steve@pi4tech.com)