OSGi, Modularisation and Micro-services

I have now refreshed my memory on OSGi thanks to Richard Nicholson's comments. Richard is correct of course. Modularisation is a key component of scale too. But in todays world of Docker, Kubernetes, OpenShift and the plethora of other DevOps/CDCI related modules, we only deal with the software and hardware infrastructure and not the behaviour of a solution to a business problem. Now that we have execution venues (PaaS and SaaS) we can focus more of our attention in the complexities of delivering that behaviour over a collection of loosely coupled micro-services. 

For read-only (stateless) we have an internet-scalable pattern that works well, is transparent (that is we can monitor and understand in the business domain with little additional effort) and is flexible against changing need, through DevOps and CDCI pipelining.

But for stateful behaviours we are less equipped and often try to map stateful to stateless idioms to achieve scale, but at the cost of flexibility and transparency. The purpose of scribble is to bridge that gap and make it just as easy to build, if not easier, stateful solutions that are closer to the business domain that they support with no less flexible.

That is to say, that with scribble, we can be deployed just as fast as stateless solutions so we do not slow the velocity of change, we reduce any additional effort to monitor in the business domain by keeping ensuring our stateful interactions are a reflection in the business domain, and enabling rapid recovery in the event of a "modular" container by being able to instantiate on the fly and in a specific state.

From a Total Cost of Ownership (TCO) perspective we reduce the cost of development by removing errors related to stateful interaction and moving SIT into a virtual SIT world, we maintain the same cost of deployment into test and production environments, we reduce the cost of monitoring in the business domain and add further value by being able to measure what we observe against what was expected and we lower the operational costs associated with tenancy through lazy, just-in-time behavioural instantiation.

A Living Chronology on scaling micro-services

I decided to post this because I wanted to give more context to my previous blog entitled "Scaling micro-services with scribble types".

This chronology, which I kick off today, is all about the journey to solving some of the hardest problems in scaling stateful micro-services, of which the first entry in the chronology was that previous post. In the next entries I will chart the progress, describe the challenges along the way and how they get resolved. It so doing I plan to take input from anyone. The blog is one means of gathering suggestions, which if you do give any, I would like to share.

Having looked long and hard at the gap between practitioners and academics in this digital world and having looked at how we try to overcome and improve through lean and agile ways of working, I think that involving as many as I can will lead to better tangible results for some key academic gains in recent years. Foremost of these gains is session types, which derive from Robin Milners pi-calculus.

In effect we take session types, embodied in scribble and use scribble to help us implement just-in-time, behavioural serverless architectures that lower the cost of ownership, to help us write better collections of micro-services which work faster, to help to lower the cost of value delivered whilst increasing the quality of what get delivered, and to help us understand the processes that get enacted by micro-services on our behalf in real-time, which results in lower cost and higher volumes of business transactions.

There may well be many pitfalls ahead but thus far it is all looking good. I look forward to wider collaboration.

Part1: Scaling micro-services with scribble types

Video

"'Types for documentation' is one of the main benefits of types espoused by practitioners (who don't care so much for formal type safety theorems and such). Explicit languages and tools for protocols are indeed needed to obtain analogous benefits in the application domains... like microservices."
Dr Ray Hu, main scribble contributor.

"Types are the leaven of computer programming; they make it digestible."
Robin Milner, Turing award recipient, Fellow of the Royal Society.

"Types" is one of those words that we often use but do not necessarily understand nor appreciate their role. Types are the basis of all computer languages and as type theory has advanced (i.e. type inference) so have languages (i.e. Java 8, Python, Scala, Go). They were the basis of our move to structured programming embodied by languages such as Pascal and Ada, They were the basis of our move towards object oriented language from SmallTalk to Java.

Our notion of types is usually little more than a doff of the cap towards Alan Turing but without having read the paper to understand quite why. And that is all okay. The practitioners continue to take advantage of results and the academics continue to understand more about what types mean and how wide their scope if to both languages and the very process of developing software solutions. Scribble is the embodiment of a new understanding of types and what they might do for us [Benjamin Pierce]. It is a language for defining protocols, by which we mean the ordered interactions that might be considered valid between two or more participants that wish to exchange messages of some sort. We might consider it as a way of describing a bunch of sequence diagrams that illustrate such a conversation.

"Scale" is a funny old term too. We use it in one way when sometimes the receiver thinks of it in another way. To be clear we look at how scribble can be used to scale our ability to consume and deliver change independent of the increasing velocity of change and independent to the increase in the complexity of the set of micro-services we need.

What this means is that it doesn't matter how fast change needs to happen nor how many micro-services we have nor how complex the interconnectivity between the micro-services,  scribble will maintain that pace of change from the first iteration onwards. All of this set in the context of an agile continuous integration and delivery pipeline and a cloud execution venue.

Scribble is a language for defining multi-party chatter between any number of end-points, - or if you prefer the description of the choreography that describes the same between a set of micro-services. We call this a business protocol rather than a choreography but they are one and the same.

We all know by now how micro-services and the use of stateless service idioms can scale to internet size supporting millions of concurrent users all consuming and providing data. We know this in part because of Netflix, Uber and Spotify as well as many other. We also know that the world does not comprise of a stateless world and so our services need to reflect this.

When we cannot reflect back in the same terms as we understand then the complexity we are operating under reduces our ability to generate value through innovation and customer flexibility reduces because it starts to take every more time to integrate what we need with what we have. A simple new product for a retail company, a bank, an insurance company, in fact more-or-less any enterprise need to market, take orders, book sales using the same systems that are used today.

We might say that we will take a vertical slice through an organisation, taking a bit of the existing business functionality to do the market, and account for sales of the product. But this requires integration if we reuse anything and requires re-development if we do not.

We can use bounded contexts to further split things and set off several tracks of an agile train for each bounded context that deals with the product and it's lifecycle and we can find the tracks and manage the integration and development needed to do it all.

We can decide that the bounded context is the product and set of tracks of related micro-services and away we go.

It all sounds fairly methodical, but the reality is that it all starts well, if you are lucky, and seems to be working until the number of micro-services deployed reaches more than the span of control - normally around 7.

We might provide a summarised, aggregated view of what these micro-services are doing that maps to the business idiom so that the business can understand how things are doing in their own terms. But that aggregated is simply masking the underlying complexity that has arisen. Some of the complexity derives from the chatter that exists between micro-services, the choreography.

We can claim to have made it more robust against failure by using horizontal scaling inherent in stateless services.

We can claim to have made it easier to change through loose coupling and asynchronous messaging.

But in truth these mask the problem. We talk about this chatter and the loose coupling as somehow being the solution to the "choreography" but in fact we have not written it down. Rather it is evolved without much constraint and results in incorrect behaviour,  retries in production, hacked fixes and emergency remediation. If we had only written it down so that we could think of it as a type then all of our API's would be typed correctly, we would have an immediate view of how the system is performing against a described choreography - we choose to call it a business protocol.

The problem today is worse when we try to implement what is a stateful in the real world as stateless in the micro-service world. The additional complexity this adds results in the implementation, and so what is measured and monitored, to be semantically distant from the real world stateful behaviour. With nothing written down and existing as tacit knowledge within teams that own the lifecycle of a set of micro-services the risk and complexity it adds as things scale up becomes a significant business burden. If teams have a high attrition rate knowledge leaches away and over time things just become legacy.

We might of course say, but we can throw away micro-services that no longer fit our needs. And of course we can. But when this becomes the dominant idiom the cost starts to bite. The flexibility, our ability to rapidly evolve micro-services also starts to drop as we are forced into ever more testing cycles to see if we can determine what the choreography rules are and need to be.

We do have the benefit of really good CDCI pipelines and highly flexible cloud execution venues. So much of the complexity needed to support software solutions from requirements through development and deployment find use of the DevOps revolution. But the cost is certainly not optimal. Cost models tend to have a residency charge and as micro-services scale you can thing of paying a residency charge for all of them, including all of the replicants needed for horizontal scaling. That mounts up and when you step back and consider the usage of the set of instantiated micro-services amortised over the business transactions you can identify waste in the system. That waste is the set of micro-services that did not get used in a business transaction. That is a lot of waste.

All of these problems go away if we could only write down the choreography, the business protocol, in a way that we can guide our developers to do the right thing as regards micro-service communication from inception through all of the iteration to delivering software an value out of the end of a CDCI agile pipeline. If we could only write down the business protocol in such a way as to ensure the services are doing what they should at runtime and spot any deviation from it. If we could only .... in such a way as to dynamically instantiate only that that we use to eliminate waste.

What I want to do next is show you how we can write down a choreography and make manifest solutions to these problems. It is not all done yet but I hope you get the idea from the video below:

Video Link

Revolutionising Software Quality Assurance

Executive Summary

In this blog we examine current software quality assurance and software delivery methods and look deeper to understand why software defects occur. Understanding the root causes of why software defects occur is really the only assured way to move to methods of software quality assurance and software delivery that guarantee resulting quality increases with attendant software delivery efficiency. Having identified the root causes we go on to map out how we can revolutionise both our expectations of software quality and software delivery by leveraging automation founded on mathematics and engineering practices.

The context

We are increasingly seeing terms such as "Software Quality Assurance", "Total Quality", "Zero Defects" and "CMMI" in our world of IT. From heads of procurement, quality directors and CIOs the clarion cry for better quality can be heard both in the corridors and the in board rooms with suppliers. We even hear it within the software service industry itself as we all try to deliver faster, cheaper with fewer defects higher quality and lower risk of failure.

Over the past half century our software industry has come up with yet another approach to software delivery, from the standard waterfall, to the V-model to agile delivery. And yet we are not happy. Waterfall projects don't seem to reflect the needs of the business fast enough in an ever-changing world, the V-model is similar and agile delivery is challenging our need for scale, visibility and predictability.

We all want to deliver faster, cheaper without compromising quality and yet we share the view of the great and the good that "quality is the pimple on the arse of progress".

In the IT world we do look at how quality and process efficiency can help manage the tension between velocity of delivery and quality of output in a changing world. The move towards using "lean" execution derives from the automotive industry. The adoption of Quality Function Deployment (QFD) and its sister House of Quality (HoQ) into six-sigma, a sign of quality in itself, was a result of that same lean movement from Toyota.

According to a one-time examiner for the institute of quality assurance, applied largely to the aerospace industry. He says "Quality Assurance is essentially about closing the loop between a customer’s requirements and what is delivered.”8, something we often miss. When we test we test against requirements but are they a true reflection of what the customer wants and needs?

In the practice of Software Quality Assurance we often take a leaf out of the standard works on quality engineering, just as we do with lean. But we fail to understand that "both mechanical and electrical/electronic items or systems exhibit variance which does not exist in the digital software arena and much of traditional Quality Control is about that. A piece of code is non-variant. It isn’t right within defined tolerances it is either right or wrong".

We need to truly understand why software defects occur in order to postulate any solution for software quality assurance and software delivery methods that support our desired notion of quality.

The root cause of software defects

This fundamental observation, that cries out to be heard is that software is either right or wrong. This needs to be at the forefront of our desire for software that is delivered faster such that quality is increased and, by extension, customer wants and needs are satisfied. All of the delivery methods and quality methods that get deployed fail to address this fundamental observation and instead rely on people to interpret customer wants and desires in a succession of refinements from customer interviews to wire frames to the written word, the painted screen, into programming languages or configurations into executable software that does stuff.

They say the devil is in the detail and the devil in this detail is always rooted in translations “from one to another”. The more ambiguous they are, the higher the propensity for errors. We say errors rather than defects because “defect” is too softer a word. A defect in metal casting may occur because air bubbles get trapped. In software we only have errors because the software is wrong. In a perfect world, a computer program is just maths and it can be proven right or wrong against a set of axioms. And if it is wrong it can be said to erroneous.

Thus the root cause of software defects is ambiguity that leads to erroneous interpretations, from one level of refinement to another and in doing so has moved away from the customers requirements.

The way forward

If we step back and consider this problem of ambiguity it lies in the fact that the language employed at any level does not have sufficient semantics and structure to support any formal abstraction or indeed refinement.

On the one hand, refinement is the process we enact when we add detail, but in adding detail we want to preserve the semantics. In our complex world the only languages that have rigor and have really precise semantics are mathematically based; Java has an interpreter and a runtime that enforces semantics; UML class diagrams and the translation into Java is based on formal semantics. On the other hand, abstraction is really an ability to pull out some structure from something more detailed; a code review often looks at the structure to see if it matches to some higher-level description. Ideally we want languages to support requirements at different levels so that we can show formally refinement and abstraction. We want the same for design and we already have it for coding.

JT observed, "There are however methods such as mathematical methods that can provide high confidence that the system will be fault free and that is what QA is about". So what mathematical methods can we use? What are the limitations of those methods and will their use give us total confidence or are we missing something?

If we break the SDLC down into the standard phases and examine them one at time, we start to see that the axioms upon which proof relies are themselves statistical in nature.

Those axioms are an expression of a customer’s wants and needs, whereas designs, code and executables have a more formal algebraic relationship that lends itself to proof.

This latter step we take for granted but consider a simple piece of java (or any programming language). The computer on which that code executes does not understand the programming language, it understands only binary configured for it's specific CPU and operating system. We use a compiler to do the translation and we never question its correctness in so doing. Under the covers the compiler is running mini-proof checkers to make type judgments and that ensures the translation is correct.

There is no reason, if we can find the right maths, why we cannot do the same between requirements and design and design and code. And if we could, and hide it all away just as a compiler does, use it to ensure designs express requirements and code expresses designs. The maths we need to leverage, and hide, has to be "Turing Complete" so that it expresses or captures all that can be computed. It has to be capable of dealing with the complexity of integration so that it captures the way in which components and services talk to each other. The latter should be in that integration to ensure, in this modern mobile and cloud age, that we can capture changing connections (i.e. moving from one cell to another in a mobile network or from one cloud to another). It needs to be able to capture the very basics of business transactions and what they might mean and how we might understand them. The Pi-calculus, which netted the late Prof Robin Milner a Turing award, does all of this. We won't go into detail, it is sufficient to know that such a mathematics exists and can be used.

But such algebraic proofs are not sufficient if the axioms upon which they are based are statistical and not absolute. Thus proving an executable implements some code which implements some design which meets some requirements is all well and good if we can say that the requirements are correct, because then we know through proof, not testing, that the executable is correct. But, alas, we can never prove requirements are correct.

If the fundamental tenant of Software Quality is that the software can be said to meet the wants and needs of the customer who requested it then, we need to look deeper into what sort of requirements need to be met and how we know or have confidence in those requirements. If we can do this, then leveraging an algebraic proof from requirements to executable code can reduce time to deliver and reduce the number of errors because we can get rid of the translation errors and quality as a measure of confidence can rise to the same levels we have for our requirements.

We mentioned HoQ before its adoption with QFD in six-sigma. But few six sigma practitioners, even those that are black belts, use either QFD or HoQ. And yet the seeds of confidence in requirements being correct, lay here. Joining up HoQ and the mathematics of Pi-calculus in such a way provides what is needed. In effect we ensure that the confidence levels are high in the requirements being correct and that means the axioms against which more algebraic proofs are made share the same confidence. This contrasts with how we do things today where the confidence levels that requirements are correct is often not very high and is compounded through design, code and test. This is why testing costs so much and why coding is inefficient with a lot of re-work happening through iterations in test and back to coding.

HoQ works through a simple statistical approach that relates stakeholders to requirements and enables a clear alignment, as requirements are refined back to the stakeholders who are impacted by the change and back to the higher-level requirements, business drivers and goals that necessitate change. Combining both cloud technology and HTML5 enables HoQ to be used on tablets such as iPads and Androids as well as on all other devices. This combination of mobility, low set-up costs provides a way to engage stakeholders in a structured discussion in order to tease on requirements, prioritise them based on the differing views of the stakeholders in a balanced way as well as capturing dependencies along the way. This technique brings well-known, statistical confidence to the process at each stage and through the refinement into actionable requirements. It encourages consensus across the stakeholders that increases the confidence in the actionable requirements.

If our actionable requirements are now deemed to be correct at a high balanced confidence level we have confidence in the axioms against which we can use algebraic proof. If we can then show that a design is correct by proof against the requirements we can assert that the confidence in the design is at the same level of confidence as the requirements. Furthermore we can then generate artefacts to drive coding and structure what is coded from that design and we can check that when the coding comes together in systems integration testing is conformant to that design. This provides us with the same level of confidence that what is coded is also correct against the requirements as it can be algebraically checked against the design itself.

As JT points out “The art of standing in the position of the customer or end user is so often missing. For some reason the independence of thought required to forget the processes that stand between requirement and assurance and to close that loop is often missing in both hardware and software.”. We can of course leverage all of this mathematics, automate things that were not automated before but still deliver something that does not meet the customers expectations. This stems from an inability to put ourselves in the position of the customer. To do this we can leverage HoQ, because one of the stakeholders is the customer or end user of what we produce. So at one level we can ensure that their needs are met and prioritised correctly. At another level, if we ensure alignment both statistically and algebraically of those customer and end-user needs we can drive automation in testing and in conformance checking to ensure that the customer and end user needs are shown to me met.

Conclusions

We have looked at the root cause of software defects (errors) and found it to be based on ambiguous communication and the multiple interpretations that result. We have made clear reference to the customer or end-user and looked at how we can put ourselves in their position and so ensure their needs are met. We have taken the postulation that mathematics, can provide a better solution to ridding or minimizing software defects and we have shown how we might do this using statistical methods for requirements and algebraic proofs thereafter. Thus the use of mathematics, postulated by JT, is a reality not a pipe dream but a practical reality that can help revolutionise both software quality assurance and delivery methods as we use automation techniques based on both statistics and hard-core algebra to reduce ambiguity, speed up delivery and increase the quality of the result.

Last but not least, if you hadn't guessed, the ZDLC Platform does exactly what we have presented. And it's not just is that think this, look at Ovum too.

References

1.     John R. Hauser (1993) How Puritan-Bennet used the house of quality. Sloan Management Review, Spring, 61-70.

2.     John R. Hauser & Don Clausing (1988) The house of quality. Harvard Business Review, May–June, 63-73 

3.     Terninko, John; Step-by-Step QFD Customer Driven Product Design; Second Edition, St. Lucie Press, 1997.

4.     Shillito, Larry M.; Advanced QFD Linking Technology to Market and Company Needs; John Wiley & Sons, Inc.; 1994.

5.     Day, Ronald G.; Quality Function Deployment-Linking a Company with its Customers; ASQC Quality Press; 1993.

6.     “House of Quality”, Jennifer Tapke Allyson Muller Greg Johnson Josh Sieck

7.     “Strategic Priorities and Lean Manufacturing Practices in Automotive Suppliers. Ten Years After.”  Juan A. Marin-Garcia and Tomas Bonavia

8.     Private Letter From John Talbot (former examiner for the institute of  Quality Assurance”.

9.     Private copy of “The Book of Kimbleisms”. Richard Kimble.

10.  “The Polyadic pi-Calculus: A Tutorial” Professor Robin Milner, LFCS report ECS-LFCS-91-180, School of Informatics Edinburgh University

We have busy doing cool stuff

For over 3 years since I joined Cognizant we have been busy building out a new platform for the engineering of software. We call it the Zero Deviation Lifecycle and it comprises all of the work I have been doing since about 2000 and combines it with thinking from Dr Bippin Makoond. Visit the ZDLC blog page.

The passing of Kohei Honda. A great scientist and a great man

I met Kohei (and Nobuko) through Alexis Richardson who pioneered work on messaging with AMQP. Alexis was a pi-calculus guy as was I and he knew both Kohei and Nobuko (Kohei’s wife). On the other hand I knew Professor Robin Milner quite well, he was my mentor. That all happened in 2002, some 10 years ago now. Kohei joined W3C along with Nobuko and Robin as invited experts to ensure that WS-CDL, the basis of Testable Architecture, Savara and so many things that we take for granted. All based around pi-calculus for which we all owe Robin Milner a huge debt. But pi was always hard to understand except for the gifted mathematician. Kohei with a fellow researcher, Vasco Vasconselas, came up with an elegant addendum to pi called sessions and so began work on session typing. This work, above all else, informed WS-CDL, Savara, Testable Architecture and many others as to how to use pi-calculus so as to gain practical benefits. Indeed it is the core of WS-CDL in it’s search to understand behaviors over a distributed system.

That Kohei became an invited expert on ISO’s UNIFY project to unify messaging standards for financial information exchange and an invited expert to the Japanese banking system all of which demonstrated that Kohei really understood how academia could support business.

Kohei was a picture of enthusiasm with a vision and a feel for aesthetic qualities in all that he did. This was never more apparent than his work on session types and scribble (his language that has become known as son of WS-CDL) applied to business problems.

I have been so lucky to have had the chance to work with Kohei over a long period. I have been even luckier in being able to introduce him to some other great people of whom Dr Bippin Makoond is one along with Matthew Arnott from the Tsunami early warning project in the USA, to Matthew Rawling CTO at UBS and chair of UNIFY, Dr Gary Brown at Redhat (my long time friend and colleague) to all my old colleagues in W3C. Spreading his enthusiasm and explaining complex things in easy ways was a gift that few of us have but that Kohei had in spades.

I was humble and grateful to know him as a colleague and as a friend. He was gentle with the heart of a giant and the mind of a great scientist. His work will stand the test of time and his presence will, for this soul, be forever there.

I shall miss him greatly and my heart and support goes out to Nobuko and Kohei’s wider family for their loss.

Albertina Sisulu, Who Helped Lead Apartheid Fight, Dies at 92

Some of you may know that I am half South African. Born of an English/Irish father and a Cape Coloured Mother. I grew up in Dover in the days when racism in England was rife and I like many other suffered at the hands of it.

I started working with Anti-Apartheid organisation in the mid-1980's until we gained freedom and democracy in South Africa. In 1988 I was very luck to meet Ma Sisulu. Not too many people manage to silence me without doing anything but Ma Sisulu I was in complete awe of. I also met Lindwe who at that time as studying in the UK and I met Zwelakhe, who was the subject of one of the longest banning orders meted out by the apartheid regime.

Ma Sisulu for me is a true patriot for all that is fair and just. She was and will remain the mother of the Rainbow nation. And I feel honored to have met her and sad to see he passing.

Amandla!

Creativity and Failure

In the history of human endeavour and in particular creativity we strive and often fail. Management metrics are not particularily supportive of failure and this causes problems in fostering innovation.

I am not going to dwell too much on what we need to do but in the first instance take a look at the link below for many a glorious failure.

Glorious Failures

This is what Tom Peter's says:

Education and Creativity

And this is what Steve Johnson says:

Where good ideas come from

Enjoy!

SOA and Cloud Symposium Oct Berlin 2010

Last year in Berlin I had the fortune to speak back to back at the SOA and Cloud Symposium and then at JBoss World.

The SOA and Cloud Symposium released the following recording of the talk. Hope you enjoy it.

Happy to send the ppt to anyone that wants it. Just email me at steve.ross-talbot@cognizant.com.

Cheers

Steve T

The role of metaphor in computational science and innovation in general

Of late I have been doing a lot of work on innovation. In particular how to explain it, how to foster it and how to manage it. Metaphor plays a key role in the process of explaining and describing innovation and how we foster and manage it. And it is to this use of metaphor that I want to further examine.

When we step back and look around at what is going on in the wider world metaphor is all around us. In the case of human endeavor and creativity we often use the term "the lights going on" as a metaphor for the point at which we understand some complex problem enough for a solution to jump out at us. In the field of human conflict we often use the term "collateral damage" as a metaphor for human death through accidental friendly fire and indeed deaths that arise from poorly targeted munitions that result in civilian deaths and casualties. In the world of science Dorkin uses the term "the selfish gene", not to denote any real act of selfishness but as a metaphor for the process of natural selection. And finally in the world of computer science we habitually use the term "bug" as a metaphor for "defect" and we use the term "bus" to denote some mechanism for the transportation of information.

We use metaphor well when it fits properly to some conceptual model of that which we wish to describe and we use it poorly when we introduce ambiguity and sometime simply it is wrong.

My bug bear is the metaphor of the funnel in innovation. According to wikipedia "A funnel is a pipe with a wide, often conical mouth and a narrow stem. It is used to channel liquid or fine-grained substances into containers with a small opening. Without a funnel, spillage would occur." Consider this carefully. You have 100 ideas (early innovative ideas) and you use a funnel. 100 ideas go it and 100 ideas come out. And yet we habitually use this to denote some process of capture, prioritization and selection of ideas. When we get the wrong metaphor to denote a process which needs management it makes it harder to implement the correct process because it focuses out attention in the wrong way and at the wrong things. It might seem small but getting it right is as important as getting names right in descriptions of software systems [Milner - Wot's in a name].

So what would the correct metaphor be for this innovation mechanism? When we think about what we are doing to a stream of ideas and innovations it is easy to get the right one. What we have is a filter and not a funnel because we are not selecting all ideas and innovations to go forward, we are selecting the ones that are appropriate in some way. If we imagine having the usual array of crazy mechanisms for gathering ideas such as drop boxes, inevitably we will get ideas such as "I want more rice pudding in the canteen" along side some worthy ideas. But we would not for a moment imagine running with "rice pudding" as a creative or innovative idea, hence the funnel is wrong and the filter is right.

BTW if anyone want's a copy of the Miler paper let me know. Alas since Robin passed away his paper is not longer online.

Common SOA Gotchas

I get asked a lot what not to do as opposed to what to do in the SOA and wider world. So here is a compilation of some common SOA Gotchas in part they are mine and in part my team. Thanks to them.

• SOA Governance - The three most important things to remember are governance, governance and governance. It doesn’t matter if you design a fantastic technical solution, doesn’t matter if you create the slickest best designed services you have ever seen, doesn’t matter etc. etc. etc. Get the governance right from the start and you will be fine, get it wrong and you are stuffed. Once you have that in place, then take all the other advise given below and elsewhere. When it goes wrong it can be pretty demoralising. Governance in this instance includes all aspects of change and budgetary alignment to support it. In one famous case the technology worked, the services were great but the structure of the governance board itself and the budegtary alignment that supported it were out of kilter leading to a popular service degrading over time because budgets could not support the needed change in scale. Do not concern yourself with functionality alone. The non-functionals become ever more important as services become consumable items in an enterprise wide landscape.
  

• Comopsite Services – Uncontrolled indirection can add complexity to any system. BPEL (process orchestration) is used to create composite services. It is the main re-use mechanism for services in a SOA. However should indirection of composites be uncontrolled then the impact of change at the leaf nodes of services can have a major impact on those that rely on them. It is far better to establish a principle of maximum level of indirection and deal with exceptions to this principle dealt with by the SOA Governance board.
  

• Service Identitication (80/20) – Service Orientation and the tools to support the creation of a SOA enable change to be implemented far faster than traditional approached. In large part this is because of the use of standards, de-coupling and investment by vendors in leveraging standards and taking advantage of de-coupling. Often Service Identification slows down in pursuit of initial perfection where evolution is good enough. Sometimes we call this analysis paralysis and it causes many SO programmes to slow down to a crawl and fail to delivery benefits fast enough at the start. If one considers multiple projects under a SOA enable-ment workstream the first project will have no re-use and the second limited. Reuse will build up over time. It is only by use that we truly understand the utility of services at all levels of the service taxonomy. Rather than get bogged down in the pursuit of perfection it is better to time box the initial projects and implement based on the outcome. When the next project starts as long as the previous project has some budget for re-work SOA tooling is kind in enabling such change to be made with little cost and so services can evolve far easier. The time-boxing should seek to deliver an 80/20 solution to the service descriptions and their supporting data.
  

• SOA Roadmapping - Development of an SOA road map is a key communication tool that can be used to show the business how their overall landscape will change, and most importantly, the time it will take to change. Some peoples perception is that once you are an SOA road all future developments will take 10 mins!! Clear communication of the benefits which can be presented to the key stakeholders. When implementing SOA you get situations where the front end applications do not change, and this is what the business sees, they are sometimes not aware of the benefits. Its like replacing an engine within a car, there is no change for the driver.
  

• Data. When you start looking into the information model to support a SOA it is important to contextualise the data and its use. Few projects or programmes are successful when a huber-XMLSchema is imposed. It is far better to recognise the economic, political and social boundaries in which services need to exist and reflect the data in a similar way. At the same time it is important when having a multiple schema approach to use some design principles in ensuring that multiple XMLSchema’s do not cause a huge governance overhead. The use of polymorphism to incorporate differing needs has had a great deal of success (see FpML). Does think that Schema validation alone gets you quality data, look into other tools that can do cross attribute checking (schematron and the use of rules engines and RuleML). Don’t forget about identity and provenance. Knowing where data came from *the services it has traveled through) and having string identity leads to a faster diagnosis of problems and a far faster auditing (something no one wants to take anytime at all).

The passing of Robin Milner

I heard yesterday that Robin Milner died on the 20th March. The passing of giant. I can say little more. My blog is undeniably my small humble tribute to a man who taught me so so much. I am feel so very sad that both he and Lucy are with us no more but very privileged to have known both of them.

Robin and Lucy thank you for just being who you were. May we be as blessed with your humility and wisdom in our lives.

SOA Manifesto comments

There have been over 600 signatories to the SOA Manifesto as I write. I do surf and look at what people have to say. Some are +ve some are -ve. The -ve's are loud to say the least but offer little in terms of concrete suggestions. Which in some cases is a great pity because some of the voices may have something to offer but don't do so.

Anyway I thought I'd blog a while on my thoughts on the -ve's as a generic thing rather than dealing with specifics. One of the complaints seems to stem from a violent reaction against SOA and often it is confused with REST over SOAP arguments.

The REST over SOAP argument is irrelevant. An instance of a SOA can use either, could use both or indeed neither.

The violent reaction against SOA I cannot quite understand. One of the key aspects of any instance of a SOA is some notion of a Service Contract. I don't claim WSDL is a good way of doing it but it is a way. There may well be other descriptions that are better, capture more and so allow us to reason over them. Are Services Contracts a good thing? For those that think not I'd like to understand why. For those that think they are, why are they a good thing?

From my perspective without a Service Contract (ideally more than is available in WSDL) we cannot understand "type". And "types" are what make programming languages effective. The use of "types" in composition is not confined to programming languages which manipulate "types" and in which a complier checks "types" which leads to less errors. Indeed "types" in a service composition are surely a useful thing for all the same reasons that "types" are part of any good programming language.

Now if an instance of a SOA is dominated by the presence of a Service Contract (and therefore some notion of "type") why is it a bad thing?

I cannot find a reason. Perhaps you readers can and if so feel free to post. I will publish all.

SOA Manifesto rambilings part 4 (what else?)

The Manifesto does not tackle any relationship between PROCESS ORIENTATION and SERVICE ORIENTATION. In this blog I'll tackle this link and explain why they go hand in hand. The only reason this was not done in the context of the Manifesto was time and focus.

So here is my take on why it is in fact important and perhaps one of the most important aspects to SERVICE ORIENTATION and SERVICE ORIENTATED ARCHITECTURE.

PROCESS ORIENTATION is a paradigm that constrains a solution to focus on the underlying flows between components where a component maybe a person or an agent for a person (e.g. a portal) or a software component (which maybe embodied by a SERVICE CONTRACT).

BUSINESS PROCESS MANAGEMENT (BPM) is the management of PROCESSES that are formulated by applying PROCESS ORIENTATION. A BUSINESS PROCESS MANAGEMENT SYSTEM is a software platform that supports the construction of PROCESS DESCRIPTIONS, their simulation, execution and life-cycle management.

ORCHESTRATION (as embodied by WS-BPEL) is a mechanism for describing, simulating and executing COMPOSITE SERVICES which are “orchestrated” by means of a some flow between them to deliver a new SERVICE.

Today BPM and ORCHESTRATION are common bed-fellows. Many companies use the former to deliver the person based workflows and the latter for composing complementary SERVICES, which may then be consumed by a BPM layer. In the future these two bed-fellows will merge with the introduction of WS-HumanTask and Bpel4People. These two related standards will deliver the necessary descriptions to enable both BPM and ORCHESTRATION to be merged.

When we think of BPM and SERVICE ORIENTED ARCHITECTURE we need to understand where things are today and where they may be tomorrow and deal with the present and plan for the future. Typically a BPM SERVICE ORIENTED ARCHITECTURE approach focuses on the encoding of PROCESSES at the business level so that legacy applications involved in the PROCESS are SERVICE ENABLED along the way. The reason to do the SERVICE ENABLEMENT is that it is a cheaper and faster route to leveraging legacy application within an overall BPM context. This is the case because of the huge technology investment in tools to SERVICE ENABLE and in BPM to leveraging SERVICES.

Successful BPM and SERVICE ORIENTED ARCHITECTURE projects (together or separately) are all dominated by a clear view of business value to be delivered. They are often preceded by a Proof of Concept which is constrained and focuses on key integration aspects (for SERVICE ENABLEMENT) and key PROCESS aspects (which manifest business value).

Key to business value is the identification of value based on a clear understanding of the vision and mission of an enterprise coupled with strategic goals. This is the case because the strategic goals, which are derived from the vision and mission, provide clear guidance as to where the value lies. Identification of the strategic goals and the Key Performance Indicators that go with them enables a project to be framed that has clear business value and so when delivered can be measured to determine the value delivered. The same Key Performance Indicators and their derivatives may then be used to monitor and measure the running PROCESSES and their attendant SERVICES and the delivered flexibility provides the means by which the solution can be tuned and adjusted. This ensures that the run-time governance of the solution always aligns the solution to the business goals through their expression in the Key Performance Indicators.

SOA Manifesto rambilings part 3 (the principles)

Principles

The principles are high level principles. They do not deal with design or organizational issues but may allude to them. They are part of the manifesto because they act as a guide in how one should approach SERVICE ORIENTATION and how one should apply the principles of SERVICE ORIENTATION. Following these principles will help to ensure that whatever gets delivered provides the necessary short and long term business value to justify its existence.

“Respect the social and power structure of the organization.”

At an enterprise level SERVICE ORIENTATION and a SERVICE ORIENTED ARCHITECTURE provide a more open mechanism for accessing capabilities through their SERVICE CONTRACTS. It makes it easier to reuse existing SERVICES and create new ones in line with business needs. What is very important in looking at this from an enterprise perspective is who owns the SERVICE, who has the budget and how do we cost account for the use of a SERVICE. Respecting the social and power structures of an organization is critical in understanding the economic effect of a SERVICE, how it is used and how it can continue to be used over time. Aligning budgets for SERVICES and aligning the use of SERVICES to some cost accounting mechanism requires that budgets are aligned to the power structures and that the SERVICE is cost accounted for, based on the social and economic power structures within an organization. Getting this wrong can result in SERVICE degradation over time as more people use it and budgets for its subsequent maintenance and operational use fail to reflect it use and importance.

“Recognize that SERVICE ORIENTED ARCHITECTURE ultimately demands change on many levels.”

This is a corollary of the previous principle. Adopting SERVICE ORIENTATION and implementing and using a SERVICE ORIENTED ARCHITECTURE requires alignment through the organization. SERVICES are used, SERVICES change, and SERVICES are composed, all of this can happen across the entire organization. Use happens from all parts of the organization, request to change comes from all parts of the organization and composition of new SERVICES may happen anywhere in the organization and may use SERVICES from other parts of the organization. This is why SERVICE ORIENTED ARCHITECTURE demands change at many levels. The capabilities that SERVICES expose become un-siloed and that makes them enterprise assets and not just assets for one business unit. This democratization of capability requires change on all levels of an organization as users express future desires driven by changing needs and as the implementation of a SERVICE ORIENTED ARCHITECTURE makes flexibility easier to achieve.

“The scope of SERVICE ORIENTED ARCHITECTURE adoption can vary. Keep efforts manageable and within meaningful boundaries.”

This is a principle derived from the Evolutionary value statement. It is far better to aspire to the benefits of a SERVICE ORIENTED ARCHITECTURE but realize them incrementally. Typically one might start with the portalisation (portal-led SOA) of some business unit and SERVICE ENABLE along the way. The additional cost for SERVICE ENABLEMENT is mitigated by a standards based approach to the integration of the underlying capabilities with the needs of the portalisation. What we see in Cognizant is that this approach is complemented by additional concurrent but offset projects that then seek to further SERVICE ENABLE along the way as the portalisation efforts move to another business unit. Portalisation is not the only way to start. One might have specific needs to reuse components or applications simply to make available data and process from several application but to do so in a more flexible way. SERVICE ORIENTED ARCHITECTURE adoption starts by making it manageable within meaningful boundaries and so can be effectively measured in terms of the business value that it yields. Never forget Return on Investment, it pushes and pulls development.

“Products and standards alone will neither give you SERVICE ORIENTED ARCHITECTURE nor apply the SERVICE ORIENTATION paradigm for you.”

Products and standards do not apply SERVICE ORIENTATION to deliver a SERVICE ORIENTED ARCHITECTURE but they may help. It is people who wield them, making sure you have the right skills to get the most out of a SERVICE ORIENTATION approach and get the most out of a SERVICE ORIENTED ARCHITECTURE is a key aspect of any SERVICE ORIENTED ARCHITECTURE program regardless of how it is constructed or the boundaries in which it fits. In Cognizant we specialize in matching the right resources to achieve a positive outcome and the role of the Enterprise Architect becomes a key to ensuring that all of the moving parts of the Software Development Lifecyle (SDLC) work together.

“SERVICE ORIENTED ARCHITECTURE can be realized through a variety of technologies and standards.”

A SERVICE ORIENTED ARCHITECTURE does not have to rely on an ESB, WSDL, BPEL or any other standard or technology. The key aspects of SERVICE ORIENTATION and SERVICE ORIENTED ARCHITECTURE are the descriptions that we call SERVICE CONTRACTS and the decoupling of them from the implementation of their capabilities at both design and runtime. An instance of an SERVICE ORIENTED ARCHITECTURE may use asynchronous messaging and pass documents between SERVICES. An instance may use WSDL invocations with some form of remote procedure call. An instance may use BPEL and it may use business rules. In all cases the de-coupling should be self evident. A yard stick for an instance of a SERVICE ORIENTED ARCHITECTURE is to locate a SERVICE REPOSITORY, because without one, de-coupling invocation on a SERVICE CONTRACT from a SERVICE IMPLEMENTATION becomes impossible to achieve in any uniform way.

“Establish a uniform set of enterprise standards and policies based on industry, de facto, and community standards.”

A bit of motherhood and apple pie but something that the can certainly catch you if you don’t do it. Establishing a uniform set of enterprise standards and policies can reduce the risk of mis-delivery of a solution. Where there are industry standards they often leverage XML as a means of encoding information and it is better to use them than not. It can cut down integration costs and reduce the time it takes to deliver an enterprise information model as well as the processes that manage that information. The enterprise standards should cover all aspects of governance and should support rapid change. The policies should guide security choices, transactions, and service levels. One key relationship between a SERVICE ORIENTED ARCHITECTURE and this principle is that many industry and de-facto standards are seeking to use the same SERVICE ORIENTED approach to describing the processes that underpin standards, this is true in International Standards Organization, the International Swaps and Derivatives Association, Healthcare Level 7 and many others. For those SERIVICES that are outward facing this may mean that the SERVICE CONTRACTS are already defined which can help reduce the cost of external connectivity and provide cross enterprise interoperability.

“Pursue uniformity on the outside while allowing diversity on the inside.”

This is a cry for SERVICE CONTRACTS and process uniformity on the boundaries of the enterprise and diversity as to how you achieve implementation on the inside. SERVICE CONTRACTS mandate a degree of uniformity. We can go further by publishing our external boundaries in terms of the SERVICE CONTRACTS we wish other to see and the ordering rules over those contracts that make up the processes that we wish to make visible. In so doing we ensure a higher degree of interoperability.

“Identify services through collaboration with business and technology stake-holders.”

From a technology platform perspective it is important to identify core TECHNOLOGY SERVICES. These will have the highest reuse rating and you can only do this by understanding the technical requirements from the technology stake-holders. Likewise the only way to gain reuse at the business level for BUSINESS SERVICES is to do it with the business stake-holders. The widest collaboration, across business units will yield the widest reuse, even if you have a single project, it is worth gathering requirements from a wider set of stockholders so that you can plan effective reuse and change management over time. Hence “Recognize that SERVICE ORIENTED ARCHITECTURE ultimately demands change on many levels” and “Identify services through collaboration with business and technology stake-holders” are complementary.

“Maximize SERVICE usage by considering the current and future scope of utilization.”

Wider collaboration for effective SERVICE IDENTIFICATION is part of the process of understanding the usage of both current and future SERVICES and their utilization. Don’t just look at the functional aspects because the scalability will be compromised as usage rises. Taking into account the scalability issues mitigates the risk of retrenchment and ultimately failure of large scale enterprise adoption.

“Verify that SERVICES satisfy business requirements and goals.”

This principle may well seem obvious, but it has profound consequences. One of the things that SERVICE ORIENTATION and SERVICE ORIENTED ARCHITECTURE instances provide is a clean separation of SERVICE CONTRACTS from SERVICE IMPLEMENTATION. It matters not if you use WSDL to do this or SOAML or Abstract BPEL, what is important is that before a SERVICE is constructed (implemented) we can, at worst, manually check that the collection of SERVICE CONTRACTS match the requirements that gave rise to them. If these requirements are themselves derived from higher level business requirements we can check the derivation. And if the higher level business requirements are derived from business goals we can check their alignment too.

When we do this manually, and most good SERVICE ORIENTED ARCHITECTURE practitioner do exactly this, we call it SERVICE ORIENTED ARCHITECTURE GOVERNANCE and it is part of Enterprise Architecture Governance. It is part of the enterprise alignment process that is done manually.

Given that SERVICE CONTRACTS can be properly described in a formal and structured way (i.e. WSDL , SOAML and Abstract BPEL) and given that we can encode the requirements that give rise to these SERVICE CONTRACTS, it would make a lot of sense to get computers to check the SERVICE REQUIREMENTS against SERVICE CONTRACTS and automatically show that the "SERVICES satisfy their business requirements". We are not suggesting at this point that we can do the "satisfiability" test through the different levels of requirements or indeed the business requirements against the business goals - although even this is possible. What we are suggesting is that the immediate requirements (SERVICE REQUIREMENTS) that give rise to the SERVICE CONTRACTS can be automatically checked to show that the SERVICES meet their SERVICE REQUIREMENTS.

This is what Testable Architecture is all about and the wider "satisfiability" is what Savara will deliver over time.

“Evolve services and their organization in response to real use.”

Accept that we live with imperfection but strive to attain it. Accept that things change. In this way we can focus on evolution to ensure that SERVICES meet their business requirements as those same requirements evolve over time. Equally we may choose to re-organize services for the same reasons. We may choose to uncouple a composite or to change it to reflect changes to real use. The best user interfaces are always tested by users and Apple have been very much the masters of getting it right. Doing the same for SERVICES from the top down, the middle out and the bottom up by evaluating their use in real situations will reveal improvements on design, improvements to requirements and manifest improvements with continual alignment to business requirements and goals.

“Separate the different aspects of a system that change at different rates.”

Engineering for change is not easy. It requires an understanding of what is needed today against what might be needed tomorrow. A key metric is how executive management and in turn their immediate management team steer the ship. What are the controls and the points of change that they need to do their job. Understanding industry influences and trends on an enterprise and understanding the vision provide a way of scoping what needs to be made flexible in the future. Understanding the mission and strategic goals provide a scoping of the changes needed in the near term. Having effective change management procedures in place and effective change request processes are imperative to continual improvement.

Another key aspect of rates of change is that processes change infrequently, but they do change, but the decision point, we might think of them as business rules, change far more frequently. Hence outboarding business rules and using a BUSINESS RULES SERVICE is an important architectural pattern which complements outboarding of processes through orchestration and workflow.

Don’t think that what you deliver will not change or is optimal. Let the users use and make suggestions. If you get flexibility engineered in from the start then the cost of suggested change will reduce and the ability of the enterprise to optimize how it does things increases.

“Reduce implicit dependencies and publish all external dependencies to increase robustness and reduce the impact of change.”

When we create COMPOSITE SERVICES, just like when we create views in a database, it is important to understand the SERVICES that they derive from. Without documenting and making these dependencies clear we potentially run into huge change management and governance problems. It should be standard best practice to publish the key dependencies both in terms of the services that may be used and the underlying technology dependencies too. Such dependencies should be documented and published as some form of attachment to services in the SERVICE REPOSITORY.

“At every level of abstraction, organize each service around a cohesive and manageable unit of functionality.”

Good design coalesces concepts into some form of cohesion. This is true for any piece of software that is constructed. SERVICES are no different in this regard and at each level of abstraction be it a BUSINESS SERVICE, a TECHNICAL SERVICE or a DATA SERVICE the principle of cohesion applies. Organizing and making things manageable takes effort and discipline but the rewards are often greater use and reuse because the focus of the SERVICE becomes clear and the concepts coalesce to deliver appropriate functionality for the level of abstraction.

SOA Manifesto rambilings part 2 (the value statements)

The value statements reflect a desired prioritization of one value over another. It does not mean that the value that has a lower priority is not important it just means that the value with higher priority is seen as more important. The value statements are there to help prioritize key decisions. For example having a technical strategy with no business value makes little sense. But a technical strategy to support business value makes huge sense. Equally having project specific goals make sense but not at the cost of strategic goals. Should the need arise to support project specific goals over strategic goals then the decisions should be made with our eyes wide open.

“Business value over technical strategy”

SO and SOA is not about technology. One may use technology to deliver it. The value is the value that the business identifies or agrees to. In the world of Enterprise Architecture and Solutions Architecture we measure the outcome of deliverables against agreed business expectations. The technical strategy that is employed is created based upon what we need to focus on technically to deliver business value. Many projects fail because they loose sight of the alignment with business imperative. Ensuring alignment to business value is the key to good governance.

“Strategic goals over project-specific benefits”

In the world of effective Enterprise Architecture applied to a business landscape, we seek to understand the vision of the enterprise and we seek to understand the mission statement. The vision is aspirational and forward looking, the mission is actionable and focused on the immediacy of the enterprise. The vision and mission are used to frame strategic goals for the enterprise. The strategic goals might be implemented by one or more programs of work within the IT space of an enterprise. These programs might incorporate many projects. One of the roles of an Enterprise Architect is to ensure that the programs align to the strategic goals and that the projects reflect them. Without the transparency and traceability of strategic goals to the vision and mission, projects cannot expect to deliver against them. Drift is a common concern and when siloed projects drift away from the strategic goals and focus on short term issues at the expense of the bigger picture. The result is a more costly solution as it has to be re-factored and/or re-built sooner than was anticipated because it failed to meet the strategic goals.

“Intrinsic interoperability over custom integration”

The easier and faster it is to compose new capabilities the more cost effective IT becomes in supporting the business and its changing needs. If we always have to place a mediator between services to reuse them then we are falling into the trap of custom integration and what this tells us is that the design of our services does not support intrinsic interoperability. From a principle perspective, moving to a more document centric approach encourages greater interoperability but compromises the ease of use or understanding of a service's capabilities. Probably no where else in designing services is it more important to get the functions and the types of the parameters correct and indeed to be unafraid of re-factoring over time to move towards greater intrinsic interoperability.

“Shared services over specific-purpose implementations”

The granularity of services in an implementation of an instance of a SOA is a common cause for concern. Reuse and therefore the sharing of capabilities through the embodiment of services and their contracts seems to be much harder than anticipated. Several methodologies have been espoused to help. What we all know instinctvely is it is never a good idea to re-develop the wheel. But of course we constantly re-develop the car. Cars are created through the composition of many existing components. The new Beetle makes use of the Golf chassis, but it feels like a very different car. Services as components for the construction of new capabilities is a key principle of SO but doing it is harder that saying it. What is needed is good component repositories with good documentation and reviews. Back in the 1980’s Shlaer and Mellor had the same idea but expressed it as a book of components, like those that electrical/electronic engineers might use, for things we use to create new capabilities. Sometimes a customer implementation is needed but it is not that often and measurements of customer implementation vs reuse should always be made on a regular basis to ensure that the benefits of shared services are being realised.

“Flexibility over optimization”

There is an old software proverb that rings true. Make it work before making it fast. Making things flexible, choosing where the points of flexibility need to be is not easy but by following some simple principles we can at least move much of what is needed outside of code making services configurable. Business rules can be used in place of coded logic, orchestration can be used in place of hardwired configuration. A heuristic approach that might be in a document for service implementers would be to ensure that business rules and processes stated in requirements are exposed to enable them to be changed.

“Evolutionary refinement over pursuit of initial perfection”

The old adage of don’t try to boil the ocean comes to mind. We always seek perfection but we always need to temper it with the realisation that perfection is a focus not a goal. Good software projects and programs are phased. Occasionally we have the big bang approach but they rarely succeed. So we need to be mindful that we cannot do it all in one go and so phase the approach. It is common for many SO projects and programs to be led by the user interface inwards. Service enabling as you go is a better approach than trying to service enable and then figure out how users interact with the service. This is the case because without the users we cannot be sure of the pain point the wishes and desires of those users and so cannot address their concerns. And it is the users (customers) that matter.

---