Software Development has come a long way indeed. Gone are the days, when developers used to code until the day of roll out, all by themselves, without any second set of eyes, validation, testing etc. Today, we have got the luxury of a variety of independent teams doing uniquely different functions within the life-cycle, be it specification of requirements, Architecture, design, development, testing, implementation, production support etc. Bringing in independent teams has really benefitted the overall software development process in terms of validating the work done at the points of handshake, whether these handshakes points are seamless as in agile or distinct as in waterfall. These validations performed by independent teams within the life-cycle are cumulatively providing a “level of assurance” to the project stakeholders. Today, we have come to a stage where the validation is being performed by specialized testing teams within these independent teams, much of this is warranted with complex architecture and technology advances. Again, this type of validation is preferred to either increase the assurance level or bring it to par as one would expect within a conventional architecture and technology.

The introduction of globally diverse and distributed teams has helped in reducing the go to market time although it has indirectly added to the complexity, specifically when large amount of software testing is performed in an outsourced and off-shored model. Along with the evolution of the SDLC we have also seen the evolution of comparable maturity models like CMMi, TMMi etc., which bring in a level of assurance to the Software development process. These maturity models standardize, benchmark, measure and improve the SDLC process and also rate the processes as managed, defined, quantitatively managed etc. Also, these processes have brought in efficiency, effectiveness and improvement across multiple SDLC functions in an organization. There is no doubt that some of these maturity models have helped in bringing down the costs, improving the on-time delivery, improving the productivity and also in improving the customer satisfaction to an extent. Today, even the SDLC models used for Software development are undergoing changes. Clients today pick and choose from traditional waterfall models, to iterative models, Lean models etc. From a complexity perspective, with the advent of new technologies like mobility, cloud and new ideas like social networking, Digital consumerism etc, software is becoming more and more complex. To add to this mayhem with fraud and international terrorism on the rise regulatory agencies across the spectrum have increased the checks and balance to ensure that the occurrence of these grievous criminal activity are reduced and in some cases prevented.

In today’s scenario, being a CIO is NOT just being on the hot seat! It is like your seat is the BBQ Wild wings oven! It has become absolutely imperative for every CIO to be on his / her toe, to ensure a successful system roll out that gets implemented on time, with the required quality, with in the estimated budget and more importantly, wins customers. If we take a bank’s IT department as an example, at any given time there will be at least 10 important programs going on, aiming at regulatory compliance, efficiency improvement, new technology embracement, leaner and efficient operations etc. It is very important for a CIO to ensure that all of these are on track at any given point in time. With this context, let us look into today’s IT landscape. Baring parts of continental Europe, all the other geographies have outsourced and off shored more than 75% of their software testing process.

Today’s CIO, needs more than just a warm and fuzzy, he needs a great level of assurance that things are on track for all his marquee programs and nothing will go bust before / after the program roll out. Typically, most of these outsourced organizations are considered experts in technology and to an extent, knowledgeable in domain. They do implement maturity models in their organization; get audited by third party auditors on a frequent basis. The traditional maturity models do provide hygiene, when it comes to processes, but It is important to understand the depth to which the maturity model gets into. While, this may be at a holistic level of an outsourced organization, it is very important to assess and understand how it helps each of the projects and programs that are being run for a specific client.​Let us look at some examples: (1) Service Oriented Architecture is used to develop an authoring system, which has a web front end. The testing approach aims at performing the testing in a traditional fashion, i.e. by creating numerous documents using the authoring system by way of manual testing. Is this the right approach? Would the manual method validate all the variations of the web services? Would there be enough time on the schedule to do it manually? These are basic questions that if prompted and answered at the right time would greatly increase the level of assurance. (2) In an equities trading platform, there are 15 different applications that need to work in a sequence. The applications range from applications that capture the order, routing, to processing back office work like archiving.

At any point in time, in an agile development context, all the 15 applications would undergo change so it would be very difficult to acquire a change free environment, so the testing team would have to test under constrained circumstances. Also most of the time the upstream and downstream applications’ versions will be wrong. Again, should constrained mean out of sync application versions? Should there be a process to set-up a proper test environment? Will it lead to a incorrectly validated product being rolled out to production? If this fatal strategic error could be caught and corrected, it would really go a long way to increase the CIO’s assurance on delivery.​(3) In a payments landscape covering 250 applications, a new payment engine is being introduced. The testing team comes out with 3000 test scenarios to test the new system that is being put in place. There is a traceability matrix being maintained by the testing team to prove that every requirement is covered through test cases. End users do not have the time to validate the test cases and it is left to the outsourcing service provider.

How does it give an assurance to the CIO that the right tests are performed to validate the new system? There are many such scenarios that we encounter in our daily life. In light of this, it is pertinent to ask a question here – “Do the traditional maturity models provide enough “Assurance” to a CIO at any given point in time?” “Do they need to be complimented with something that provides more “real” assurance?”

Meanwhile, if you have any thoughts or suggestions or comments, kindly post the same.