Chapter 2: First Cooperate and then seek to Collaborate

The priorities of the organisation, and thus the IT Department, is to be able to deliver as much of what was requested for by the Business Units so that the organisation would stay competitive in the marketplace. So that these organisational strategies does not get exposed in the market earlier than envisaged by the organisation, the details of the planned products is only available with the Business Units and with the Service Delivery Teams. The Service Delivery Team is the part of our IT Department that interfaces with the Business Units for the requirements. Based on the requirements gathered by the Service Delivery Team, the IT Department develops the products and IT QA tests the same and certifies them for IT Operations to release to Production.

I realised very early that IT QA would not have the details of the products and thus the priorities of the projects, usually, in early stages of the project. It was incorrect for us to try to extrapolate these priorities based on data available with us, as this would be subject to many assumptions. However, prioritization is a basic requirement to be able to handle multiple tasks. Added to this was that we were handling more than 120 projects per month, besides regular weekly regression runs and testing for production bugs. 60% of our projects are launching of promotions and thus do not take more than 2 to 3 days to develop. However, testing for these changes in the code and/or configurations by the IT Development does not imply that we could compute testing time proportionately as is done in any Software Development House. All through my career in Software Development, we had benchmark that we would devote between 15% to 25% time of the project on testing of the product. However, such benchmarks did not apply in our environment. This was because for even a small change, we have to test the complete system across all potential transactions.

For example, consider a very simple case of launching a new package in the market. This would imply that we customise and/or configure the Siebel CRM and e-Portal and mGate and BMS (an alternate Customer Care System) so that the new package could be sold. This may or may not involve changes in the Mediation. The ability to charge and bill for the same could be accomplished by configuration and/or some customisation in the Oracle BRM system and the Ericsson IN. The credit policies in most cases could be implemented by configuration and/or some customisation in the Credit Control System. The process orchestration can be accomplished by some customisation and mostly configuration in the EAI layer. There could be some changes to other systems as well. All of the above are development activities are accomplished by different product specialist teams of Mobily IT working in parallel.

However, all the changes need undergoing tests by a single IT QA team. To test for a new package, it requires to be checked whether the new package can be sold as planned through the available front-end channels collecting payments from all possible payment channels. Once this package was sold, whether the subscriber was getting enabled in the network so that he/she could use our services. Once the subscriber starts using the services, whether our system was able to charge for the usage (there are usually various scenarios of usage like on-net calls, off-net calls, international calls, data download/upload volume, etc.) as per the terms and conditions of the package. Whether it is possible for the system to switch customers from other associated packages to this new package. Or whether the system allows subscriber to switch from this new package to any other associated packages. Whether the system applied the correct charges for when subscribers on this new package applied for services like SIM Replacement or Number Replacement or Transfer of Ownership. Whether the system was able to produce the bills correctly for subscribers on this new package. Whether the system was able to bar a subscriber if he/she did not make the appropriate payments. Whether the system was able to automatically renew subscription and/or reconnect subscription when the subscriber made a payment. Whether the system was able to disconnect a subscriber, who wanted to close his/her connection. And there are many more test cases like whether the revenues from the new package are being booked under the correct General Ledger codes, etc.

So, even a small development effort translates to a large scope of work for IT QA. Usually, for projects in a Telecom Operator environment, the effort from IT QA is more than the joint effort of all the IT Development units. IT QA cannot allow any product to be launched till all possible business scenarios are adequately tested so that the chances of failure of any transaction for the product is minimised in the marketplace.

It is a general rule in any software development life cycle that the most compromised phase in any product launch is the testing phase. This is because testing is always the last phase in the product development life cycle and thus all delays from the earlier phases needs neutralising in the testing phase. And for a Telecom Operator, not being able to launch a product on time is the biggest sin as it upsets other plans like advertisement campaigns, etc. besides the possibility of loss of opportunity.

I found that IT QA was involved in testing of the products after IT Development Team completed the development. Further, IT QA was involved in assisting the User Acceptance Testing of the product. User Acceptance Testing (UAT) is conducted by the Business Users to make sure that the product is developed as per their requirement. And after the product goes live, IT QA has to test the product on the Production Environment to verify whether it has been deployed correctly and is behaving as required.

I found that the IT QA team was not adequately staffed to be able to handle the demand from the department. Further, the fact that IT QA was performing UAT and Production Verification was not considered as a part of the effort of IT QA. However, all departments of IT blamed IT QA for any delay in launch of a product and/or when any defect was found in the product once launched. There was no way I could deny handling the workload, as it would upset the complete business process. So, the first requirement was to have adequate capacity in IT QA to be able to handle the demand. In Dec11, I was lucky to receive the roadmap for planned products in 2012 from the Business Streams (2 out of 4 to be precise). The Business Streams that provided their roadmap were the Consumer Business Unit and the Corporate Business Unit. The road map for e-Marketing and Transformation was not available. However, Transformation was not a big problem as I was able to find out the acquisition plan for the year 2012. Using these roadmaps, I calculated the capacity requirement for the Consumer Projects and Corporate Projects taking into consideration that IT QA would be conducting Project Testing, UAT and Production Verification.

I was told that generally projects were not conducted as per the road map provided at the beginning of the year as business priorities changed often based on activities at the competitor organisations and other changes in the market. However, I could not add anything called buffer in my calculations, as it would be rejected. So, I asked all my Managers in IT QA to give the level of efficiency of the IT QA Team Members on a scale of 0 to 100 (The planning department was also seeking this information at the same point of time). From the efficiency levels provided by the IT QA Managers, I computed that the average efficiency level of the department was about 70%. I reduced this to 60% and decreased the available capacity by multiplying by this factor. So, I had obtained 10% buffer. I planned that I would increase this buffer by increasing the efficiency level of team members.

In the meanwhile, I received the plan for the 1st quarter for e-Marketing. I used this data to calculate the capacity for the year by extrapolation. The plan for Transformation projects was made on assumptions available for planned acquisition projects. I added the requirement for enlarging the team for Regression Testing and proposed a separate team for testing for Production Bug Fixes. The IT Operations was extremely happy at the proposal for setting up a separate team for testing for Production Bug Fixes as they could now get exclusive attention from IT QA. This was beneficial for IT QA as well because now the IT QA Team Members assigned for project testing would not have to switch between testing for projects and production bug fixes. Production Bug Fix testing always has higher priority and giving these attention delays the schedule for project testing.

I presented the calculations to the management. I did not have to struggle as the same was approved. The one supporting reason could have been that the budget required for the additional resources was within the budget for the department. I managed by utilising the budget allocated for procurement of tools for our department in procuring additional resources.

Having the procurement plan approved was relatively very easy. Now, the bigger challenge was to get the needed resources from the market within a short period. I found that we had frame contract with 5 vendors. So, I approached 3 of them for supply of the resources. Ejada could only supply on site resources, as it was an organisation based in Riyadh. Maverick and Wipro could offer resources both on site and offshore. Both Maverick and Wipro started sending in candidate profiles and we started interviewing the candidates. I found that we were rejecting most of the candidates. I realised that at this rate it will take an enormous amount of time to get so many candidates. Wipro made a very good proposal.

Wipro proposed that they would supply the needed number of resources. If we agreed to accept the resources without any interview, we could use the resources on the floor for 1 month. At the end of 1 month, we could decide whether the candidate was suitable or not. If not suitable, they would supply a replacement for the resource and would not charge for the duration. Otherwise, the resource would continue and Wipro would bill for the resource from the start of the engagement. I liked the proposal and accepted the same. By beginning of Mar12, we had most of the required resources. Now IT QA was the single largest team in Mobily IT. IT Development and IT Operations were larger department than us. However, they had many teams like IT Development had teams for Siebel, BRM, EAI, etc.

Seeing the flood of Wipro resources being inducted, Maverick Senior Management came to meet me. They were extremely upset with me that they could not supply the resources and had a major loss of opportunity. However, the fact was that when Wipro made the proposal, I was not sure whether they could offer so many resources under that agreement and had thus extended the same proposal to Maverick. Maverick had refused the proposal as they told me that they could not take such a risk being a small organisation.

We were managing the demand by working lot of extra hours. Even after having the resources, the pressure did not reduce, as the new resources needed time to become useful for the purpose. Intense pressure continued on the team till Apr12. However, we did not fail to meet any request made to the IT QA department. This was helped because from Feb12, we adopted risk based testing. We announced to the IT Department that IT QA would prepare test plans for each project and assess the risk eliminated by each test case. IT QA would arrange the test cases in the descending order of risk elimination potential. Then IT QA would test starting with the test case eliminating greatest risk. IT QA would test as many test cases as possible within the allocated amount of time and report what was the risk eliminated from the project by IT QA. The Business Unit would have to decide whether they would like to go ahead and launch the product under that circumstance or give more time for testing the product. If IT QA required testing more, then other projects would have to be reprioritised. We will discuss Risk Based Testing strategy adopted by us in more details in later chapter.

Another major issue we faced was that the requirement specifications were not provided under most circumstances. Even if they were provided, the quality of requirement specification was very poor. I spoke to the Service Delivery Team and found out that they did not have enough number of Business Analysts. This resulted in no requirement specification or scanty requirement specification. There was no point in lamenting this fact, as it was a fact of life. I started a process in IT QA to record what are the essential documents, required by IT QA as an entry criteria, and were actually available with IT QA for every project. I found that for majority of the projects, most of the essential documents were not supplied to IT QA. I started preparing a weekly report of this document availability and started sending to all functional head and the head of IT Development. Everyone called this report as the RED report as most of the area of the report was red in colour. After about 2 week, the Service Delivery Manager for Consumer Projects thanked me for sending the report. I was surprised. He told me that he had used my report and made a case to the Management for providing him added Business Analysts and this was now approved. Once we started circulating the report on a weekly basis, we started seeing gradual increase in availability of essential documents.

We started another initiative. We created a checklist of essential sections needed being available in any of the documents that was supplied to us. Service Delivery Team liked this idea very much and agreed to comply with our evaluation of the artefacts. They wanted to create a similar template for evaluating the document quality of the documents they received from the Business Units. We included the analysis of document quality as well. So, now we were reporting what was the amount of documents we were receiving and what was the quality of the documents we were receiving. These reports helped everyone in the IT Department to appreciate the constraints under which the IT QA Department was working and thus they were more kind to us. Also, we saw a gradual increase in quantity of documents and improvement in quality of the documents.

In May12, the Service Delivery Team requested if we could help the IT Development during the Integration Testing as well. As a process, the project is delivered to IT QA once IT Development has completed the Integration Testing and has provided the results of the Happy Scenario Test Cases. However, Service Delivery Team found that there was a bottleneck in the process as the functional teams did not have the full perspective of the end-to-end process of the transactions and thus were having difficulty in conducting essential tests during the Integration Testing phase. I discussed with the Managers and we agreed that we would support the process by assigning an expert who would guide the IT Development Team members in conducting essential tests during Integration Testing. This investment was good for IT QA as the quality of tests performed during the Integration Testing was better quality. Also, now we had the chance to catch defects during the Integration Testing phase. This reduced the efforts during the QA Testing phase, as there was lesser number of issues.

In Mar12, we faced the first escalation that our products for charging data usage in UAE and Egypt at local rates were charging this usage at roaming rates. We analysed our test results and found that our tests results had found these charges to calculate correctly. On further analysis, we found that this had resulted due to a release that was made without QA approval. The incidence happened between 2 regression cycles and thus Regression Testing could not catch the error. So, IT QA could not be blamed for the same. However, we announced to the IT Operations that they need informing us when a release was made and IT QA would do Production Verification for each release moments after the release.

We allowed total control of the IT QA team to the Service Delivery Team. They could decide which project to test. They could call any member of the IT QA Team at any time of the day for conducting any tests. Our team being distributed in Riyadh and Bangalore could offer services 7 days a week as we followed different workweek in the 2 locations. However, majority of us worked on most of the weekdays at both locations.

There were a few instances when the Service Delivery bypassed IT QA and launched products in production without informing us. There were also occasions when they requested me to approve a project release, as it was very critical for business. I approved a few such releases keeping note in the approval system that the project was not tested by QA. IT Operations was very upset with me for this and called me a very lenient QA Manager. However, the fact was that we did not want to be bottleneck in the business process and when we allowed such release we silently assessed the level of risk of such releases and/or finding out what was the level of tests that had been conducted and by whom. Only after being reasonably confident that nothing major could go wrong we approved such releases.

The result of our strategy has been as follows.

  • There has been no escalation where IT QA has been seen as a bottleneck in 2012 till date. In fact, the Head of IT Development declared that IT QA was the only team in IT, which did not have any capacity crisis.
  • There have been 11 instances so far in 2012 where our system has broken down due to bug in the delivered projects. On these 11 occasions, it was established that the error was not due to any lax in the testing conducted by IT QA. Each time, it was established that the breakdown was due to unauthorised and/or emergency releases. IT QA cannot absolve itself of blame for these cases as IT Development made these releases as it must have felt that IT QA did not have the needed bandwidth to test these releases.
  • IT QA has not turned down a single request for mid-week release in 2012 till date.
  • IT QA has not complained about many unplanned projects in all tracks and supported each of them.
  • Though IT QA has pointed out to lack of adequate documents, IT QA has not rejected any project where adequate documents were not available. Instead, IT QA was resourceful in most occasions to use their relations with the Business Users in extracting the requirements through dialogues. Also, IT QA used their knowledge of the systems and the products to extrapolate requirements of projects.
  • IT QA extended support to IT Development and IT Transformation teams in conducting System Integration Testing.
  • IT QA helped the design teams in identifying a few design flaws.
  • IT QA helped the IT Development on a few occasions where they provided solutions for some projects.
  • IT QA has received many appreciations for all the IT QA Team Members from every section of the organisation.
English: Illustration showing DevOps as the in...

English: Illustration showing DevOps as the intersection of Development (Software Engineering), Technology Operations and Quality Assurance (QA) (Photo credit: Wikipedia)

%d bloggers like this: