Penalty

I have been involved in Product Development for many years. Also, I managed to sell all the products that we have developed till date to at least one customer. And thus, I have been involved in Application Support for many years as well. I would like to add a dimension of Application Support i.e. Penalty. As a Business Manager, it is possible to make a decent EBIT for the Department, if we sold our products, and if we prevented ourselves from being penalised for any application flaws. I present one of our strategies for prevention from being penalised.

Let me give you an example of how a Telco calculates the penalty for a vendor.

We supplied our (Interactive Voice Response System) IVRS to Reliance. The IVRS was used primarily for 2 purposes – 1) for Balance Enquiry and 2) for recharge.

The Balance Enquiry transaction had no direct revenue impact. It had indirect revenue impact as if it did not work, the customer would be dissatisfied and this could result in churn.

The Recharge transaction had direct revenue impact. Our IVRS used to process at an average 3 recharges per subscriber per day. This results in 900 million recharges per month. This means 30 million recharges per day. This means 2.14 million recharges per hour (as we considered one day = 14 hours as this is the active time on the network). This mean 35,715 recharges per minute. If the average recharge amount is taken as Rs. 30, then net revenue from recharge is Rs. 1.07 million per minute.

So, if the IVRS is down for 10 minutes, then the net revenue loss is Rs. 10 million.

So, if the IVRS is down for 10 minutes, then Reliance could charge us a penalty of Rs. 10 million. This would be the cost of failure.

Now, how did we guard ourselves from this risk?

We designed an architecture in which we placed about 300 IVRS Clients. This was also because we used Intel Dialogic Cards and these cards had capacity limitations. Behind these IVRS Clients, we placed about 50 computers (ordinary PCs), each running the IVRS Socket Server. We could have bought bigger machines (servers) or even one medium-sized UNIX server. This would have reduced the number of installed computers for running the socket server. There would not have been much difference in hardware cost, except that we would need less rack space. Many people in our company thought our architecture was stupid and always criticized us. They also told us that we did not have enough UNIX ability and thus were sticking to Windows-based IVRS Socket Server.

The reality was that we had so much redundancy that the chance of total failure was next to impossible. At any point, even if 10 socket servers (1/5th of total socket servers) failed and were down for 10 minutes, the net revenue loss would be (Rs. 10 million / 5) = Rs. 2 million. This reduced the risk enormously. To pay a penalty meant reduction in EBIT for our department which meant reduction in amount of bonus for every team member, besides lowering of appraisal scores.

We further programmed a deamon that could not be killed under any circumstance, except unless the machine was switched off. This deamon could bring up a socket server in less than 5 seconds if the socket server went down and the deamon did this job automatically. (We called it WATCHDOG). So, we never had any failure for more than a few seconds. If the machines switched off, Reliance could not penalize us as this was their responsibility to keep the machines in running condition.

When we needed installing a change to the socket server program, we used 5 test machines and ran the new program there for 14 hours as a rule to check for any possible failure. When it ran for 14 hours at a stretch without failure, we pushed the new program into the 50 computers.

In 4 years of supplying software and supporting Reliance, we did not pay single paisa as penalty.

Do you agree? What could have been a better strategy for us?

 

English: Architecture of Interactive Voice Response Systems Deutsch: Architektur von Interactive Voice Response Systems (Photo credit: Wikipedia)