What is a Metric?

Effective management of any process requires quantification, measurement and modeling. Software metrics provide a quantitative basis for the development and validation of models of the software development process.

Why Metric..?

These metrics can be used to improve software productivity, quality and overall process improvements by means of appropriate and timely preventive and corrective measures. These are essential as it provides quantifiable justification of facts and accordingly helps us re-align towards the goal.

What are the metrics that are essential for Software development?

Following are few metrics that must be applied to get useful information:

1. Management:
   1. Schedule v/s Effort Variance
   2. Budget v/s Revenue
   3. Profitability
2. Team Members (Developers and Testers)
   1. Time spent per task per person
   2. Time spent on rework (COPQ -> Cost of Poor Quality)
   3. Study of Impact Analysis wrt changes and defects
   4. Test cases planned, executed and its result
   5. Code Review Effectiveness
   6. Defects/KLOC
   7. Defect Removal Effectiveness
   8. Test Coverage

If you notice most of these metric is related to time taken for a task, code review and rework time. Very often timesheet is the most neglected area wrt process or this is the one that takes the beating when it comes to timely quality delivery.

Let’s discuss few important metric that are useful:

1. Cost of poor quality (COPQ)
   

   This is one of the metric that must be given very high importance. COPQ comprises of undesired costs which are generated as byproducts of defective and inconsistent processes.

   If one converts these numbers into dollars, it will give true information on ‘how much effort went into fixing issues’ OR ‘how much revenue was lost due to bad coding’. Having said that ideal scenario will be to have zero COPQ, but truth is every software has bugs associated with it. Most of these software products go live with known issues or bugs. These are prioritized and released as patch releases.

   Overall goal should be to reduce rework as much as possible and apply certain processes such as:

   1. Following coding standards and its checklist
   2. Logging and tracking code review bugs against the developer and using this as a measure to provide timely feedback. This way it helps individual to raise their own quality bar and reduce the number of rework
   3. Every check-in must follow coding standards, checklist. It must be peer reviewed by Senior tech force in the team
   4. Updating these standards and checklist on regular basis depending upon the new learning’s.

    

   Lot of free tools is available to ensure best coding practices are followed, such as FxCop, StyleCop, NUnit etc. Also note the cost of these poor quality decreases if you catch these early in the cycle. This metric is closely tied with code review effectiveness as well.

2. Code Review effectiveness (55% to 60%)
   

   As mentioned in previous section, the goal should be to catch the bugs early in the cycle. Ideally lot of these bugs must be caught while conducting code review. The industry standard wrt code review effectiveness is 55% to 60%. This means that one must catch 55% to 60% of bugs during code review itself. This means only 40% to 45% of bugs can leak through and these may be related to functionality issues, user workflow issues.

3. Defect Removal Effectiveness (DRE around 98%)
   

   Once the QA team certifies that the software product is ready to go live. The number of defects that can be found during UAT(User Acceptance) must be just around 2%. Let’s say during QA phase the number of bugs that QA team found for a particular software product was 400, then the number of UAT bugs can be at the max 2% of 400 which is 8.

   Aim should be to ensure that 98% quality is achieved along with list of few agreed upon known issues. One must keep this as benchmark and improve further.

4. Key Management metrics
   
   1. Effort Variance (10 to 12% max)
   2. Schedule Variance (0 to 5% max)

    

   These metric benchmarks are different in different companies. Most widely used metric range for the effort variance is 10% to 12% max. In case this is exceeding, it means Scope Management has issues and needs to be fixed. It all boils down to expectation management and applying proper estimation process. Learnings from previous retrospection must be applied to ensure success and continual improvements.

   Schedule variance, ideally should be aimed as 0% variance; which means delivering quality product on time (on the agreed upon date). These dates are initially agreed upon depending upon the scope of work, number of resources and the time it takes to build the scope, test, integrate and deploy.

   The list of these metric is huge. But one thing at the end finally matters is “Were you predictable wrt quality and time“. If yes, you have met the project objectives.

Some sample metric are shown below:

Case Study Project: ABC Project

Problem Statement: To build a customer portal that integrates with defect tracking database, provides content management, helps provide licenses info, download and upload relevant licensed Software and its patches.

Project Challenges: Initial 1-2 weeks of RA phase was getting dragged till 4-5 weeks. The key stakeholders had confusion on the requirements, wireframes and were debating what the best use case wrt be usability. They felt its better to be absolutely clear before development phase begins. The onsite PGM was fairly new wrt MOSS technology and had to commit to client on ‘what is possible out of the box and which requirement requires customization’. In such a situation what was committed at the end. This project was a fixed price project to be delivered within 3 months duration and with fairly new MOSS developers. Dev’s would have sat idle if we had waited for requirement to be frozen. How did we resolve these challenges and much more…to know this read further? J

First proactive action: Development was initiated much before the FS was frozen. The assumption was that 30% of the requirement may change. This means certain nitty-gritty would come up and we had to apply these changes. Technical design sessions and approach was discussed and documented. This laid the solid foundation for the team to get started in the right direction.

Second proactive action: Since requirement had evolved while development was in progress and to ensure there is uniform understanding among all the key Stakeholders, we planned 3 demos before the final release. This helped us to re-align and set right customer expectation. Early demos of the product helped these key stakeholders to decide upon what exactly they wanted. The changes wrt user workflows, changes in feature, UI changes helped this project raised a CR that was almost 30% of its initial value. Multiple benefits…isn’t it?!! J

Other critical challenges: Since development was scattered and requirement were not getting frozen, the check-ins were done in working folders. The code review happened only after 1 month’s time. This was the time when we first reviewed the code and realized that the team was way behind the coding standards. The data below speaks for itself. Check the defect numbers 35, 27, 32 and 17…

Corrective Action: Detailed code review feedback was given to individual developers. Based on this a code review checklist was prepared, coding standards were re-circulated and we walked through these standards and checklist for 3-4 hours. The checklist was updated as and when area of improvement was required by the team members. FxCop was implemented, defects/KLOC was published with individual developers and expectation was set.

What was the improvement?

Applying these simple rules such as following coding standards, checklist and peer reviews helped the team achieve better quality. Check the new defect numbers: 6, 8, 8 and 7 as against initial defects 35, 27, 32 and 17.

Well, you are right, team had to stretch and go extra mile to ensure quality standards which the MOSS Solution Architect was expecting were met. Check out the Effort variance thereafter:

Achievements: This project was delivered on time with required quality and by the end of this project; we had quality developers who were MOSS experts as well. This dev team is grown into Senior Developers and they cherish the learning’s of the project even today.

Written by: Shrinath Inamdar on April 27, 2010.on March 14, 2011.