Quality Metrics to Consider For Your SaaS Product Series — Part 1

Back in the day, quality was synonymous with testing — how many tests were run, how many passed and how many failed. Quality of a feature being released was determined by these failures, and by the number of issues that the customer reported.

Cut short to the world of software today — we have moved to the world of services. Multitude of these services interact with other services in complex ecosystems, with third party integrations, accessing and manipulating terabytes of data at near real time speeds. Do the features work? — is no longer a sufficient metric to track the quality of a product. The questions about quality have gotten more diverse and complex, spanning multiple tools, teams and initiatives:

• Does the feature work? How many test cases passed / failed?
• How much time did the operations team spend in deploying / maintaining the new service?
• How does the feature impact security posture for the product?
• Do we have code hotspots that generate a lot of bugs?
• Is the product maintaining its SLAs and SLOs?
• How quickly can someone debug and fix a P0 issue for this feature?
• What is my deployment cadence and how is it affecting incoming defect rates?

To sufficiently answer the quality question, today you need data from multiple engineering tools — Jira, SCM, PagerDuty, Security Tools, CICD tools to name a few. You need to be able to successfully co-relate metrics and draw insights on the success of your quality program.

In my previous roles, as I helped build and scale quality and performance teams for a rapidly growing SaaS product, along with regression results and test coverage, there were a few metrics that helped me track and answer quality questions around the product:

Change Failure Rate

In a SaaS world, deployments are frequent. Unfortunately, so are customer impacting outages. As we scaled teams, it became critical to find a balance between how much and how often we deploy vs how stable the product is. By tracking deployment frequency and deployment sizes against incoming outages and customer issues, we were able to make data driven decisions on how often we deploy. Transitioning from weekly to bi-weekly deployments immediately improved the stability, gave features more time to soak in staging and reduced outages

Deployments vs Incoming Bugs, Volume of Change being deployed

Code Hotspots

We all have parts of the code that are complex, critical or just simply magical. Any changes to these areas of code require rigorous and careful testing and even an altered deployment plan to cover for rollbacks. We had a 2 fold problem here — we did not fully know where these hotspots were and we did not know if any of those hotspots were touched in the upcoming deployment. By correlating incoming issues to the areas of code where the fixes went in, we were able to get a rudimentary map of hotspots, review the release change list against the map and determine the risk factor for a deployment.

Code Hotspot HeatMap

Release Security Metrics

It is a no-brainer that security is a critical aspect of every SaaS product. But with rapid development and deployment, we put security testing into the final steps of release planning. It was a hard lesson learnt when after a feature was developed and tested, we realized that we had overlooked some security aspects and had to go back to the drawing board. Incorporating security tools into the CICD pipeline and tracking vulnerabilities and best practice violations on the release development dashboard saved some unpleasant surprises downstream

How are quality metrics changing in your engineering environments? Let us know in comments below.

In the next part of the series, we will go over other metrics that impact your quality.