How to measure the code quality of your project?

Long story short

Don’t measure it. Measure consequences of the code quality which matter the most:

• Customers’ satisfaction;
• Stakeholders’ satisfaction;
• Engineers’ satisfaction.

The problem

Since the moment I read “Clean Code” by Robert C. Martin and “The Pragmatic Programmer” by Andrew Hunt and David Thomas 10 years ago, I have been trying to improve my skills in writing the “good code.” Moreover, I also have been thinking about how to measure the current quality of the code, search for weak points and control the improvements process.

So far, what do we have for our engineering excellence? We have a lot of best practices such as TDD, BDD, something-else-DD, design patterns, pair programming, code review, continuous delivery, etc. We have many tools like linters, type-checkers, formatters, generators, IDEs, and so on. We even have a ton of metrics we can calculate on our code-base: code coverage, cyclomatic complexity, code churn, cohesion and many more.

They all are very useful, but can we be sure that if we have used all of them, we are safe? Imagine you are a technical lead, and you are asked: “is our technical solution good enough?” or “should we worry about any technical issues?” You have found all possible tools, applied all possible best practices, and measured all possible software metrics — all checks are green. Then we can relax and go on a vacation — right?

I don’t think so. In my career, I have seen several examples when applying some of these practices and tools didn’t help or even made the situation worse. For example, one day, we decided to improve our code excellence and integrate TDD, pair programming, push code coverage and some other best practices and tools. After some time, we noticed that the result was quite the opposite of what we expected. Engineers started to leave the company, product owners complained about the development speed, and the customers’ churn increased. Later we understood that some people hate pair programming because they think that it is equal to supervising other’s coding process for several hours without switching. The coverage was almost 100%, but the tests were flaky and complex, so people spent more time fixing tests and keeping coverage high than implementing features. All of these decreased the speed of the development and motivation of employees. As a result, our application became slower and buggier, which could not improve our revenue.

How is it possible, and why does it happen? Based on my experience, the root cause is the wrong target metrics. With the wrong metrics, we are blind to the real consequences. We have to measure what matters. It does not mean that we should not measure code coverage, for example, but we must understand that this metric is collateral. In other words, the high code coverage is not what we really need at the end of the day. But what do we need? What really matters, and what makes the difference?

What matters

To answer this question, let’s answer an even more fundamental question first: Why do we write this code? What is the purpose? If you work in a commercial company, the purpose is profit, right? I know all companies have their missions, visions and all of them want to change the world. But if it is not a charitable foundation, they need profit at least to survive. No profit — no salaries — no results — no changes of the world. Ok then, we write our code to make a profit.

How can we make a profit from it? What makes our users pay our company? This answer is specific for each company, but there is one common principle for the competitive market.

Your product should be valuable for your customers, and your customers should be happy using it.

Small remark that it matters only for a competitive market because otherwise, it can be that you can somehow force customers to use your software even if it has no value or your customers just have to use your product if there are no other options.

Customers’ satisfaction

One of the metrics we can use to measure customer satisfaction is NPS. Of course, you can find more ways and more metrics, and there are people who know much more about how to measure customer satisfaction. I just suggest using it as an example. I like this metric because it is representable and straightforward, in my opinion. In other words, if somebody is ready to promote your product and recommend it to their friends — it looks like your product is really making a difference for them.

Wait a second! Why are we talking about NPS — how is it related to the code quality?

The truth is it is directly related. We should make our code good not only to satisfy our own ambitions but to make our product work good and satisfy customers, not technical architects. As a result, the first key metric for code quality is — “How happy are our customers?” I suggest using NPS for that, but it is up to you how to measure it.

Stakeholders’ satisfaction

Let’s go forward. The NPS can show us the current situation, but in every company, there are people whose duty it is to think about the future. They investigate the market, customer needs and plan the following steps to improve revenue and beat competitors. In all companies where I have worked, we called them “stakeholders.” It can be product owners, product managers, marketing experts, sales gurus, executives and even literally shareholders.

All of them should be happy about engineering results. For now, I don’t know a better way to measure this than just ask. To be honest, in my experience, it has never been a problem at all. These people are providing feedback even more often than you may think about it. At the same time, just in case some of your stakeholders are not mature regarding feedback, it is crucial to keep in mind that you have to be sure about their goals, their measures and their feelings about your job. If, for some reason, you are not sure what plan they have or how they evaluate your work in the last quarter — reach out to them and ask as many questions as needed till you have a complete picture. Your stakeholders have to be happy.

Engineers’ satisfaction

Eventually, what about our engineers?..

Well, we can think that making technical decisions and choosing the development strategy in the engineering department we should think only about objective technical consequences. I think that it is possibly one of the greatest mistakes technical leadership can make. Based on my experience, motivation to do technical improvements, confidence in the current technical choices, comfort in the daily development process and all other things related to engineers’ feelings about their work environment — at least as important as, or even more important than, technical characteristics of the chosen tools and solutions.

For example, let’s imagine the situation when you need to choose between frameworks A and B. What lines in the compare table first come to mind? Probably, it can be performance, size, tooling, the number of stars on GitHub, licence, contributors, documentation, etc. All of them are important for sure. At the same time, what if you, as a leader with your closest subordinates, have chosen the best framework B based on these characteristics, but the rest of the engineers hate this framework? I have to assume that it will be a disaster. No matter how good the framework is if people don’t understand this choice and don’t accept it the result of their job will be messy.

The key conclusion here is that you should always keep your eyes on your engineers’ feelings about technical decisions and improvements. It is important to understand that it is not only about framework choices, it works in the same manner with any changes: integrating new practices, refactorings and standardizations.

cNPS

How to use NPS to measure code quality?

If you just measure NPS in your team or a whole department, it can be hard to differentiate the impact of the code quality and other work-related factors in this number. To solve this problem, we can try to ask engineers specifically about their opinion of the code quality in the same manner as NPS does.

My suggestion is to use the following question: “Based on your engineering experience with our project, how likely is it that you would recommend our project to a colleague or peer?”. The idea behind this question is pretty much the same as behind the NPS itself — the more you like to work on the current project from the technical perspective the more you are ready to promote it to your friends. We can call it “Code Net Promoter Score” or just “cNPS”.

The technique is to send a poll with this question to your engineers periodically (monthly for example), collect data and calculate it as usual NPS. Then you organize meetings and a technical backlog based on the feedback from such polls.

Eventually, with all these metrics you will have the best technical solution your engineers can offer, and be sure that your customers and stakeholders are happy as well.

p.s. if you have any concerns, suggestions or questions about the idea described above, please, leave a comment and I will do my best to address it.