How Much Do Bugs Cost? And How Can Product Managers Optimize
Building first and testing last is a surefire way to make bugs more expensive. But how much do bugs cost product managers in reality?
The wrong bug at the wrong time can end someone’s career, kill a startup, and even abort a space mission. Quite literally: NASA’s Mars rover was once brought down by a simple out-of-memory error.
Software bugs can be really costly, but surely there’s a way to make them cheaper?
What could happen as a result of a single bug depends on the error just as much as it depends on the business itself. Perhaps users will cancel their subscriptions. Maybe poor reviews will make it more difficult to attract new customers.
You may find that a security vulnerability leaves you open to a breach, and hefty fines.
Some estimates say that poor-quality software cost businesses $2.08 trillion in 2020. What’s even more critical than bugs is to balance the cost-to-reward ratio: bringing the bug count down to an acceptable level.
Move fast and fix things: Why does identifying bugs before a release matter?
All things being equal, the earlier the bug is identified, the easier it will be to fix it.
By discovering an issue early, chances are that the programmer who wrote the code is still on the same project.
This is great because the team that introduces a bug is usually best positioned to find its cure — everyone else would have to familiarize themselves with the code first, wasting precious time.
This idea of testing quickly to keep costs to a minimum is supported by numerous studies.
One study conducted by the National Institute of Standard Technology (NIST) estimated that it takes three times as long for developers to fix a bug in the production stage as it did during the coding stage. Similarly, research suggests that it can cost 100 times more to fix a bug after release, compared to before a release.
How to be at an advantage when fixing bugs
To be at an advantage when fixing bugs, a lot depends on how the software team is structured, and how the code is written:
Have proper documentation
It’s a lot easier to write code than it is to read it, and developers don’t particularly enjoy writing documentation.
But docs are super important.
There’s no high-quality software without them, in fact, one definition of “software” is a “program with documentation”.
Teams that test the apps regularly have many test cases written, and those scenarios themselves become one of the most important, most up-to-date — and sometimes the only — part of the software documentation.
Use popular frameworks
Using popular frameworks to build the apps makes it easier to transfer knowledge between developers, and to write bug-free code.
These days most developers will have similar tools in their toolbox: backend people may use Spring, Play, Django, frontend folks tend to opt for React, and even data scientists have popular tools such as Pandas or TensorFlow.
When a software developer is transferred over to a project that uses a familiar framework, the code is already familiar to them, making onboarding a breeze.
These tools have been embraced by a multitude of developers who tested them extensively in production, and they also come with large unit test suites — making frameworks safer to use than any home-grown platform.
Test, test, and test
Integration- and manual end-to-end tests are useful, but unit tests are a must.
And the best idea is to make sure people write tests for parts of the system that they didn’t themselves build.
Writing good unit tests requires people to understand the problem space deeply, which means that knowledge transfer happens organically.
Friction logs can also help here and you can also ask users to participate in this during a beta test, with a simple in-app modal.
Frequent releases
Releasing often means that rollbacks are simple.
If a dev team can remove the changes from the live service after finding a single bug, they can work on the issue “offline”.
By not having to “fix the bug forward” while it impacts users in production, developers can allocate more time to work out all corner cases around the issue.
Popular frameworks, documentation, testing, and frequent releases all help developers move fast without having to move with urgency.
They allow teams to allocate sufficient time to QA the completed work, which in turn helps avoid introducing more bugs while fixing one.
The easiest bug to fix is one that wasn’t made in the first place.
The clean-up experiment
Startups are often seen as risk-takers, but in reality, this can’t be further from the truth.
Most tech teams run short experiments (fake door tests are quite popular) before spending a lot of time on developing a fully-fledged solution.
They would build a minimum viable version of the idea first using a continuous discovery framework, give that to their customers to use, collect feedback and measure the results.
Some of the new product features will work out well and become part of the core offering in the long run. And most of the features will fail, which is also part of the plan.
That’s the reason why the product team only spent a limited time on building it.
As a result, people are quite often at work building the “wrong thing” at a tech company.
In software, the goal post keeps moving fast: a feature that the whole team worked on for months might turn out to be obsolete a week after it was released. This happens so often with software.
So the developer looks at the code one morning and finds that all the code that was added in the last few weeks isn’t actually needed any longer.
They have two options here: one is to keep the functions and infrastructure set up because they might be repurposed for a similar feature later. Or, they could remove everything — which is again a whole lot of work: they’d have to first change the code base, then re-test everything to make sure the product still works exactly as expected.
Deleting something usually takes longer than simply leaving it there, plus it’s also hard to roll back in case the product team changes their mind.
It’s not a surprise that most people will err on the side of caution, and opt for the former option: keep the decommissioned functions in the code base. Unfortunately for the developer who’s chasing a persistent bug sometime in the future, those functions might prove to be a red herring while looking for the source of the issue.
If there is one principle that should be added to the UNIX philosophy, it is: “Finish your project.”
It’s a super simple, yet often disregarded software engineering principle, because “shipping” a product means quite a bit of extra work compared to just hacking away.
Tech teams don’t have to clean up the code base after every twist and turn, but there should be time allocated to stop to organize and prioritize what’s next. Planning ahead to properly ship features might cost time at the moment, but will pay huge dividends later.
Cleaning up the unused features, and restructuring the code for features that are actually used will make all the future bug fixes cheaper.
Designed for simplicity
Programmers are not generally good at UI/UX, so an app designed by developers is rarely user-friendly or easy to use.
When teams build more sophisticated products, they often start with hiring a designer who creates high-fidelity screen layouts. The developers then take the blueprints and build the working app.
By designing the app first, many of the to-be bugs can be removed before a line of code is written — bringing the error discovery even sooner in the process.
It’s also a great time to have a feel for how specific workflows will work in the software, and simplify everything that’s possible. The best user experiences are the ones that are invisible, where even first-time users can navigate the interface.
Simplifying takes a lot of effort. When people work on all kinds of problems, they tend to favor additive solutions and rarely start by removing things.
Consider the quote, widely attributed to Mark Twain:
“I apologize for such a long letter –- I didn’t have time to write a short one.”
A study recently published in Nature showed that, in a series of problem-solving experiments, people systematically overlooked subtractive changes. Only when participants were reminded that both adding and subtracting could lead to potential solutions were they more likely to explore both approaches.
“Additive ideas come to mind quickly and easily, but subtractive ideas require more cognitive effort,” says Benjamin Converse, one of the study’s authors. “Because people are often moving fast and working with the first ideas that come to mind, they end up accepting additive solutions without considering subtraction at all.”
“Subtraction is the act of getting to less, but it is not the same as doing less. In fact, getting to less often means doing, or at least thinking, more,” says Leidy Klotz, one of the researchers on the project and author of the book Subtract: The Untapped Science of Less.
Software complexity is eating the world
Not all bugs are created equal — some are easier to live with, while others make the team drop everything else and focus on fixing the problem right away.
Many businesses build a prioritization system centered around two important elements:
- The investment: How much will it cost to fix the bug today compared to fixing it tomorrow, or next week? Is it better financially to invest in a speedier solution?
- Impact on the business: Does the bug affect the customer experience, or can the software still be used OK? How urgently do users need the problem to be resolved?
The largest critical bug I personally saw at a QA company was when a client broke their user signup flow.
For a startup, this is one of the worst scenarios: they spend a huge amount of cash to get signups, and all of that goes to waste when potential users arrive at the site that doesn’t work. People who see a website broken on their first visit rarely ever come back.
Moving fast and breaking things is just the way to go in startup land, where we’ll definitely see more unicorns than bug-free apps.
Bugs can pop up in literally every feature, even the ones that people use often.
Murphy’s law dictates that the bad code will go into production right after the marketing team has blown most of their budget on attracting new customers. In fact, the area that testers often think is ‘safe’ is the most prone to errors — because that’s what will get skipped first when the QA team runs out of time.
Nothing in the code base is safe
During active development, the app’s latest features tend to be tested multiple times a day. The developers, the QA team, the project managers, and everyone else is keen to check out how the latest features work.
But all the while, nobody looks at the quiet, cobweb-ridden corner of the old features – the ones that customers are actually using right now.
Everything that developers do — adding a new feature, removing an existing feature, re-designing the UI interface, integrating modules, managing the database — all has a ripple effect right through the app.
In every code base, functions depend on each other.
When a developer works on a new feature, they often rely on other code previously written — by themselves or by others.
There isn’t always enough time to understand exactly how the old function’s internals work, let alone re-architect the code to be more reusable — so when someone changes the old functions for the new feature… well, most of the time nothing bad happens.
But sometimes, bugs start showing up.
Who ‘ya gonna call?
Quality is a habit.
A big mistake that many businesses make is that they get a product ready for release and only then carry out a test, just before it goes out to customers. By this time, a bug could already have had an impact on the usability of multiple features, and developers may need to spend time to re-familiarize themselves with their previous work.
Rather, quality assurance should be a continuous effort.
It’s generally the QA team’s responsibility to find bugs early on, so the first point of attack is: making sure error monitoring is set up, and testing is happening regularly.
The quality assurance team should be involved in development from the very start, helping to identify possible issues early on, and providing developers with an opportunity to fix them quickly.
To make that process as efficient as possible, having a good idea of where bugs are most likely to occur in your app can be hugely beneficial. That’s why it’s considered best practice to create testing checklists that analysts can tick off every time they perform their regression tests. Never underestimate a QA team! They’re the secret to reducing bug costs.
About the author
Richard Dancsi holds a master’s degree in mathematics and computer science. He has built diverse software products across cultures and teams, including the likes of IBM, Orange, and Vodafone.
As a tech entrepreneur and independent consultant, he worked with many investors and teams over the past 20 years. Richard helped founders get from zero to one as well as top-tier clients to build and scale. His current engagements focus on project rescue and improving product teams (at Bunt), and helping companies ship high-quality software (at TestTheTest)
