Understanding Software Defects

We tend to focus a lot of attention on tools and methodologies for improving software quality. I thought it would be worth while taking a step back to try to understand what the root causes of software defects are. Fortunately there has been decades of research that have analyzed the most common sources of software defects.

After also looking at some related development sins, I'll summarize what this new understanding means to me as a software developer.

An often sited article in IEEE Computer is Software Defect Reduction Top-10 List (Vol. 34, Issue 1, January 2001, 135-137) . Here's a summary (from Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research):

1. Developers take 100 times less effort to find and fix a problem than one reported by a customer.
2. Half of software project work is wasted on unnecessary rework.
3. Twenty percent of the defects account for 80% of the rework.
4. Twenty percent of modules account for 80% of the defects and half the modules have no defects.
5. Ninety percent of the downtime comes from 10% of the defects.
6. Peer reviews catch 60% of the defects.
7. Directed reviews are 35% more effective than nondirected ones.
8. Discipline can reduce defects by 75%.
9. High-dependability modules cost twice as much to produce as low-dependability ones.
10. Half of all user programs contain nontrivial defects.

This list is based on empirical research and is a good starting point for understanding how to avoid predictable pitfalls in the software development process.

A broader perspective is provided by Pursue Better Software, Not Absolution for Defective Products -- Avoiding the "Four Deadly Sins of Software Development" Is a Good Start. Here are the four deadly sins:

The First Deadly Sin: Sloth -- Who Needs Discipline?

The Second Deadly Sin: Complacency -- The World Will Cooperate with My Expectations.

The Third Deadly Sin: Meagerness -- Who Needs an Architecture?

The Fourth Deadly Sin: Ignorance -- What I Don’t Know Doesn’t Matter.

The SEI article concludes:

We believe that the practice of software engineering is sufficiently mature to enable the routine production of near-zero-defect software.

🙂 How can you not smile (or even LOL) at that statement? Despite that, I like the reduction of the problem into its most basic elements: human shortcomings. That's why the conclusion is so preposterous -- software development is a human activity, and a complex one at that. You're trying to produce a high quality software solution that meets customer expectations, which is a difficult thing to do.

Another list of software development sins can be found in The 7 Deadly Sins of Software Development.
#1 - Overengineering (in complexity and/or performance)
#2 - Not considering the code's readership
#3 - Assuming your code works
#4 - Using the wrong tool for the job
#5 - Excessive code pride
#6 - Failing to acknowledge weaknesses
#7 - Speaking with an accent (naming conventions)

There are some tool/language specific items here, but this list generally follows the same trend of discovering typical developer shortcomings that can be avoided.

Another source of software defects is poor project planning. More sins (deadly again) can be found in the Steve McConnell article: Nine Deadly Sins of Project Planning.

It's pretty easy to see from these categorizations where a lot of the software development and management techniques, tools, and practices came from. As you might have expected, many are focused on human behavior and communication as a key component for improving software quality. For example, take the Agile Manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

This vision is very telling about what the manifesto writers considered to be a primary cause of software defects.

Another perspective is Fred Brooks' famous 1986 'No Silver Bullet' paper (also see here) that distinguishes "accidental" repetitive tasks from "essential" tasks. From the article:

There is no single development, in either technology or in management technique, that by itself promises even one order-of-magnitude improvement in productivity, in reliability, in simplicity.

Even after twenty years of significant progress in software engineering, I believe that this is still a true statement.

Conclusion:

There are many complex factors that contribute to software defects. There is clearly no one-size-fits-all solution. As a developer, this means that I have to:

1. Be aware of my own shortcomings and biases.
2. Continually try to improve my development, communication, and management skills.
3. Understand how each new tool or methodology that claims to improve software quality fits into the bigger picture of what I (both personally and as an organization) am trying to accomplish.