Bugs: Inevitable, frustrating — and manageable

“[My ship’s computer] is smart, but she’s literal-minded…She is very flexible but the “garbage-in-garbage-out” law applies. She had many extra fail-safes because I make mistakes. [She] can’t; anything she does wrong is my mistake. Since I’ve been making mistakes all my life, I surrounded her with all the safeguards I could think of.”

Number of the Beast, Robert A. Heinlein

Bugs are not computers’ fault; computers do exactly what we tell them — it is the code we write that is flawed. It can be frustrating to developers that our minds are creative enough to write software filled with elegant and simple solutions to complex problems, yet our logical reasoning and analytical skills aren’t good enough to avoid mistakes along the way.

Since bugs are a result of human error, that means that you, or someone on your team, will be taking the blame when a bug presents itself.

In this article I’ll cover why bugs are are so frustrating — and why they are inevitable. Then we’ll focus on what programmers can do to limit the pain. Finally, I’ll give managers advice for creating a culture that minimizes bugs.

But first, let’s ground this in a story: The “phantom subscriptions” bug featured in Hard Refresh’s first episode.

We learned from Pippin and Chris that even the most senior developers can make simple mistakes that have catastrophic consequences. Neglecting to unset() a variable caused it to hang around when it should not have, and as a result, customers were charged for subscriptions they didn’t buy.

The cause of this error was frustratingly minuscule and simple, yet it had the potential for disaster — it could’ve driven them or their customers out of business.

Instead of finding someone to blame, Pippin and Chris got to work. Despite the scary, stressful situation, they got all hands on deck, dispensed with blame, put their heads down, and communicated openly. And they got through it together.

In the end, they took lessons from the experience and applied them to their work. They improved their process with new practices designed to minimize the potential for similar issues to arise in the future. They emerged stronger as a team. And, they were able to build trust with their customers by handling and communicating about the issue so transparently.

To get similar results when you face similar challenges, follow the recommendations below.

Bugs can infest more than just your code

Bugs can infest your team as well. Nobody wants to create a problem with expensive consequences — but sooner or later, everyone writing software takes their turn.

It can be difficult to explain the inevitability of bugs to non-technical people. So, owning up to a bug can be stressful if your co-workers (especially managers) have a hard time understanding what happened, or why you made a costly mistake.

Frequently, the solution to a bug is a single line of code, a missing command, or even a missing character, such as a curly bracket to close a conditional, or a missing semicolon to end a line. Who among us hasn’t made that last mistake, more times than you can count?

Having to tell your boss, “I forgot a semicolon and it broke everything,” never feels good.

But it’s important for everyone — especially managers — to understand that bugs will happen. Especially with older code bases, or teams with a lot of contributing developers, problems can mount quickly and unpredictably, regardless of developers’ skill levels.

Healthy communication is vital. If your team has not fostered a community of trust, transparency and kindness, developers can feel judged by coworkers for making “stupid mistakes”. Cultivating a team culture where code problems are acceptable — because they are expected — is the best way to make sure bugs are they are identified and fixed as quickly as possible.

What programmers can do to minimize bugs

Use powerful software writing tools that analyze problems for you

Programs called “linters” analyze code after you’ve written it; use them to find syntax errors.

Integrated Development Environments (IDEs) are an even more sophisticated class of software. They automatically check syntax and analyze logic while you code, and call your attention to potential problems as you type, much like spell check.

Use syntax highlighting

Syntax highlighting means coloring keywords and symbols in code differently, depending on their syntactical role in the program. This enables programmers to more easily spot problems as they type, even without the aid of the program recognizing and pointing out the error. If the code colors don’t match your expectations, that signals there is probably a typo somewhere interfering with your syntax.

Just about all software for writing software, even basic programs, include this feature. If you’re still using Notepad or some other basic text editor to write code, you’re missing out on this powerful, basic benefit.

Do code reviews

Get your coworkers to read your code line by line with fresh eyes. They will catch some syntax errors you missed, and catch logical errors that hadn’t occurred to you.

This works if more experienced developers review younger developers’ code, for obvious reasons, but it’s a best practice for all code to be reviewed, no matter who on the team wrote it.

Most all developers have something to learn, and something to teach. And, most all developers will make mistakes that a fresh pair of eyes will catch, regardless of their experience. If nothing else, newer developers will learn a lot from reading code written by their more experienced teammates.

Code defensively

Anticipate the unexpected. Start with fresh data each time through a looping process, as in Pippin and Chris’ story with their use of unset() to clear variables. Make as few assumptions as possible — ideally none — in code.

Take the time to generate errors when assumptions are violated while the program is running. You’ll earn that time back when you’re called on to debug some urgent problem that has management sweating bullets.

Recognize that coding in a team environment is social

Even if your code works fine and you understand it, others may not. That invites problems to creep in when others add new features that interact with yours. They will make assumptions (consciously or not) about how your code works and how they should interact with it.

For this reason, work together as a team and over-communicate, to make sure you’re rowing in the same direction.

Don’t be clever

Write simple code instead of elegant or complicated code, even if it makes you feel proud.

Code needs to be easy to read and understand. It’s surprisingly easy to do your coworkers (and even your future self) a disservice by being too clever.

Follow community-accepted code standards

Think of coding standards as institutional knowledge: They are accumulated wisdom of good practices to help avoid common errors.

Some developers view coding standards as busy work, or arbitrary. Don’t be one of them. They may not always make sense at first, but there’s value in consistency, even when applying rules you may not understand at first.

Think of using coding standards as a requirement for your work as a professional. Commit to learning standards for the languages you write in — and the reasons for them.

I guarantee that following standards will benefit you and others.

What can managers do to make bug management less painful?

Calling all senior developers, product owners, project managers, team leaders, CEOs, and CTOs — this stuff is for you.

First and foremost, create an atmosphere of trust and openness on the team.

Do this by validating developers even when things go wrong. Know that bugs happen, and all programmers create them. Thus, you should highly value the skill of open and clear communication in developers. Encourage your entire team to learn from all incidents together, so they build everyone’s experience.

What would Pippin’s team have gained if they had angrily lashed out at the developer who left out the unset() command? The developer would’ve been demoralized and shamed, not motivated or corrected. They wouldn’t have been available to fire on all cylinders, at a time when that’s what the team needed most, to fix the problem and restore their customers’ trust.

Any shaming or punishment of the individual who submitted the bug should be discouraged at all levels of management. (This includes self-deprecation. We developers are often hardest on ourselves, and need reminders that we’re only human.)

To foster an open and supportive team, individuals need to feel safe to notice and report bugs. Otherwise, a developer might be more motivated to to protect their job, or the jobs of their teammates, instead of prioritizing quality code.

Recognize that software is inevitably complex, and will occasionally break

People often think of writing software as similar to building a house: You plan and build a strong foundation, erect a strong, load-bearing structure, add a layout of walls, and layer over everything with windows, doors, and finish that all fit together perfectly — and all according to plan.

Unfortunately, writing software is quite different. Houses are fairly standard. Software is generally, almost by definition, unique. So, imagine building a house without standardized materials, without common practices for most problems, and without a complete blueprint from the get-go. Imagine your builders having to invent the door, figure out how to make a window, and how they should fit together in a wall… That is a much better analogy.

In software, a significant part of what you know about the problem you are solving is learned while solving it. As projects begin, developers may have a pretty good idea of the final outcome. But, as the project progresses, the plan will change.

Because of that, it is generally inefficient to plan things out in complete detail from the start, because plans will change. As developers focus more attention on specific features and aspects of the program, they surface details that couldn’t have been anticipated or planned for. When they apply their creativity to address these details (which is often the right thing to do, depending on context), they might introduce complexity. When they check in with project managers for input (which is also often the right thing to do), the requirements can change. They may suggest ideas for improvements on the original plans, which can introduce entirely new and unplanned features. Their skills even improve during the project as they learn new techniques, which can make code written near the end of the project very different from code written at the beginning. All of these cases typically lead to places where the system doesn’t work together quite as intended.

These are called bugs. And the point is, creating them is inevitable.

This can be difficult for project managers and owners to accept, because it introduces uncertainty and risk — two things that managers are supposed to reduce. But in almost all cases, this is a firm truth.

Plan projects to accommodate wild uncertainties by rethinking estimates

We instituted a rule at Rocket Lift this past spring, on the recommendation of a good friend: When a developer gives you an estimate, triple it.

Don’t think about it. Just do it.

I ask them to give me conservative estimates that they feel confident in. I ask them to assume I’m going to use exactly the estimate they give me for project planning (securing budgets, setting client expectations, etc.). They give me numbers that they truly believe will be accurate.

Then, I assume that they’ve underestimated by a factor of 3. I know that this happens for reasons that are completely outside of their control, and have nothing to do with their skill level, so there’s no judgement.

I apply this rule 100% of the time, with everyone on the team. Sure, experience helps a little, so more senior developers tend to be more accurate, but trust me, it’s still not as accurate as you would think.

Here’s a list of things that I can say since we’ve instituted this policy:

  • We regularly come in on or under budget
  • We regularly meet the expectations we have set for ourselves with our clients
  • My stress level has dropped precipitously
  • We often finish early, which lets us to do more, or make things better than we had promised
  • Our clients seem to like us a lot more

Commit to paying down technical debt

“Technical debt” is a useful way for managers to think about risk in software projects. It encourages you to think of all software programs as on a spectrum: On one end, so full of bugs that it doesn’t work at all. On the other end, 100% perfect and problem-free (a nearly impossible ideal, but always the goal).

Technical debt accrues when developers write code the first time, and get something working, but aren’t able to apply the lessons they learned in the process that will make it more stable. (This typically happens even the second or third time through as well, although the technical debt introduced tends to get less-impactful over each iteration.)

Technical debt accrues when a feature is built but isn’t fully tested: There are probably hidden bugs. It accrues whenever shortcuts are taken to meet a deadline. Similarly, it accrues whenever you hear a developer say, “I’ve got improvements in mind for when we can come back to this, but it’s good enough for now.”

Managers, train yourself to hear this and translate it in your head as “I just introduced future problems in order to meet short-term project goals.” Project goals and deadlines have your team borrowing time from the future to accelerate short-term progress.

Technical debt compounds like interest on a financial loan; the longer you wait to start “paying back” that interest, the harder it will be. And you’ll have to pay it down one way or another down the road: There will either be bugs that crop up later, or the software that worked for a while will suddenly start to let you down as business grows or conditions change.

In our house-building analogy, paying down technical debt is like stopping to re-build portions of the house that didn’t fit together correctly, now that you know the shape of the whole.

If you choose not to rewrite along the way, which time and budget constraints often dictate, you could end up stuck with a shoddy house — say, with bricks in the third story that aren’t supported by a foundation constructed out of paper. Eventually, it will collapse.

Plan for refactoring

If you plan to rewrite code along the way — to remove inconsistencies and redundancies, to reduce complexity and make code more organized — then you have a plan to manage your technical debt. I award you the highest of fives.

request-fiveCommit budget, time, and resources to rewriting code for features that have already been delivered, based on developers’ recommendations and priorities. This is called refactoring.

Recognize that the developers probably have a clearer view than you of the extent of looming technical problems. Trust them when they tell you that some area of code needs attention.

If you are use an Agile methodology such as Scrum, we recommend prioritizing both bug fixes and refactoring above new features. If you don’t have this policy set in stone, it can become difficult to justify budgets for refactoring to the people holding the purse strings — despite the fact that bugs and code cruft will accumulate and compound, costing even more in the long-term to fix. (For long term projects we run on Scrum, we insist on having sprints entirely devoted to refactoring at regular intervals, such as once every 6 weeks.)

Conversely, if you do take this approach, you’ll be managing technical debt gradually, continually, rather than letting it get out of hand and becoming very painful. Think of this as a “move slow to move fast” approach.

Make sure project goals are clear

You, the manager, have perspective that developers may not have on the most important features and milestones for the project.

Ensure that the intended outcomes are clearly understood by the team. Additionally, make sure the software’s “architecture” — its overall plan, which may have emerged during the project — is as clearly understood by your team as possible. You may need to depend on your senior developer for this.

If the team shares your understanding of goals and priorities, they can bring to your attention technical issues with a high risk of threatening the project.

Be aware of coding best practices

Be familiar with our recommendations for developers, above, so that you can ensure best practices are actively used.

Make it clear that you want quality code. No sub-standard code should be acceptable. That will make your interests align with developers, which will make them happy — at least, it will make good developers happy. (This may help you identify developers that you don’t want on your team.)

Make it clear that, because you want quality code, you support the team using coding standards, professional software writing tools, code reviews, and so on.

Be good to your people

Support your people. Encourage them to aim for beautiful code always, and give them what they need to hit it. Secure time and budget resources for them to do things right.

Also, aim for transparency and trust. Foster a healthy, trusting work environment. Structure your process to encourage openness, directness, code reviews, and individual improvements.

Do these things, and you’ll not only have a happy team, but you’ll reduce your instance of bugs.

What’s worked for you?

Are you a developer, or a manager of developers? Do you agree, or disagree with our recommendations?

Share your experiences in the comments below! [Comments are now closed.]

matthew__jpg_340x468_max_q85Author Matthew Eppelsheimer is Hard Refresh’s executive producer, and the managing director of Rocket Lift.