Updated on December 19, 2016
Disclosing security vulnerabilities publicly is absolutely essential, but it’s nearly impossible to do it without upsetting some people.
For some, the WordPress update to version 4.2.3 caused, to put it lightly, a bit of a headache. (Listen to Richard Tape’s account here!)
The fallout from that update cascaded through developers all the way to front-end users. And that’s what you never want: for your front-end clients to experience site malfunction. We call these, “breaking changes,” or a “break in backward compatibility.” There was a lot of discussion in the community about why it happened, what steps could have been taken earlier to avoid panic and frustration, and why it felt to WordPress users that this update was sprung upon them, like a trap.
An update that causes surprising breaking changes is like your landlord finding out that your apartment building is made of super flammable material, like tissue paper. And because your landlord doesn’t want the arsonist down the street to realize that the apartment building is a tinderbox, she comes over in the middle of the night and replaces everybody’s walls with brick. How nice of her!
But now you can’t hang any of your pictures on the walls! And what if you don’t really like the color of brick? Also, when your landlord came by she actually knocked some of your pictures off the walls and broke them! You storm out of your apartment in a righteous rage, but your landlord is sitting exhausted in the lobby and says, “Sorry buddy, but I got here just in time. Joe the Arsonist was already on his way. I had to be sneaky, and I had to be fast.”
A little warning might have been nice
But “warning” is a double edged sword. If your landlord broadcasted to the neighborhood, “Hey! I’m going to be walling up this building with brick here soon. No big deal, nothing to see here, but yeah. Brick! And soon!” Joe the Arsonist is going to be like, “Huh? Brick? Wait a second, what are these buildings currently made of? Tissue paper!? Oh, this is going to be fuuuuunnnnn.”
If nothing else, publishing any kind of warning or advisory tells an attacker what to focus on, and possibly how to exploit whatever is about to be fixed. There’s unfortunately no way around this though, especially with an open source codebase. As soon as your landlord swaps out the tissue paper for brick, Joe is going to be looking around at the other buildings that don’t yet have brick.
As soon as you publish the change that fixes a vulnerability, even without otherwise calling any attention to it, a spotlight is shone on that specific code. If someone knows how to read the code, and can work through the ramifications of the change, they’ve got all the information they need to figure out what the vulnerability was, and how it might be exploited.
My house is made of brick now. So, now what?
Now you spread the word to your other neighbors. In the WordPress world, code is “open source,” which means the code is published openly; it’s free for users to develop, use and manipulate as they please. This goes for everyone, regardless of their personal motivations.
Publishing a code release with an explicit security patch shines a spotlight on the problem. Doing so gives users only a tiny window of time to update their site before “Black Hat Hackers” have the opportunity to exploit the security hole. This is why it is so important to keep your site’s WordPress core software and plugins up to date.
It’s vital for developers to be honest and upfront with users about security vulnerabilities. That goes for bugs in general, but especially with security-related things. Publishing security advisories is a must, but deciding how and when is delicate and nuanced.
Even with closed-source software, keeping mum about things only adds layers of obscurity, rather than keeping the existence of the vulnerability a true secret.
What could the WordPress core developers have done differently?
Possibly, not much. There may have been some way for them to communicate some information ahead of time, but that would risk tipping their hand and calling attention to the vulnerability.
There are protocols in place for even average WordPress users to notify core contributors if they suspect they’ve discovered a bug. There’s usually a period of limbo, where the vulnerability, the change that fixes it, and all conversation about it remains private right up until the shipping time.
Once you publish the fix, all your cards are on the table, completely. The “bad guys” and the “good guys” get everything at the same time. Now it’s a race, between the “bad guys” trying to craft exploits and compromise sites, and the “good guys” trying to get the update installed so they’re no longer vulnerable.
A heads up for the good guys is the same as a heads up for the bad guys
Your landlord could have gone around knocking on doors and said coyly, “You might want to pull all of the paintings and pictures off of your walls. Just saying.” Joe would have wondered what was about to go down.
If the WP Core team had published info ahead of time about how shortcodes needed to be changed, that’s still basically telling the world, “there’s something funky with this part of our codebase.”
Even as normal developers and administrators would be looking at this information and figuring out “what does this mean for my shortcodes?”, exploit authors will be figuring out “why did this need to happen? What is it about how the old shortcodes worked that’s broken, and how might I exploit this?” It kicks off the same race, except the “good guys” aren’t able to actually protect themselves. The patch hasn’t been released yet.
This is actually a huge reason why the core team has made automatic updates such a huge priority. Sites that update themselves will always be faster at improving their code than sites requiring human intervention. When updates install automatically, this window of time between “an exploit is made known” and “the exploit no longer works” gets drastically narrowed.
Get your helpful landlord a glass of lemonade
I think the Core team was stuck between a rock and a hard place here, and I think they probably made the correct decision.
If you have to choose between shipping a fix that’ll protect millions of your least self-sufficient users but will break the sites of your hundred or thousand most savvy users, vs holding back the fix for some unknown amount of time to solve a truly complex problem in order to “do it right” while leaving everybody vulnerable, I’d choose to just ship the thing ASAP, too.
All this said, there was one point of communication failure that the core team could have done better. Any breaking change like this must be highlighted when it gets released. You can’t assume that everybody will read a release’s changelog before they install an update. It’s important to highlight to users what features are likely to break before the update gets installed, and especially when it’s a point-release where there’s an entirely reasonable assumption that there’ll be no breaking API changes.
Sometimes a bug is a feature. Bugs are generally “bad” and don’t add value, but in this case, the vulnerability itself was not just a vector for exploit, but also a useful behavior that people relied on. Users were manipulating the shortcodes in clever ways to enhance their personal sites. Because the bug presented a security vulnerability, they had no choice but to fix it. And in order to fix this bug, the core team had to remove the feature.
What do you do if you suspect your apartment building is made of tissue?
If you believe that you have found a security hole either in your own site or someone else’s, the best way to proceed is to communicate privately with the core team. Do your best to do so securely, as well. Use the project’s security-specific reporting mechanism, rather than posting to any public support channels or bulletin boards.
Make sure you’re also giving them valid contact information. You don’t have to use your “real name”, or anything else that’s tied to any of your established identities, but make sure you’ll actually see any messages they send you in response to your report. They’re almost certain to have questions for you, and may need your help testing the fix. If nothing else, it gives them the opportunity to thank you for helping them out and saving millions of people a mountain of grief.
Posted on November 22, 2016
/bakwərd kəmˌpadəˈbilədē/ — noun
The ability for hardware or software to interface with each other, regardless of its current version. If an update to one system interferes with another system’s ability to function, we would say that the former system is not “backwards compatible.”
We have all come across this at some point: after using the same program for years, one day we are told that we need to update our computer, and after doing so, that program no longer works. It is a source of endless frustration and grief.
Software developers are aware of this common pain-point. For many companies, it is a specific goal or even a core value that their systems remain backwards compatible. But, not all companies have the desire (or the resources) to do so.
Economically, backwards compatibility is expensive. In fact, money can be made by designing systems that are not backwards compatible. Remember how awesome Zelda: Ocarina of Time is? Or Mario Kart? Well, you can’t shove your Nintendo 64 games into a Nintendo Wii, can you? Nope. And that’s how the game makers like it. As the technology improves, some things get left behind.
WordPress strives to always remain backwards compatible. (And can cause huge upset when it fails to do so.) This means that (ideally) the plugins you installed last year should continue to work with the new WordPress core software update. The amount of testing and experimentation that has to go into these updates is unfathomable. There are 46,886 plugins available to add to your WordPress site, as of this publication. These are plugins submitted by individuals or organizations with varying degrees of upkeep.
A three way balancing act: the needs of the user, the needs of the developer, and the needs of the business
Backwards compatibility tradeoffs are that the software can’t be developed as quickly. New innovations in software (new features of language, or just new features being developed for an application) tempt developers to use them, to either improve on old ways of doing things, or add entirely new capabilities.
Supporting old platforms comes at a price: at worst, it rules out using tempting new technologies, and at best, it requires a lot of time, consideration, and care in the development of the new technologies.
Therefore, software that is devoted to backwards compatibility will tend to advance more slowly, but with a lot less frustration for its users. Software that isn’t devoted to backwards compatibility will tend to be shinier and newer, but with a lot more frustrations for its users when things break all of the time.
Again due to this tradeoff, a devotion to backwards compatibility is essentially a decision to prioritize the needs of users — stability, consistency, reliability — over the needs of developers — novelty, creativity, tinkering, pushing the envelope to make things better, often at a cost to the business. Backwards compatibility testing is expensive. Developers appear to lose out in this equation, but not when you take the long view: Developers serve users, so user happiness with a piece of software means developers’ work is valued by others. If users lose out due to developers chasing the shiny new thing, eventually the users will abandon those developers’ projects in favor of more stable platforms, and then what has the developer gained?
Updated on October 21, 2016
“[My ship’s computer] is smart, but she’s literal-minded…She is very flexible but the “garbage-in-garbage-out” law applies. She had many extra fail-safes because I make mistakes. [She] can’t; anything she does wrong is my mistake. Since I’ve been making mistakes all my life, I surrounded her with all the safeguards I could think of.”
– Number of the Beast, Robert A. Heinlein
Bugs are not computers’ fault; computers do exactly what we tell them — it is the code we write that is flawed. It can be frustrating to developers that our minds are creative enough to write software filled with elegant and simple solutions to complex problems, yet our logical reasoning and analytical skills aren’t good enough to avoid mistakes along the way.
Since bugs are a result of human error, that means that you, or someone on your team, will be taking the blame when a bug presents itself.
In this article I’ll cover why bugs are are so frustrating — and why they are inevitable. Then we’ll focus on what programmers can do to limit the pain. Finally, I’ll give managers advice for creating a culture that minimizes bugs.
But first, let’s ground this in a story: The “phantom subscriptions” bug featured in Hard Refresh’s first episode.
We learned from Pippin and Chris that even the most senior developers can make simple mistakes that have catastrophic consequences. Neglecting to
unset() a variable caused it to hang around when it should not have, and as a result, customers were charged for subscriptions they didn’t buy.
The cause of this error was frustratingly minuscule and simple, yet it had the potential for disaster — it could’ve driven them or their customers out of business.
Instead of finding someone to blame, Pippin and Chris got to work. Despite the scary, stressful situation, they got all hands on deck, dispensed with blame, put their heads down, and communicated openly. And they got through it together.
In the end, they took lessons from the experience and applied them to their work. They improved their process with new practices designed to minimize the potential for similar issues to arise in the future. They emerged stronger as a team. And, they were able to build trust with their customers by handling and communicating about the issue so transparently.
To get similar results when you face similar challenges, follow the recommendations below.
Bugs can infest more than just your code
Bugs can infest your team as well. Nobody wants to create a problem with expensive consequences — but sooner or later, everyone writing software takes their turn.
It can be difficult to explain the inevitability of bugs to non-technical people. So, owning up to a bug can be stressful if your co-workers (especially managers) have a hard time understanding what happened, or why you made a costly mistake.
Frequently, the solution to a bug is a single line of code, a missing command, or even a missing character, such as a curly bracket to close a conditional, or a missing semicolon to end a line. Who among us hasn’t made that last mistake, more times than you can count?
Having to tell your boss, “I forgot a semicolon and it broke everything,” never feels good.
But it’s important for everyone — especially managers — to understand that bugs will happen. Especially with older code bases, or teams with a lot of contributing developers, problems can mount quickly and unpredictably, regardless of developers’ skill levels.
Healthy communication is vital. If your team has not fostered a community of trust, transparency and kindness, developers can feel judged by coworkers for making “stupid mistakes”. Cultivating a team culture where code problems are acceptable — because they are expected — is the best way to make sure bugs are they are identified and fixed as quickly as possible.
What programmers can do to minimize bugs
Use powerful software writing tools that analyze problems for you
Programs called “linters” analyze code after you’ve written it; use them to find syntax errors.
Integrated Development Environments (IDEs) are an even more sophisticated class of software. They automatically check syntax and analyze logic while you code, and call your attention to potential problems as you type, much like spell check.
Use syntax highlighting
Syntax highlighting means coloring keywords and symbols in code differently, depending on their syntactical role in the program. This enables programmers to more easily spot problems as they type, even without the aid of the program recognizing and pointing out the error. If the code colors don’t match your expectations, that signals there is probably a typo somewhere interfering with your syntax.
Just about all software for writing software, even basic programs, include this feature. If you’re still using Notepad or some other basic text editor to write code, you’re missing out on this powerful, basic benefit.
Do code reviews
Get your coworkers to read your code line by line with fresh eyes. They will catch some syntax errors you missed, and catch logical errors that hadn’t occurred to you.
This works if more experienced developers review younger developers’ code, for obvious reasons, but it’s a best practice for all code to be reviewed, no matter who on the team wrote it.
Most all developers have something to learn, and something to teach. And, most all developers will make mistakes that a fresh pair of eyes will catch, regardless of their experience. If nothing else, newer developers will learn a lot from reading code written by their more experienced teammates.
Anticipate the unexpected. Start with fresh data each time through a looping process, as in Pippin and Chris’ story with their use of
unset() to clear variables. Make as few assumptions as possible — ideally none — in code.
Take the time to generate errors when assumptions are violated while the program is running. You’ll earn that time back when you’re called on to debug some urgent problem that has management sweating bullets.
Recognize that coding in a team environment is social
Even if your code works fine and you understand it, others may not. That invites problems to creep in when others add new features that interact with yours. They will make assumptions (consciously or not) about how your code works and how they should interact with it.
For this reason, work together as a team and over-communicate, to make sure you’re rowing in the same direction.
Don’t be clever
Write simple code instead of elegant or complicated code, even if it makes you feel proud.
Code needs to be easy to read and understand. It’s surprisingly easy to do your coworkers (and even your future self) a disservice by being too clever.
Follow community-accepted code standards
Think of coding standards as institutional knowledge: They are accumulated wisdom of good practices to help avoid common errors.
Some developers view coding standards as busy work, or arbitrary. Don’t be one of them. They may not always make sense at first, but there’s value in consistency, even when applying rules you may not understand at first.
Think of using coding standards as a requirement for your work as a professional. Commit to learning standards for the languages you write in — and the reasons for them.
I guarantee that following standards will benefit you and others.
What can managers do to make bug management less painful?
Calling all senior developers, product owners, project managers, team leaders, CEOs, and CTOs — this stuff is for you.
First and foremost, create an atmosphere of trust and openness on the team.
Do this by validating developers even when things go wrong. Know that bugs happen, and all programmers create them. Thus, you should highly value the skill of open and clear communication in developers. Encourage your entire team to learn from all incidents together, so they build everyone’s experience.
What would Pippin’s team have gained if they had angrily lashed out at the developer who left out the
unset() command? The developer would’ve been demoralized and shamed, not motivated or corrected. They wouldn’t have been available to fire on all cylinders, at a time when that’s what the team needed most, to fix the problem and restore their customers’ trust.
Any shaming or punishment of the individual who submitted the bug should be discouraged at all levels of management. (This includes self-deprecation. We developers are often hardest on ourselves, and need reminders that we’re only human.)
To foster an open and supportive team, individuals need to feel safe to notice and report bugs. Otherwise, a developer might be more motivated to to protect their job, or the jobs of their teammates, instead of prioritizing quality code.
Recognize that software is inevitably complex, and will occasionally break
People often think of writing software as similar to building a house: You plan and build a strong foundation, erect a strong, load-bearing structure, add a layout of walls, and layer over everything with windows, doors, and finish that all fit together perfectly — and all according to plan.
Unfortunately, writing software is quite different. Houses are fairly standard. Software is generally, almost by definition, unique. So, imagine building a house without standardized materials, without common practices for most problems, and without a complete blueprint from the get-go. Imagine your builders having to invent the door, figure out how to make a window, and how they should fit together in a wall… That is a much better analogy.
In software, a significant part of what you know about the problem you are solving is learned while solving it. As projects begin, developers may have a pretty good idea of the final outcome. But, as the project progresses, the plan will change.
Because of that, it is generally inefficient to plan things out in complete detail from the start, because plans will change. As developers focus more attention on specific features and aspects of the program, they surface details that couldn’t have been anticipated or planned for. When they apply their creativity to address these details (which is often the right thing to do, depending on context), they might introduce complexity. When they check in with project managers for input (which is also often the right thing to do), the requirements can change. They may suggest ideas for improvements on the original plans, which can introduce entirely new and unplanned features. Their skills even improve during the project as they learn new techniques, which can make code written near the end of the project very different from code written at the beginning. All of these cases typically lead to places where the system doesn’t work together quite as intended.
These are called bugs. And the point is, creating them is inevitable.
This can be difficult for project managers and owners to accept, because it introduces uncertainty and risk — two things that managers are supposed to reduce. But in almost all cases, this is a firm truth.
Plan projects to accommodate wild uncertainties by rethinking estimates
We instituted a rule at Rocket Lift this past spring, on the recommendation of a good friend: When a developer gives you an estimate, triple it.
Don’t think about it. Just do it.
I ask them to give me conservative estimates that they feel confident in. I ask them to assume I’m going to use exactly the estimate they give me for project planning (securing budgets, setting client expectations, etc.). They give me numbers that they truly believe will be accurate.
Then, I assume that they’ve underestimated by a factor of 3. I know that this happens for reasons that are completely outside of their control, and have nothing to do with their skill level, so there’s no judgement.
I apply this rule 100% of the time, with everyone on the team. Sure, experience helps a little, so more senior developers tend to be more accurate, but trust me, it’s still not as accurate as you would think.
Here’s a list of things that I can say since we’ve instituted this policy:
- We regularly come in on or under budget
- We regularly meet the expectations we have set for ourselves with our clients
- My stress level has dropped precipitously
- We often finish early, which lets us to do more, or make things better than we had promised
- Our clients seem to like us a lot more
Commit to paying down technical debt
“Technical debt” is a useful way for managers to think about risk in software projects. It encourages you to think of all software programs as on a spectrum: On one end, so full of bugs that it doesn’t work at all. On the other end, 100% perfect and problem-free (a nearly impossible ideal, but always the goal).
Technical debt accrues when developers write code the first time, and get something working, but aren’t able to apply the lessons they learned in the process that will make it more stable. (This typically happens even the second or third time through as well, although the technical debt introduced tends to get less-impactful over each iteration.)
Technical debt accrues when a feature is built but isn’t fully tested: There are probably hidden bugs. It accrues whenever shortcuts are taken to meet a deadline. Similarly, it accrues whenever you hear a developer say, “I’ve got improvements in mind for when we can come back to this, but it’s good enough for now.”
Managers, train yourself to hear this and translate it in your head as “I just introduced future problems in order to meet short-term project goals.” Project goals and deadlines have your team borrowing time from the future to accelerate short-term progress.
Technical debt compounds like interest on a financial loan; the longer you wait to start “paying back” that interest, the harder it will be. And you’ll have to pay it down one way or another down the road: There will either be bugs that crop up later, or the software that worked for a while will suddenly start to let you down as business grows or conditions change.
In our house-building analogy, paying down technical debt is like stopping to re-build portions of the house that didn’t fit together correctly, now that you know the shape of the whole.
If you choose not to rewrite along the way, which time and budget constraints often dictate, you could end up stuck with a shoddy house — say, with bricks in the third story that aren’t supported by a foundation constructed out of paper. Eventually, it will collapse.
Plan for refactoring
If you plan to rewrite code along the way — to remove inconsistencies and redundancies, to reduce complexity and make code more organized — then you have a plan to manage your technical debt. I award you the highest of fives.
Commit budget, time, and resources to rewriting code for features that have already been delivered, based on developers’ recommendations and priorities. This is called refactoring.
Recognize that the developers probably have a clearer view than you of the extent of looming technical problems. Trust them when they tell you that some area of code needs attention.
If you are use an Agile methodology such as Scrum, we recommend prioritizing both bug fixes and refactoring above new features. If you don’t have this policy set in stone, it can become difficult to justify budgets for refactoring to the people holding the purse strings — despite the fact that bugs and code cruft will accumulate and compound, costing even more in the long-term to fix. (For long term projects we run on Scrum, we insist on having sprints entirely devoted to refactoring at regular intervals, such as once every 6 weeks.)
Conversely, if you do take this approach, you’ll be managing technical debt gradually, continually, rather than letting it get out of hand and becoming very painful. Think of this as a “move slow to move fast” approach.
Make sure project goals are clear
You, the manager, have perspective that developers may not have on the most important features and milestones for the project.
Ensure that the intended outcomes are clearly understood by the team. Additionally, make sure the software’s “architecture” — its overall plan, which may have emerged during the project — is as clearly understood by your team as possible. You may need to depend on your senior developer for this.
If the team shares your understanding of goals and priorities, they can bring to your attention technical issues with a high risk of threatening the project.
Be aware of coding best practices
Be familiar with our recommendations for developers, above, so that you can ensure best practices are actively used.
Make it clear that you want quality code. No sub-standard code should be acceptable. That will make your interests align with developers, which will make them happy — at least, it will make good developers happy. (This may help you identify developers that you don’t want on your team.)
Make it clear that, because you want quality code, you support the team using coding standards, professional software writing tools, code reviews, and so on.
Be good to your people
Support your people. Encourage them to aim for beautiful code always, and give them what they need to hit it. Secure time and budget resources for them to do things right.
Also, aim for transparency and trust. Foster a healthy, trusting work environment. Structure your process to encourage openness, directness, code reviews, and individual improvements.
Do these things, and you’ll not only have a happy team, but you’ll reduce your instance of bugs.
What’s worked for you?
Are you a developer, or a manager of developers? Do you agree, or disagree with our recommendations?
Share your experiences in the comments below!
Updated on October 24, 2016
Now you devoted Apple device toters can easily subscribe using your favorite podcasting app of choice!
This comes one week after the launch of HardRefresh.audio, where you can also download, listen, and manually subscribe using RSS if you prefer.
Why wait? Subscribe through iTunes here, and be sure you don’t miss the next episode!
For fans of the show reading this, we have a big favor to ask: Would you leave us a five star review?
It will take you just a few minutes, and it will help us tremendously, by helping other iTunes users find Hard Refresh. (This will help our listeners as well: By boosting our listener base, we’ll be able to produce more frequent, better episodes, as well as extend our reach to find more interesting stories to explore.)
To leave an iTunes review for Hard Refresh:
- Open iTunes and go to our show page. This magical link should take take you directly there, but if it doesn’t, just search for “Hard Refresh”.
- Click the “Ratings and Reviews” tab.
- Under “Customer Reviews”, click the “Write a Review” button.
- You got it from here.
Thank you! ❤️
Posted on September 20, 2016
WPTavern.com, a premier WordPress news sites, published a writeup on our new show!
WP Tavern Editor Sarah Gooding interviewed our executive producer Matthew Eppelsheimer. They got into the backstory for how Hard Refresh came to be, where the show’s name came from, and how we hope to stand out amongst the many other WordPress-oriented podcasts.
We also divulged some facts about the many hours of work that went into our first episode:
Many technical podcasts don’t follow this format because it’s very time-consuming. Eppelsheimer said his team spent more than 120 hours producing the first 30-minute episode, which amounts to more than four hours per minute. They conducted four interviews with the guests, held weekly editorial meetings for four months and produced four iterative drafts before publishing the final version.
We’re grateful to WP Tavern for helping us spread the word!
Updated on March 29, 2021
Rocket Lift isn’t just any sponsor. They’re our first sponsor!
But more than that — Rocket Lift is… well, us. That is to say, Hard Refresh is a project of Rocket Lift.
So, who is Rocket Lift? A web agency serving organizations that make a difference, with super-custom design, development, and marketing services. We are a talented team of designers and developers who care a whole lot about doing good work and making the world a better place. We’re based in Portland, Oregon in the United States, and work with people from all over the world.
Rocket Lift specializes in ongoing support for mission-critical websites as a long-term partner. We turn around development projects that have gone off the rails. We support teams by customizing workflow, and adding processes and systems that stabilize volatility. Managers love how we help them reduce complexity and technical debt, and thus risk.
Rocket Lift created Hard Refresh because we wanted to listen to something like it, but it didn’t exist yet. We’re thrilled to tell stories about exciting work in our industry, which is often high stakes and challenging, but usually invisible. We want to cast light on the hidden drama and glory of keeping the digital lights on.
Hard Refresh is Rocket Lift’s love letter to our industry.