Why Security Vulnerability Disclosures Aren’t Easy to Do

Disclosing security vulnerabilities publicly is absolutely essential, but it’s nearly impossible to do it without upsetting some people.

For some, the WordPress update to version 4.2.3 caused, to put it lightly, a bit of a headache. (Listen to Richard Tape’s account here!)

The fallout from that update cascaded through developers all the way to front-end users. And that’s what you never want: for your front-end clients to experience site malfunction. We call these, “breaking changes,” or a “break in backward compatibility.” There was a lot of discussion in the community about why it happened, what steps could have been taken earlier to avoid panic and frustration, and why it felt to WordPress users that this update was sprung upon them, like a trap.

An update that causes surprising breaking changes is like your landlord finding out that your apartment building is made of super flammable material, like tissue paper. And because your landlord doesn’t want the arsonist down the street to realize that the apartment building is a tinderbox, she comes over in the middle of the night and replaces everybody’s walls with brick. How nice of her!

But now you can’t hang any of your pictures on the walls! And what if you don’t really like the color of brick? Also, when your landlord came by she actually knocked some of your pictures off the walls and broke them! You storm out of your apartment in a righteous rage, but your landlord is sitting exhausted in the lobby and says, “Sorry buddy, but I got here just in time. Joe the Arsonist was already on his way. I had to be sneaky, and I had to be fast.”

A little warning might have been nice

But “warning” is a double edged sword. If your landlord broadcasted to the neighborhood, “Hey! I’m going to be walling up this building with brick here soon. No big deal, nothing to see here, but yeah. Brick! And soon!” Joe the Arsonist is going to be like, “Huh? Brick? Wait a second, what are these buildings currently made of? Tissue paper!? Oh, this is going to be fuuuuunnnnn.”

If nothing else, publishing any kind of warning or advisory tells an attacker what to focus on, and possibly how to exploit whatever is about to be fixed. There’s unfortunately no way around this though, especially with an open source codebase. As soon as your landlord swaps out the tissue paper for brick, Joe is going to be looking around at the other buildings that don’t yet have brick.

As soon as you publish the change that fixes a vulnerability, even without otherwise calling any attention to it, a spotlight is shone on that specific code. If someone knows how to read the code, and can work through the ramifications of the change, they’ve got all the information they need to figure out what the vulnerability was, and how it might be exploited.

My house is made of brick now. So, now what?

Now you spread the word to your other neighbors. In the WordPress world, code is “open source,” which means the code is published openly; it’s free for users to develop, use and manipulate as they please. This goes for everyone, regardless of their personal motivations.

Publishing a code release with an explicit security patch shines a spotlight on the problem. Doing so gives users only a tiny window of time to update their site before “Black Hat Hackers” have the opportunity to exploit the security hole. This is why it is so important to keep your site’s WordPress core software and plugins up to date.

It’s vital for developers to be honest and upfront with users about security vulnerabilities. That goes for bugs in general, but especially with security-related things. Publishing security advisories is a must, but deciding how and when is delicate and nuanced.

Even with closed-source software, keeping mum about things only adds layers of obscurity, rather than keeping the existence of the vulnerability a true secret.

What could the WordPress core developers have done differently?

Possibly, not much. There may have been some way for them to communicate some information ahead of time, but that would risk tipping their hand and calling attention to the vulnerability.

There are protocols in place for even average WordPress users to notify core contributors if they suspect they’ve discovered a bug. There’s usually a period of limbo, where the vulnerability, the change that fixes it, and all conversation about it remains private right up until the shipping time.

Once you publish the fix, all your cards are on the table, completely. The “bad guys” and the “good guys” get everything at the same time. Now it’s a race, between the “bad guys” trying to craft exploits and compromise sites, and the “good guys” trying to get the update installed so they’re no longer vulnerable.

A heads up for the good guys is the same as a heads up for the bad guys

Your landlord could have gone around knocking on doors and said coyly, “You might want to pull all of the paintings and pictures off of your walls. Just saying.” Joe would have wondered what was about to go down.
If the WP Core team had published info ahead of time about how shortcodes needed to be changed, that’s still basically telling the world, “there’s something funky with this part of our codebase.”

Even as normal developers and administrators would be looking at this information and figuring out “what does this mean for my shortcodes?”, exploit authors will be figuring out “why did this need to happen? What is it about how the old shortcodes worked that’s broken, and how might I exploit this?” It kicks off the same race, except the “good guys” aren’t able to actually protect themselves. The patch hasn’t been released yet.

This is actually a huge reason why the core team has made automatic updates such a huge priority. Sites that update themselves will always be faster at improving their code than sites requiring human intervention. When updates install automatically, this window of time between “an exploit is made known” and “the exploit no longer works” gets drastically narrowed.

Get your helpful landlord a glass of lemonade

I think the Core team was stuck between a rock and a hard place here, and I think they probably made the correct decision.

If you have to choose between shipping a fix that’ll protect millions of your least self-sufficient users but will break the sites of your hundred or thousand most savvy users, vs holding back the fix for some unknown amount of time to solve a truly complex problem in order to “do it right” while leaving everybody vulnerable, I’d choose to just ship the thing ASAP, too.

All this said, there was one point of communication failure that the core team could have done better. Any breaking change like this must be highlighted when it gets released. You can’t assume that everybody will read a release’s changelog before they install an update. It’s important to highlight to users what features are likely to break before the update gets installed, and especially when it’s a point-release where there’s an entirely reasonable assumption that there’ll be no breaking API changes.

Sometimes a bug is a feature. Bugs are generally “bad” and don’t add value, but in this case, the vulnerability itself was not just a vector for exploit, but also a useful behavior that people relied on. Users were manipulating the shortcodes in clever ways to enhance their personal sites. Because the bug presented a security vulnerability, they had no choice but to fix it. And in order to fix this bug, the core team had to remove the feature.

What do you do if you suspect your apartment building is made of tissue?

If you believe that you have found a security hole either in your own site or someone else’s, the best way to proceed is to communicate privately with the core team. Do your best to do so securely, as well. Use the project’s security-specific reporting mechanism, rather than posting to any public support channels or bulletin boards.

Make sure you’re also giving them valid contact information. You don’t have to use your “real name”, or anything else that’s tied to any of your established identities, but make sure you’ll actually see any messages they send you in response to your report. They’re almost certain to have questions for you, and may need your help testing the fix. If nothing else, it gives them the opportunity to thank you for helping them out and saving millions of people a mountain of grief.