I wish I could say I found out about the Heartbleed bug very quickly and from some super deep technical source. But I didn’t. I found it on my favorite Tmblr blog, filled with animated gifs about DevOps that gets me through the day. And once I realized what this particular Gif was telling me, I too was in a small state of shock.
As I went through all the phases of grieving, and hit the “move on” phase. I realized. Oh wow. The number of websites that need to now be patched, published, users notified, and damage control and assessment run is staggering. And because the owners of these applications have to respond immediately there isn’t much time to do it the right way.
This makes you realize very quickly that Heartbleed is now a massively large QA problem for much of the interwebs. But this is not just your standard functional QA problem. This problem requires configuration management, and testing of not only the application layer but also the infrastructure it sits on. So your standard functional testing environment especially wont help you. And your integration and unit testing environment, if on-prem, can’t be used to test because it’s in your local network, and not an accurate test of intrusion. So what do you do?
The job is to replicate not only your code base, but the configuration, and infrastructure it’s on in an isolated environment. Once you have that you need to test the whole suite probably w/o the Heartbleed patch and with. So you will need to incorporate some sample data and user accounts as well that are not high risk. And once you are sure that the patch did the job, make sure that your typical functionality, especially that associated with certificates such as logins, data transport and all are intact. Oh and do this all before your site becomes a target. Meaning get the release out two days ago.
The only way this can be done is in the Cloud. And that is exactly what a lot of organizations are doing. They create exact copies of their infrastructure in cloud environments that are separate from on-premise integration environments, and their production environments. They may create several duplicate instances of this infrastructure in order to test with the bug and without. And finally they load all the automation tools for unit, integration, and functional testing in these environments so ALL tests can be run, just in case.
To do this rapidly requires some smarts on the infrastructure side. A cloud service that has a full blown API, and templated environments available to use. Otherwise provisioning, and duplicating instances of your production environment in the cloud will take days, that you do not have. The cloud service must also allow you to have full control of networking layers, and even the ability to test attacks on your site to verify that the all clear bell can be rang. This includes being able to do things such as creating multiple vLAN in the same environment, and performing packet sniffing. And finally the environment must be accessible to the public web and safe from your existing infrastructure.
There are not many cloud services built for this type of QA. Purely functional testing environments wont allow you to test the configuration of the patch, only how it appears to the users. And on-premise unit and integration testing environments are too cumbersome to do any testing like this in a short amount of time, without your IT getting deeply involved.
Once you have established confidence with the cloud-based instance of your newly patched applications front-end, back-end, and infrastructure, it’s safe to move to production.
It’s good to know solutions like CloudShare exist to help web application developers quickly react to these types of issues without setting up infrastructure, and systems manually. To find out more about Heartbleed bug and patch visit heartbleed.com. And to sign up for a complete suite of rapid testing infrastructure visit here.