Testing, Testing, 1 2 3

As we discussed in my previous blog post, the whole agile development process depends on testing – it has to be good, obviously, and it also has to be fast – we don’t have the luxury of taking months to test our flagship product. Nope, we get two weeks and two weeks only.

So, how do we test all of our code in two weeks? We cheat – we do it in four weeks! Remember, for each staggered release, we have two weeks of development and two weeks of testing. So we do some of our testing during the two weeks of development.

Actually, we do most of our testing during the two weeks of development. Moreover, testing is so critical to our development process that about half of the time developing the code is actually spent on writing automated tests for the code that we are writing!

By the time the product reaches the final two weeks of testing, it’s already been extensively tested. And since all of the previously written tests are running, we’re not just testing the code we wrote in the allotted two weeks, but also all previously written code.

Unfortunately, testing during the development phase cannot be as extensive as testing during the testing phase. Testing during the development phase, usually called “unit testing”, tests the components of the system separately from one another, but does not test all of the components integrated together. Which is why there is final testing – to find bugs that are a result of integration between the various units.

So let’s discuss the two types of testing:

  • Development testing – the testing we do during development
  • Final testing – the testing we do during the testing phase

Development Testing

Developers don’t just write code, they have to test it. There are traditionally two ways of testing code:

  • Running the application and manually testing what you wrote
  • Writing code that tests the code you just wrote

Back in the old days, most development was done the first way – by manually testing the application. It’s the easiest way, the fastest way, and when the application isn’t too large, it’s good enough. And when all we had was 640K of memory, the application really couldn’t be large.

But applications have grown larger (and larger, and larger). To really test new programs, you can’t just run them on your computer in seconds. Nope. Their startup time is measured in minutes. All developers rely on a fast code/compile/run/find bugs/debug cycle. If that cycle is measured in minutes, their productivity is dampened in a major way.

It gets even more complicated – most new applications are not self-contained. They need a web-server to run, a database to connect to, and lots of other external components. In our case, we need a whole VM infrastructure (VMWare) to run. This means that the developer station cannot contain all of these components. And running them all together makes testing v..e…r…y….. s……l…….o………w…………

Not agile at all.

So, what to do? Unit testing. In unit testing we takes chunks of our code (a “unit”) and we test it by writing more code that runs the unit and checks that it’s doing what it needs to do. For example – if we have code that tests for the validity of a phone number, we write a test that pits it against various valid and invalid phone numbers, and check that it returns the correct result. Another example is code that registers a user. We write a test that runs the code and supplies it with various users to register (e.g. a valid user, an invalid user, an existing user…) and check that the result is OK – the correct data is to be found in the database. A third example would be testing the code that creates an environment.


The second and third example are important. Let’s start by looking at the second example. In it, the code that is being tested is accessing the database. But, didn’t I say above that accessing external components is precisely what we are trying to avoid with unit testing?

Well, yes and no. We’re trying to avoid running all of the code with all of the external components. But if an external component is fast enough and small enough, we’re good with it. So, database, yes. But VMWare, no way! VMWare is too big to run on a developer machine, and while it is very fast, it is not fast enough for our unit tests.

So how do we run the third example, the test that checks that creating an environment works? We mock.

No, not that kind of mocking! The process of “mocking” is the process of creating code that fakes an external component. You can think of it as code that simulates whatever external component we need.

We don’t want to use VMWare in our tests so we create code (and lots of it) that simulates exactly how VMWare works. No, it won’t run a VM, but it will create, modify and delete the files that VMWare uses (using a mock filesystem running in RAM). It will “create snapshots” and “run” a virtual machine. And if a virtual machine is not registered, then it will refuse to “run” it. Just like VMWare does. Even more – since we’re writing the code, we can write all sorts of checks and assertions that even VMWare doesn’t have, and check our code more thoroughly than when testing using the real VMWare.

And VMWare is not the only external component we mock. We use storage managers like NetApp extensively, so we mock that. We have an external component that handles our routing needs, so we mock that too. In the end, we can run and test all our code without any external components.

It gets better – remember the three components of the CloudShare Universe? The Frontend, the Backend, and the VMWare infrastructure? We have already seen that testing the backend means mocking the VMWare infrastructure. But what about the Frontend? To test it, we also need to run the Backend code. But for the Frontend, the Backend feels like an external component.

So, the Frontend mocks the Backend! The Frontend team have created an elaborate simulation of the Backend, which behaves like the Backend, only much much faster. It can even simulate cases where the Backend fails, just to test what the behavior of the GUI is in these cases.


My editor told me not to write posts that are too long, and I haven’t yet covered Final Testing. Aaaarrggghh! Oh, well, see you next time, when we’ll be discussing Final Testing.