Testing

Let's believe for a minute we build German cars. Those are the best cars in the world. Let's mentally go to one of those factories. Everything is in order, clean. There are no oil spills on the floor.  They certify everything: screws, engines and seats.

Let's see what it would feel like in our world.

Software factory

Welcome to Kode Meister GmbH

We are agile people and use some iterative methodology. Our iterations are a week long. We have 3 developers and 1 tester. The developers finish between 3 and 5 features and solve between 5 and 10 bugs per week. For each feature and bug between 5 and 10 tests are defined. The tester must execute them all before the week ends. Our numbers are:

Average	Value
Feature finished per iteration	3.6
Bugs solved per iteration	7.4
Tests per feature	6.92
Tests per bug	7.03
Tests per iteration	76.9
Minutes to perform a test	2

The numbers have been randomly generated with this.

Our developers constantly break existing features and previously solved bugs so in each iteration we must execute the tests for previous iterations (regression tests). Only 10 weeks after we started, this is what we have:

At this point the tester is overwhelmed by the number of tests. With this progression we will need more testers soon. Do you honestly believe your human testers and developers can keep up with this? You might even solve unemployment issues in Europe.

What's really bad is that it will keep growing and to make it worst, we are not being that rigorous.

You could say: that cannot be done, we cannot test this much, we'll just have to be more careful when developing. Ok, if you believe that: Go ahead to the comments and troll. Nothing from what comes next will be interesting for you. But keep in mind...you won't be making German cars no more.

If you wanna be spared of the mathematics jump to the solution.

The mathematics behind.

Let $f_{\mathrm{i\; j}}$ be the feature $j$ in the iteration $i$, $F_{\mathrm{i}}=\left\{f_{\mathrm{i\; j}}\right\}$ the set with all features of the iteration $i$, $b_{\mathrm{i\; j}}$ the bug $j$ in the iteration $i$, $B_{\mathrm{i}}=\left\{b_{\mathrm{i\; j}}\right\}$ the set with all the bugs of the iteration $i$, $t\left(x\right)$ the number of test needed for feature or bug $x$. For our first iteration we will have a total of $s_0$ tests, where:
$$s_0=\sum t\left(F_0\right)+\sum t\left(B_0\right)$$ The regression tests for the iteration $i$ will be:
$$r_{\mathrm{i}}=s_{\mathrm{i}-1}$$ Generalizing our simple model, the total number of tests:
$$s_{\mathrm{i}}=\sum t\left(F_{\mathrm{i}}\right)+\sum t\left(B_{\mathrm{i}}\right)+r_{\mathrm{i}}$$ Using the numbers from Kode Meister GmbH we have:

To make it worst

If you think you can pay for previous. Perhaps you can afford being more rigorous as well. Before closing a feature or a bug to ensure nothing has been broken the regression tests must be executed. This means $\left|F_{\mathrm{i}}\right|$ times more that regression tests should be executed. Same goes for the bugs: $\left|B_{\mathrm{i}}\right|$. Finally the testers must run all the tests again, that is one more time. This new formula would be:
$$s_{\mathrm{i}}=\sum t\left(F_{\mathrm{i}}\right)+\sum t\left(B_{\mathrm{i}}\right)+\left(\left|F_{\mathrm{i}}\right|+\left|B_{\mathrm{i}}\right|+1\right)r_{\mathrm{i}}$$
Using same numbers again we have this values:

And this graph, now in a ${Log}_{10}$ scale:

It has gone from an arithmetic progression to a geometric one. Still think you can afford it?

Automize your tests

Testing adds an additional cost to your product, a big one. But if you want quality there is nothing else that would provide as much. I believe that in the long run it even reduces costs. It is not easy, some times you'd have a lot of ideas you won't be able to take out of your head because testing gets in the way. Think of it this way: better testing in the way, than lots of broken features.

I plan to dive deeply into testing. It basically goes like this:

Integration Tests

This is where I would start. Developers would create small scenarios were they would test how components interact together. They would use actual databases, files, network connections. Some components might be mocked. This are very cost effective. Should be constantly executed but they are slow so maybe not that often. They should be executed from the CI.

End to end tests

This is the coarse grain tests. They test against the whole system like a final user would. There are tools which allow automatizing application usage. There are tools to record actions and then replay them. The downside is speed. They require the whole environment to be running. They made changes in the databases and take about a fraction of what a test user could take to perform the whole regression test list. You could try to perform them a couple of times a day.

Functional tests

Finer than previous. Developers, working together with testers would create testing scenarios (small versions of the program). Testers would create and modify test cases, which are input and expected results. This tests are fully automatic and should be faster than previous. They could be a table with input values and expected outputs. They could be run with the CI.

Boundary tests

Used to test all the boundaries of external libraries. That is all features you use. Use it as the learning process, then keep the tests. It's important to run them when upgrading to ensure nothing has been broken. Should not be executed very often.

Unit tests

My favorites. Unfortunately, they are very very expensive. Prerequisites are astonishing. In the end, they really provide very little information about the whole. They are the best support for refactoring and keep the technical debt at bay. They also provide component specification. Must be run constantly, on every single change. They must be blazing fast. Tiny tiny tiny. Only a single component should be tested at the time, all its dependencies must be mocked.