In this context I hear often the argument that the majority of tests should be UnitTests over all others. Mike Cohn suggests this as well with the layout of his Testing Pyramid, because UnitTests can be executed faster. Robert C. Martin (Uncle Bob) is a strong advocate of this too – see one of his recent blog entries. Later on I will come back to his blog post and his statement that a UnitTest framework must not slow down the developers’ pace. But first to the “argument” that UnitTests can be executed faster than the others:
When I talk in the following text only about End-To-End (E2E) tests, then as a pars pro toto for all kind of tests that check not just a single unit, but complex business logic or complete features.
So shall we write UnitTests “only” because they are executed faster than E2E tests? This sounds to me like some kind of a workaround. Because we cannot get easily execute broader applied tests fast, we test something that we can test faster? Exaggeratory, isn’t it somehow similar to the old joke below?
Shouldn’t we better try to solve the primary problem: Why do take the E2E tests take that long? Isn’t writing automated tests and a slow execution time some kind of normal software usage with a performance problem? What do we normally do, when we have a performance problem? We make a detailed analysis with a profiler or a similar tool and search for the bottleneck(s). That area that consumes most of the time is optimized until we have reached a point that we are satisfied. During this process of optimization, I think, one should consider the following questions as well: Do we really need that stack of framework over framework where each layer introduces additional execution time on top of everything? (As a side question, do you know what your CPU is doing within one second of your application running time? A typical PC-CPU can today execute between 3,000,000,000 and 18,000,000,000 instructions per second depending on the number of cores, sometime even more if it can parallelize certain ones. Well, I know that the OS needs its time too and that a certain amount of time the CPU just sits there and waits for the RAM, but never the less a question that one should ask him/herself.) As well we might improve the overall performance by using faster HW: e.g. SSD, RAM-Disks, or parallelize the tests over more machines when we have reached the point that not a single bottleneck is left but only a flat area of many functions that need nearly an equivalent amount of time? If the bottlenecks are inside a framework, we should check there for the problems. Otherwise, if the problem is because of a used protocol, we should consider using a more efficient one. (In a future post I will describe in detail how we handled in our company the slow functional test execution problem of our application.)
The nice side effect of such an optimization is as well that the product is faster for the end user and probably its power consumption overall is lower, because operations are executed faster and the CPU can throttle down in a low power mode earlier.
So when the performance problems are solved, we can put our focus back on tests that verify directly business value.
At the end I want to follow up on Robert Martin’s statement: “Slow running tests represent a design flaw that reflects on the experience and professionalism of the team. A slow running test suite is either a freshman error, or just plain carelessness.” From my point of view this is too simplistic. I think any team has to balance the features and possibilities that a slower UnitTest framework gives, against the compile and link time. I don’t know which framework the developers where using, but I know that one of the currently most powerful ones for C++ is GoogleTest. Because of its power, it probably compiles and links slower than others. But I personally do not want to miss the excellent reporting features and capabilities of templated test cases. Without them I would not get these detailed failure reports (hunting down a problem with a less detailed report would take longer) and I would have to write and maintain in the long term much more non-template test code.
At the end only the features – of course fully functional without the creation of legacy code – delivered to the customer and its development time count. If a team decides to accept longer UnitTest compile and link times, because they can achieve nevertheless in the same time more value to the customer, then this is – from my point of view – a correct and professional decision.
Many thanks to Jim Coplien for the inspiring discussion over the last months and for reviewing this text!