At the GOOSGaggle event last weekend, Nat Pryce gave a presentation on “System Test Driven Development” — an approach he uses to apply TDD when working with complex heterogeneous infrastructures.
According to Pryce, people often jump into unit testing the business domain core of their context, and with complicated technical infrastructures such as in heterogeneous financial systems the result is that the domain model often doesn’t really fit into how it needs to plug into the infrastructure. Sometimes the resulting design is impossible to integrate at the end.
Pryce recommended an alternative approach when working with complicated infrastructure: start with end-to-end system tests and a very rough model of the core (“only enough of the process that it needs”). Then “jump in and out of the domain level [implementing scenarios] to ensure easy integration”. In this approach, end-to-end technical tests are used to guide the development of whole feature sets, very similar to how high level acceptance tests are used when the risk is in the complexity of the business domain. They are used to guide development at a larger scale, pointing out which details need to be fleshed out by unit tests and helping to avoid the “TDD equivalent of analysis by paralysis”, where teams keep writing unit tests for all possible cases even if they don’t contribute to the design. Jim Shore, in his article Microsoft Gets TDD Completely Wrong, wrote that figuring out what test will best move your code towards completion is the hardest step for beginners. End-to-end system tests can certainly help with this.
Pryce gave several examples where system end-to-end tests pointed to infrastructural solutions and configuration changes as alternatives to developing custom software. With complicated technical infrastructures, some existing components might be able to provide parts of the required functionality. In one case 20000 lines of Java code was replaced with a few XSLT files that transformed messages in a message broker. The system tests ensured that the system still gave the same functionality. He proposed that starting with system end-to-end tests might help point out similar situations in the future where such unnecessary development won’t be done in the first place.
Isolation
This approach isn’t without challenges, though. “In a large system, your system fits into a set of byzantine systems controlled by someone else, chosen for political reasons and hard or impossible to test”, said Pryce. Major challenges to implementing an end-to-end system test efficiently are scaling to handle computing intensive tasks, reliably handling asynchronous operations, concurrency and distribution. Figuring out where exactly are the borders of your sub-system and isolating it from the rest of the world are key decisions for this approach to succeed. Pryce provided a rule of thumb: “If I can change it or replace it, it’s part of my system”.
He said that it is crucial to simplify byzantine APIs and protocols, in order to speed up development but also to provide natural isolation points. This approach is very similar to Anti-corruption layers from Domain Driven Design, with a difference that Pryce advises isolating at the level of the simpler protocol rather than providing an alternative implementation of the simpler API. Such isolation allowed him to keep more code under test (the real implementation of the simpler API) and also to choose whether to run tests against a fake test dependency or a real system. Isolating at the protocol level also allowed him to control the load that automated tests were putting on other real components.
Pryce advised running overnight tests again snapshot releases of external systems to get early warning of release candidate problems. Talking about a project for a financial institution where he applied this approach, he said that they got it to such a level that teams working on legacy components checked build monitors of his team before signing off a release of their systems.
Side-effect: systems easy to support
Because of a large number of moving parts, implementing end-to-end tests reliably requires us to be able to answer the following questions:
- Where are the system edges?
- What is the system doing?
- Has it stopped doing it yet?
- Has it failed?
- Why has it failed?
The same answers are important for supporting the system in production. Because of that, Pryce concluded that development driven by system tests led to a system that is very easy to support. Implementing end-to-end system tests reliably requires us to design and implement modules, provide integration and inspection points to answer these questions. The same mechanisms can be used in production to monitor and support the system. Similar to how a side effect of (unit) TDD is a system that is easy to modify, end-to-end system test driven development leads to a system easy to support.
