nowucca.com - personal software technology blog
There has been an uptake of continuous integration systems (Bamboo, jenkins) in recent years. The goal is to automate a graph of dependent component builds, generating binary artifacts and associated component documentation, test results etc. At one extreme, some companies use these systems as a pipeline to push to production from every developer checkin, or be ready to push to production at predefined times every day.
To achieve close to that extreme, it becomes necessary to heavily optimize the speed of your tests. This is usually done by ensuring that the testing is done in-memory, using in-memory databases, email servers and other testware and mock frameworks. To make use of these mocks and testware, one learns to code to interfaces, or at least APIs that can be easily mocked.
In the effort for speed, what if something goes wrong and a test fails? The idea of progressively-enhancing tests in a deductive manner is to run test suites in phases that following the logic of what a developer investigating a failure would take.
Phase 1: "Lightning": developers maintain tests and they succeed more than they fail, so perform a full run optimized for success and speed.
Phase 2: "Sherlock": the first thing a developer now does is run failing tests again in isolation, so run just the failing tests again. If they now succeed, we likely have a test-ware or intermittent issue. Yuck, but at least we found out quickly. If the tests still fail, we can proceed to the next phase.
Phase 3: "Watson": the developer will now want more information on the failing tests, so we re-run the tests but with debug logging turned on for all the test-ware components to gather as much evidence as possible for the failure. If they now succeed, we have a test-ware issue, or a logging issue. Yuck again. If the tests fail again, we have more information now.
Phase 4: "Deductions": the developer will now want to examine all the evidence, arrange to learn and write helper scripts to analyze the evidence over time as developers see certain classes of failures, and put summary information about failures front-and-center.
We should AUTOMATE THIS LOGIC that developers use to investigate issues into our continuous integration systems.