Software that runs a business unattended has to be boring. A scheduled job fires once. A message is handled once. A report shows each person only their own data. You prove that with tests, and then you have a new problem: who tests the tests?
Here are three times our own tooling told us everything was fine when it was not, and the habits we took from each.
"Duplicates: 0"
We have a test that floods the platform with messages and then checks that none was handled twice. Its first run printed a clean line: duplicates, zero.
It had found zero duplicates because it had found zero of its own messages. The log it searched came back empty, and counting the duplicates in nothing gives you nothing.
A zero from "looked and found none" and a zero from "could not look" are the same character on the screen. The test now has a third answer, inconclusive, and it gives that answer whenever it cannot find its own traffic. An inconclusive run is not a pass.
Nine fires, no rows
Customers run Camaan as more than one copy of the engine, so that losing one does not stop the work. That raises a plain question: if two copies both see that a job is due at nine o'clock, does it run once or twice?
We built a small rig to find out. Two engines, one database, one job every minute. The answer was 2.00 runs per minute. Each copy had been keeping its own clock.
The fix is a queue in the database that both copies claim work from, with a token on each claim so a late finisher cannot overwrite a newer one. After it, the rig read 1.00.
The rig kept going. To count runs it compared two sources, the engine's log and the database table of run history. The log said nine. The table said none. Run history was being kept in memory and never written down. Nobody had asked for that test. It fell out of counting the same thing two ways.
The habit: when you can measure something from two independent places, do it, and treat a disagreement as a finding.
Green on the laptop
Developers run tests against a small embedded database because it is quick. Production runs on Postgres. The two agree almost everywhere.
Almost. A sum of whole numbers comes back as a different type. A date sent as text is accepted by one and refused by the other. Each of these passed every test on a laptop and failed the first time a customer used the feature.
Now any query that has to work on both databases gets a test that runs on the real one, and the checks that guard concurrency run on both.
Performance features come from outages
It would be nice to say our limits were designed from a model of load. They were not. A dashboard with two dozen tiles once sent two dozen heavy queries to the data store at the same moment, and the store ran out of memory.
What came out of that incident is what the product does today: a cap on how many queries one page may run at once, shared results for tiles that ask the same question, paging done by the server, and a cache whose key includes what the viewer is allowed to see.
What we do differently
- A check that cannot find its evidence reports inconclusive, never zero.
- Anything that must happen once is tested with two engines, not one.
- Counts are taken from two sources where two exist.
- Queries are tested on the database customers run.
- A measurement that looks too good, or too bad, is rerun before anyone quotes it.
None of this is clever. It is the difference between a report that says "passed" and one you would stake a customer's month-end on.