One Signal, Two Worlds

Premise

They all have the same shape. Something reports a state that is [technically] true, and there are two different worlds in which that same report makes sense. The code, the document, or the reader picks one of them. Usually the wrong one.

None of these are hard bugs. They are all cheap to find once you know to look, and expensive precisely because nothing prompts you to.

First, a missing file means two things

os.IsNotExist(err) pkg/storage

Observed On open, snapshot.json is not on disk.

World A

New database. Nothing written yet. Start empty, report success.

World B

The data exists, in the other snapshot format. Starting empty discards all of it.

Cost. The loader picked World A unconditionally. My own deployment guide told operators to roll back with GRAPHDB_STORAGE_MODE=json, which walks them straight into World B - an empty database, reported as a clean start.

step 1, mmap close:  [snapshot.mmap (904B)  wal/wal.log (0B)]
step 2, json mode sees 0 nodes          <- 3 nodes went in
step 3, json close:  [snapshot.json (452B)  snapshot.mmap (904B)  wal/wal.log (0B)]
step 4, mmap mode sees 3 nodes          <- the node from step 3 is gone

Exhibit A: three nodes in, zero nodes out, exit code 0

The asymmetry is the tell. mmap mode does check for the other format and falls back, so a legacy database migrates upward fine. The JSON path never looked. One direction distinguishes the two worlds and the other does not.

Second, a test that agreed with itself

id1 == id2 crash-recovery test

Observed Both assertions pass. The node written before the snapshot and the node written after it are each found by vector search.

World A

The index rebuild ran over the final post-replay node set, which is exactly what the test claims to prove.

World B

There is only one node. Both assertions matched it, because both IDs are 1 and k=1 returns it either way.

Cost. World B, for nearly two months. The middle session was built from a bare config, so it opened the store on the JSON path - case one, from the inside. It got an empty database, nextNodeID restarted, and the second node was handed the first node's ID.

after session1: snapshot.mmap (780 bytes)  wal/wal.log (0 bytes)
SESSION 2 OPENED WITH NodeCount=0  <- session1 wrote 1 node
id1=1 id2=1  collision=true
SESSION 3 NodeCount=1
search[1,0,0] -> [{1 0}]   search[0,1,0] -> [{1 1}]

Exhibit B: two node IDs, both of them 1, and a green test

This test was not neglected. It was written deliberately, with a comment explaining the subtle thing it existed to catch. It became wrong later, on the day an unrelated default flipped, and nothing announced it. A passing test is the one artefact nobody re-reads.

Adding a guard to production code turned out to be the cheapest way to audit a test nobody had watched fail.

Third, a build failure that was not the build's

go build ./... → 7 errors a working tree

Observed The project's most basic command fails, in a directory that is unquestionably the project.

World A

The repository is broken. Anyone who clones it hits this immediately. Fix it before showing anyone.

World B

The repository is fine. The working tree holds a gitignored directory that no checkout contains.

Cost. The model picked World A and recommended fixing it as the single most important task before publishing the project. It was World B. The repair would have changed nothing a visitor sees, and could not have been committed at all - the path is in .gitignore.

$ git archive HEAD | tar -x -C /tmp/clean
$ ls /tmp/clean/enterprise-plugins
ls: cannot access 'enterprise-plugins': No such file or directory
$ go build ./...
exit 0, no output

# positive control - can this check report the opposite?
$ echo 'package main; func main(){}' > cmd/server/zz_control.go && go build ./cmd/server/
exit 1  main redeclared in this block

Exhibit C: the worktree and the checkout are different artefacts, and they print identical failures

Worse, my own orientation file asserted World A in writing, so the false claim was not merely available, it was documented. A wrong note there costs every future reader the same mistake, which is a strictly worse failure mode than a wrong line of code. Code gets tested.

Finally, a number that was a subset

"nine" a session handoff

Observed A handoff opens: "Nine defects were found, and five of them were in the instruments rather than in the system."

World A

Nine is the total, five is a subset of it, and the ratio is the headline - more than half the defects were in the tests.

World B

Thirteen is the total. Nine is the coupling subset. Five counts a different population entirely - hollow tests, not defects.

Cost. The model quoted "13 defects, 5 in the instruments" three times in one day - a figure belonging to neither world - and proposed it as the headline of this piece. The source that settles it sat four rows further down the same file: "9 of 13 defects were coupling defects."

The real taxonomy is more interesting than the ratio that nearly got published. Of the thirteen: nine were coupling defects at the seams between components, one was inside a single component, three were in process or in the contract of a test. Separately, five existing tests turned out to be decoration.

Note where the error lived. Every claim below that summary line cited a pull request. The summary cited nothing, and it is the line everyone quotes, because it is the line written to be trusted.

In short, instruments beat diligence

Practice

None of these were fixed by looking harder. Each one needed a second, independent observation that the two worlds disagree about, which is a design problem and not a diligence problem. Five things that actually work:

Refuse, don't guess When one condition carries two meanings and you cannot tell which, the correct behaviour is an error, not a default. The loader now refuses a JSON-mode open when the other snapshot is sitting next to it, and names the file.
Positive control "No output" and "the instrument is broken" look identical. Before you believe a negative, make the check report the opposite on purpose. A deliberately broken file proved the clean build could still fail.
Name the artefact A result is about the thing that was actually read - a worktree, an archive, a tag, a suppression list. Put the artefact in the verdict, because a pass on the wrong one is indistinguishable from a pass.
Watch it fail Every fix lands with a test seen failing against the pre-fix code, and the pull request records what it printed. A test that has never been red is an assertion about the world, not evidence of it.
Pin the collision The repaired crash-recovery test now asserts id1 != id2. It cannot pass on a single node again - the shortcut that faked the result is now itself a failure.

Anyway

Closing

It is tempting to report a tally - we found N bugs, M of them in the tests - and case four is exactly what that instinct earns you. A count compresses away the only part that transfers.

What transfers is the shape. Ask it of any check you have: is there a second world in which this same output is fine, or in which it is catastrophic? If there is, the check is not an instrument yet.

Three of the four were caught by machinery built for the purpose.** The fourth was caught by reading four rows further down.

* The Edge of Reason - What LLMs Can and Cannot Do [paste the article URL]

** mutation testing, coupling coverage, per-package coverage floors measured on CI, invariant checkers, fault injection, and a self-test on every gate

Numbers and transcripts are my own - the defects, unfortunately, also.

graphdb is an open-source graph database written from scratch in Go - LSM-tree storage, a write-ahead log, an mmap-backed snapshot format with lazy reopen, and multi-tenant isolation. All four defects above are in the repository, as are the pull requests that closed them.