#Nine fields the settings page could read
Forge's streaming extension has a configuration struct with a PingInterval field, defaulting to thirty seconds. The dashboard's settings view reads it and displays it, so a running deployment would have told you, accurately, that its ping interval was thirty seconds. No ping had ever been sent.
I found that on Saturday alongside eight other config fields in the same condition, in the course of a commit that ended at 16,638 insertions across 55 files. Its first line reads: "The extension shipped several well-built subsystems that nothing called." By that point it was the fourth time in five days I had hit the same shape in a different repository, and four is where I stopped filing it as coincidence.
#The four
Monday, in a private service built on Forge. The router decides whether a controller contributes middleware by asking whether it satisfies an interface:
1// ControllerWithMiddleware applies middleware to all routes.2type ControllerWithMiddleware interface {3 Controller4 Middleware() []Middleware5}
Singular. Every controller in that service declared Middlewares(). A method with the wrong name satisfies nothing, so the type assertion failed and the router proceeded with the empty middleware set it already had. No controller middleware had ever been applied, which meant the API guard was installed, configured, and serving data to anyone who knew the address. That had been true since the first controller in the repository. Adding a compile-time assertion against the interface is what made it visible, and the line I wrote in that commit message is the one I keep returning to: a guard that exists and does nothing is worse than none, because the API looks protected.
Wednesday, a Seed field on a scenario record. Written, stored, published through the API, and read by nothing. There was no seed on the solver contract, none on the request types, and no crate in the Rust workspace draws a random number, so there was nothing for it to determine. Two scenarios differing only in their seed produced byte-identical output while each claimed a distinct binding.
Friday, in octopus: gateway.max_body_size. Documented since the day it was introduced. Validated at startup as non-zero, so a config that set it to 0 was rejected with a clear message. Enforced nowhere. The handler collected every request body with no cap, so one large POST could exhaust gateway memory.
Then Saturday's streaming extension. The auth package was imported by no file in the repository, so every deployment ran with no room or message authorization at all. processMessage, which is where rate limiting and validation live, had zero callers, leaving an unmetered path from any socket to a broadcast. JoinRoom never wrote membership through to the room store, so GetRoomMembers was permanently empty and the per-user room and channel limits could never trip. Plus the nine config fields.
#What they have in common
None of the four was unreferenced. That is the whole of it. The seed was read by a serializer. max_body_size was read by a validator. PingInterval was read by a settings view. The controllers' Middlewares() method had a body that called into a real, correctly constructed guard, so from inside its own file it looked exactly like working code.
Each of them was reachable from somewhere that describes a value rather than acting on it. A validator, a serializer, a settings view, a documentation page: every one of those is a reader. Grep finds them. The compiler is satisfied. Dead-code analysis stays quiet, correctly, because by the definition it is working from the code is live. The reference exists and it carries no obligation.
A guard that cannot fire reads as coverage.
Validation is the worst of the four, and octopus is the case that shows why. Something checked max_body_size. It checked it at startup, it would reject a bad value, and it would print a sensible error. All of that is evidence of care, and none of it is evidence of enforcement. I had looked at that config block before and come away reassured by exactly the wrong artifact.
#What actually found them
Not a test, in any of the four cases, and not a linter. Monday's came from adding the guard and then checking it against a running server, where every /api/v1 endpoint answering 401 without a credential was the first proof any of it worked. Wednesday's came from writing the field's documentation: saying what the seed did meant following it, and following it ran out. Friday's came from building a new limits feature and going to look at how the existing one was enforced. Saturday's came from writing the first tests that package had ever had.
Two of the four were found by being made to state, in prose, what a thing did. I do not have a better detector than that, and I have looked. Grep tells you a symbol is mentioned, which is the question you want answered only when the answer is no. It also fails in the obvious mechanical ways: when I removed a documentation directory in the same private service on Saturday, the first pass grepped a list of file extensions and missed two references, one of them in .gitattributes. The second pass grepped every file.
The one place I did get a mechanical guarantee is the interface case, and it is narrow. The fix is the ordinary Go idiom for pinning an optional interface, one line per controller:
1var _ forge.ControllerWithMiddleware = (*Controller)(nil)
That converts one silent failure into a build error, and it only works where an optional interface is the mechanism. It does nothing for a config field, and Forge's optional-capability pattern means there are a lot of these assertions to remember to write. An interface a type opts into by shape is pleasant to design against and gives you no signal when the opt-in silently misses.
#Pointing the same discipline the other way
The rest of Saturday went on SSE event replay, and the design problem there turned out to be the same one seen from the front. A client reconnects with a Last-Event-ID. Either the log can still resolve that position, or it cannot. If those two answers arrive looking alike, the caller picks wrong and says nothing about it:
1// Since returns the events recorded after id, in order.2//3// The bool reports whether id was still resolvable. False means the gap4// cannot be filled and the caller must fall back to a full resync; events is5// empty in that case and must NOT be read as "nothing was missed".6//7// Returning the two separately is the point of the signature. Folding them8// into an empty slice would make the case that silently serves stale data9// indistinguishable from the case that is safe.10Since(ctx context.Context, channel, id string) ([]LoggedEvent, bool, error)
The client half of that contract is two control frames, forge.resumed carrying the position resumed from and the count delivered, and forge.gap saying the gap could not be filled. A client that receives neither has learned nothing and recovers on its own timer.
I then wrote the same bug I had spent the week finding, into the log itself. Sequence numbers started out per channel. Channel eviction deletes the channel outright, so the next append recreated it with the counter back at 1 and reissued positions a still-connected client had already seen. Clients dedup by event ID. Reissued positions therefore made genuinely new events look like duplicates, and got them dropped, silently, for the remaining life of the connection. The counter moved to the log:
1// nextSeq is the sequence the next append will take, log-wide rather than2// per channel.3//4// Log-wide because a per-channel counter restarts whenever channel eviction5// drops the channel and the next append recreates it, re-issuing IDs a6// still-connected client has already seen. A counter that only ever moves7// forward cannot reissue a position whatever eviction does. The cost is that8// channels interleave and one channel's sequence has holes, which nothing in9// the contract forbids: a position is compared, never counted from.10nextSeq uint64
A dropped event that presents as a correctly deduplicated duplicate belongs to the same family as a config field that presents as configured. Both are an absence wearing the costume of a presence, and in both cases the thing wearing the costume is a legitimate mechanism doing its job.
#Open
There are now two SSE replay mechanisms in Forge and one id: field for them to share. The router's event log uses a scalar position, epoch-seq. The streaming extension's cursor is a vector of room to last delivered sequence, carried as a base64url JSON token, because one stream can serve many rooms advancing at different rates and a scalar cannot say where the client is in each. When both are active on a route the router wins, the extension falls back to sending without an ID, and the cursor never reaches the wire. Messages keep flowing and resume stops working. As of this week the refusal logs a warning once per stream, so the condition is at least visible, which is the smallest honest fix and not the right one. Whether the extension's cursor should adopt the router's primitive or whether routes should be made to pick one is undecided.
MemoryEventLog is per process, and each process gets its own epoch, so a client reconnecting to a different instance resolves to "not resumable" and resyncs. That is the honest answer and a poor one. A shared log with a single epoch across instances is the upgrade, it needs no transport changes, and it is not built.
The Redis message store is exercised through miniredis, which keeps the suite hermetic and proves nothing about cluster semantics. That is written in the test file rather than here, which is where I would want to find it.