Long-form, irregular, and mostly about the parts of a system that only reveal themselves in production.
Moving CI out of five repositories and into one shared workflow was meant to be a consolidation. It found six checks that had been passing without checking anything.
Read →Aug 2026Four times in five days, in four repositories, I found a subsystem that existed and ran and was referenced from live code, and that nothing ever called. What they had in common was the thing that made them invisible.
Read →Jul 2026Eighteen repositories that have been private for years are going out with licences. Two reasons, one of which I would rather not admit to.
Read →Jul 2026Every cross-tenant leak I have looked at came from one query missing one clause. Making the boundary structural means the mistake stops being available.
Read →Jul 2026Four stores, one handler, no transaction. How I ended up with a reconciliation job nobody owned, and what a single transactional write path actually costs.
Read →Jul 2026Returning the twenty things most related to an asset is easy. Explaining why each one is in the list is what determines whether an operator uses the result.
Read →Jul 2026Deriving node identity from a normalised natural key instead of allocating it makes multi-source ingestion converge. Here is exactly where that stops working.
Read →May 2026Routing, caching, guardrails and budgets are worth having. A third party in the path of every prompt is a data governance conversation before it is an architecture one.
Read →May 2026A single accuracy number over a small hand-picked set tells you almost nothing. Baselines, scenario coverage and regression detection are what make a change reviewable.
Read →Apr 2026Rotation that requires a coordinated deploy happens once, at audit time. A grace window where both keys validate is what turns it into a background process.
Read →Apr 2026Generating each database its own dialect instead of a portable subset. What that buys, what it costs, and the benchmark numbers with their caveats.
Read →Apr 2026Ordering detectors by cost and precision, with the right to terminate, produces decisions with a single named cause. That is worth more than the latency it saves.
Read →Apr 2026What actually degrades in a long-running agent around turn two hundred, and why bounded working memory addressed it where more tokens did not.
Read →Mar 2026Checkpointed workflows, leader-elected cron and a dead letter queue, imported into the process you already deploy. What the model gives you and where it stops.
Read →Mar 2026A hash chain makes tampering detectable, which is the point. It also makes deletion impossible, which is a legal problem. Per-subject keys resolve it, with conditions.
Read →Mar 2026Roles, attributes and relationships answer different questions and most systems need all three. Fixing the precedence rule up front is most of the design.
Read →Mar 2026Job queues, webhook delivery, authorisation and billing as libraries you compile in rather than services you operate. The reasoning, and the three places it fails.
Read →Mar 2026Most retrieval failures I have debugged had the right passage in the result set and lost it to a naive concatenation with no token budget.
Read →Dec 2025A TypeScript template, two Rust frameworks and one in Go. What each got wrong in order, and the two ideas that survived all of them.
Read →Jun 2025A manifest published by the service, consumed by the gateway. Why the configuration belongs with the service, and what putting discovery on the critical path costs.
Read →Jan 2025Minimalism does not remove decisions, it relocates them. Twenty-three things I decided before serving a single request, and the two the framework got wrong.
Read →Nov 2024Training code is a fraction of a production machine learning system. The rest is data validation, serving, monitoring and the debt that accumulates when those are afterthoughts.
Read →Sept 2024The model is a client you cannot brief. Every ambiguity in a tool description becomes a wrong call, and the fixes are the ones you would apply to any public API.
Read →May 2024Approximate nearest neighbour search has recall you chose, a build cost you pay on every update, and no transactional relationship to the data it indexes.
Read →Dec 2023The modelling is the easy part. Well-maintained equipment fails rarely, prevented failures are not recorded, and the label you need usually does not exist.
Read →Nov 2023Three questions decide where a computation runs, and none of them are about architecture preference. What breaks if the link drops, what the bandwidth costs, and how fast the answer is needed.
Read →Oct 2023The engine matters less than what you decide to keep and at what resolution, because that decision is irreversible and gets made implicitly by default settings.
Read →Sept 2023A sensor drops out for ninety minutes. Drawing a line between the readings either side is one line of code and it makes the chart look right. That is the problem.
Read →Aug 2023Reading OPC UA as a way to move values misses the point. The information model is the valuable part, and ignoring it means rebuilding it badly on your own side.
Read →Jun 2023The defence sector had this idea first, and for a specific reason. Fleet management under individual usage histories is the original problem the term was coined for.
Read →May 2023Power systems combine millisecond electrical dynamics with equipment measured in decades. A twin that picks one timescale is useless for the other, and reconciling them is a modelling decision.
Read →Mar 2023Decades of tag naming conventions, four systems of record that disagree, and connectivity that is genuinely intermittent. The problems are not the ones the vendor slides show.
Read →Jan 2023The term covers everything from a 3D render to a coupled simulation. The distinction that matters is how the data flows, and most things sold as twins do not close the loop.
Read →Nov 2022The hard part is not running in two regions. It is deciding what happens to a write when the regions cannot see each other, and that decision belongs to the product.
Read →Oct 2022A reconciliation loop over declared state is a genuinely good idea. It is not a distributed systems solution, and treating it as one produces confident, broken deployments.
Read →Sept 2022Deploy strategies get the attention, and most failed deploys I have seen failed because the outgoing version could not shut down cleanly or the schema change was not backward compatible.
Read →Jul 2022Putting slow work on a queue moves the latency somewhere the user cannot see it, and adds ordering, poison messages and a second failure domain. Notes on the trade.
Read →Jun 2022Retries are guaranteed at some point, so every write path needs an answer for the duplicate. Where the key comes from, how long you keep it, and what to return the second time.
Read →Apr 2022A breaker converts a slow failure into a fast one, which is usually what you want. Here is the configuration that turns a partial outage into a total one.
Read →Mar 2022They have different lifetimes, different audiences and different failure modes. Treating them as one bag of strings is how credentials end up in a log line.
Read →Feb 2022Firing every watcher on every change is correct for a log level and actively harmful for a connection pool. On letting the owning component decide what a change means.
Read →Nov 2021Flutter Unity Widget crossed two thousand stars and several thousand issues. What the arithmetic actually looked like, and the three habits that kept it survivable.
Read →Sept 2021Every service I have added tracing to during an outage was a service I could not reason about during the outage. On making instrumentation a property of the framework.
Read →Jul 2021An endpoint that returns 200 whenever the process is alive tells the orchestrator nothing it did not already know. On making liveness and readiness answer different questions.
Read →May 2021Guardian taught me that every authentication method wants a different shape of configuration, storage and routing. Expressing that as flags on one server produces a surface nobody can reason about.
Read →Mar 2021North modelled service startup as a dependency graph because Rust forced me to be explicit about ordering. It turned out to be the best idea in either Rust framework.
Read →Dec 2020Roughly a fifth of six years of issues came down to one question: what happens to a game engine when the activity underneath it is destroyed and recreated.
Read →Oct 2020I wanted the ergonomics of a batteries-included framework without giving up the compiler. Ultimate got about eighteen months before I admitted the type signatures had won.
Read →