Long-form, irregular, and mostly about the parts of a system that only reveal themselves in production.
A TypeScript template, two Rust frameworks and one in Go. What each got wrong in order, and the two ideas that survived all of them.
Read →Oct 2025Checkpointed workflows, leader-elected cron and a dead letter queue, imported into the process you already deploy. What the model gives you and where it stops.
Read →Sept 2025Rotation that requires a coordinated deploy happens once, at audit time. A grace window where both keys validate is what turns it into a background process.
Read →Aug 2025A hash chain makes tampering detectable, which is the point. It also makes deletion impossible, which is a legal problem. Per-subject keys resolve it, with conditions.
Read →Jul 2025Roles, attributes and relationships answer different questions and most systems need all three. Fixing the precedence rule up front is most of the design.
Read →Jun 2025A manifest published by the service, consumed by the gateway. Why the configuration belongs with the service, and what putting discovery on the critical path costs.
Read →May 2025Generating each database its own dialect instead of a portable subset. What that buys, what it costs, and the benchmark numbers with their caveats.
Read →Apr 2025Every cross-tenant leak I have looked at came from one query missing one clause. Making the boundary structural means the mistake stops being available.
Read →Mar 2025Four stores, one handler, no transaction. How I ended up with a reconciliation job nobody owned, and what a single transactional write path actually costs.
Read →Feb 2025Job queues, webhook delivery, authorisation and billing as libraries you compile in rather than services you operate. The reasoning, and the three places it fails.
Read →Jan 2025Minimalism does not remove decisions, it relocates them. Twenty-three things I decided before serving a single request, and the two the framework got wrong.
Read →Nov 2024Training code is a fraction of a production machine learning system. The rest is data validation, serving, monitoring and the debt that accumulates when those are afterthoughts.
Read →Oct 2024Routing, caching, guardrails and budgets are worth having. A third party in the path of every prompt is a data governance conversation before it is an architecture one.
Read →Sept 2024The model is a client you cannot brief. Every ambiguity in a tool description becomes a wrong call, and the fixes are the ones you would apply to any public API.
Read →Jul 2024Ordering detectors by cost and precision, with the right to terminate, produces decisions with a single named cause. That is worth more than the latency it saves.
Read →Jun 2024A single accuracy number over a small hand-picked set tells you almost nothing. Baselines, scenario coverage and regression detection are what make a change reviewable.
Read →May 2024Approximate nearest neighbour search has recall you chose, a build cost you pay on every update, and no transactional relationship to the data it indexes.
Read →Apr 2024Returning the twenty things most related to an asset is easy. Explaining why each one is in the list is what determines whether an operator uses the result.
Read →Mar 2024Deriving node identity from a normalised natural key instead of allocating it makes multi-source ingestion converge. Here is exactly where that stops working.
Read →Feb 2024Most retrieval failures I have debugged had the right passage in the result set and lost it to a naive concatenation with no token budget.
Read →Jan 2024What actually degrades in a long-running agent around turn two hundred, and why bounded working memory addressed it where more tokens did not.
Read →Dec 2023The modelling is the easy part. Well-maintained equipment fails rarely, prevented failures are not recorded, and the label you need usually does not exist.
Read →Nov 2023Three questions decide where a computation runs, and none of them are about architecture preference. What breaks if the link drops, what the bandwidth costs, and how fast the answer is needed.
Read →Oct 2023The engine matters less than what you decide to keep and at what resolution, because that decision is irreversible and gets made implicitly by default settings.
Read →Sept 2023A sensor drops out for ninety minutes. Drawing a line between the readings either side is one line of code and it makes the chart look right. That is the problem.
Read →Aug 2023Reading OPC UA as a way to move values misses the point. The information model is the valuable part, and ignoring it means rebuilding it badly on your own side.
Read →Jun 2023The defence sector had this idea first, and for a specific reason. Fleet management under individual usage histories is the original problem the term was coined for.
Read →May 2023Power systems combine millisecond electrical dynamics with equipment measured in decades. A twin that picks one timescale is useless for the other, and reconciling them is a modelling decision.
Read →Mar 2023Decades of tag naming conventions, four systems of record that disagree, and connectivity that is genuinely intermittent. The problems are not the ones the vendor slides show.
Read →Jan 2023The term covers everything from a 3D render to a coupled simulation. The distinction that matters is how the data flows, and most things sold as twins do not close the loop.
Read →Nov 2022The hard part is not running in two regions. It is deciding what happens to a write when the regions cannot see each other, and that decision belongs to the product.
Read →Oct 2022A reconciliation loop over declared state is a genuinely good idea. It is not a distributed systems solution, and treating it as one produces confident, broken deployments.
Read →Sept 2022Deploy strategies get the attention, and most failed deploys I have seen failed because the outgoing version could not shut down cleanly or the schema change was not backward compatible.
Read →Jul 2022Putting slow work on a queue moves the latency somewhere the user cannot see it, and adds ordering, poison messages and a second failure domain. Notes on the trade.
Read →Jun 2022Retries are guaranteed at some point, so every write path needs an answer for the duplicate. Where the key comes from, how long you keep it, and what to return the second time.
Read →Apr 2022A breaker converts a slow failure into a fast one, which is usually what you want. Here is the configuration that turns a partial outage into a total one.
Read →Mar 2022They have different lifetimes, different audiences and different failure modes. Treating them as one bag of strings is how credentials end up in a log line.
Read →Feb 2022Firing every watcher on every change is correct for a log level and actively harmful for a connection pool. On letting the owning component decide what a change means.
Read →Nov 2021Flutter Unity Widget crossed two thousand stars and several thousand issues. What the arithmetic actually looked like, and the three habits that kept it survivable.
Read →Sept 2021Every service I have added tracing to during an outage was a service I could not reason about during the outage. On making instrumentation a property of the framework.
Read →Jul 2021An endpoint that returns 200 whenever the process is alive tells the orchestrator nothing it did not already know. On making liveness and readiness answer different questions.
Read →May 2021Guardian taught me that every authentication method wants a different shape of configuration, storage and routing. Expressing that as flags on one server produces a surface nobody can reason about.
Read →Mar 2021North modelled service startup as a dependency graph because Rust forced me to be explicit about ordering. It turned out to be the best idea in either Rust framework.
Read →Dec 2020Roughly a fifth of six years of issues came down to one question: what happens to a game engine when the activity underneath it is destroyed and recreated.
Read →Oct 2020I wanted the ergonomics of a batteries-included framework without giving up the compiler. Ultimate got about eighteen months before I admitted the type signatures had won.
Read →Aug 2020I put it in the resolver first, which was wrong. On moving the tenant boundary far enough down that a forgotten clause cannot leak data.
Read →Jun 2020Subscriptions, projections and the schema evolution problem nobody mentions in the tutorial. Notes from extracting the NestJS event store module.
Read →Apr 2020Separating reads from writes is a modest idea that attracts immodest architecture. What I actually kept from it, and the three places it earned its cost.
Read →Feb 2020A stitched GraphQL gateway is a second schema somebody has to maintain. Federation moves that declaration into the services, and the trade is real but favourable.
Read →Nov 2019I had built the same multi-tenant scaffolding three times in a year and made different decisions each time. Ultimate Backend was an attempt to stop doing that.
Read →Oct 2019Notes from building an augmented reality toolkit where Unity handles tracking and rendering while Flutter owns the interface. Written up in more detail for Potato on Medium.
Read →Jun 2019Compositing a native view into a Flutter widget tree is not free. Notes on the synchronisation, memory and input costs, measured on mid-range Android hardware.
Read →Mar 2019I wanted a 3D product configurator in a Flutter app. Flutter could not host Unity, so I spent a weekend on a platform view and a message channel. Here is what that actually involves.
Read →