Long-form, irregular, and mostly about the parts of a system that only reveal themselves in production.
Moving CI out of five repositories and into one shared workflow was meant to be a consolidation. It found six checks that had been passing without checking anything.
Read →Aug 2026Four times in five days, in four repositories, I found a subsystem that existed and ran and was referenced from live code, and that nothing ever called. What they had in common was the thing that made them invisible.
Read →Jul 2026Eighteen repositories that have been private for years are going out with licences. Two reasons, one of which I would rather not admit to.
Read →Jul 2026Every cross-tenant leak I have looked at came from one query missing one clause. Making the boundary structural means the mistake stops being available.
Read →Jul 2026Four stores, one handler, no transaction. How I ended up with a reconciliation job nobody owned, and what a single transactional write path actually costs.
Read →Jul 2026Returning the twenty things most related to an asset is easy. Explaining why each one is in the list is what determines whether an operator uses the result.
Read →Jul 2026Deriving node identity from a normalised natural key instead of allocating it makes multi-source ingestion converge. Here is exactly where that stops working.
Read →May 2026Routing, caching, guardrails and budgets are worth having. A third party in the path of every prompt is a data governance conversation before it is an architecture one.
Read →May 2026A single accuracy number over a small hand-picked set tells you almost nothing. Baselines, scenario coverage and regression detection are what make a change reviewable.
Read →Apr 2026Rotation that requires a coordinated deploy happens once, at audit time. A grace window where both keys validate is what turns it into a background process.
Read →Apr 2026Generating each database its own dialect instead of a portable subset. What that buys, what it costs, and the benchmark numbers with their caveats.
Read →Apr 2026Ordering detectors by cost and precision, with the right to terminate, produces decisions with a single named cause. That is worth more than the latency it saves.
Read →Apr 2026What actually degrades in a long-running agent around turn two hundred, and why bounded working memory addressed it where more tokens did not.
Read →Mar 2026Checkpointed workflows, leader-elected cron and a dead letter queue, imported into the process you already deploy. What the model gives you and where it stops.
Read →Mar 2026A hash chain makes tampering detectable, which is the point. It also makes deletion impossible, which is a legal problem. Per-subject keys resolve it, with conditions.
Read →Mar 2026Roles, attributes and relationships answer different questions and most systems need all three. Fixing the precedence rule up front is most of the design.
Read →Mar 2026Job queues, webhook delivery, authorisation and billing as libraries you compile in rather than services you operate. The reasoning, and the three places it fails.
Read →Mar 2026Most retrieval failures I have debugged had the right passage in the result set and lost it to a naive concatenation with no token budget.
Read →Dec 2025A TypeScript template, two Rust frameworks and one in Go. What each got wrong in order, and the two ideas that survived all of them.
Read →Jun 2025A manifest published by the service, consumed by the gateway. Why the configuration belongs with the service, and what putting discovery on the critical path costs.
Read →Jan 2025Minimalism does not remove decisions, it relocates them. Twenty-three things I decided before serving a single request, and the two the framework got wrong.
Read →Nov 2024Training code is a fraction of a production machine learning system. The rest is data validation, serving, monitoring and the debt that accumulates when those are afterthoughts.
Read →Sept 2024The model is a client you cannot brief. Every ambiguity in a tool description becomes a wrong call, and the fixes are the ones you would apply to any public API.
Read →May 2024Approximate nearest neighbour search has recall you chose, a build cost you pay on every update, and no transactional relationship to the data it indexes.
Read →Dec 2023The modelling is the easy part. Well-maintained equipment fails rarely, prevented failures are not recorded, and the label you need usually does not exist.
Read →Nov 2023Three questions decide where a computation runs, and none of them are about architecture preference. What breaks if the link drops, what the bandwidth costs, and how fast the answer is needed.
Read →Oct 2023The engine matters less than what you decide to keep and at what resolution, because that decision is irreversible and gets made implicitly by default settings.
Read →Sept 2023A sensor drops out for ninety minutes. Drawing a line between the readings either side is one line of code and it makes the chart look right. That is the problem.
Read →Aug 2023Reading OPC UA as a way to move values misses the point. The information model is the valuable part, and ignoring it means rebuilding it badly on your own side.
Read →Jun 2023The defence sector had this idea first, and for a specific reason. Fleet management under individual usage histories is the original problem the term was coined for.
Read →May 2023Power systems combine millisecond electrical dynamics with equipment measured in decades. A twin that picks one timescale is useless for the other, and reconciling them is a modelling decision.
Read →Mar 2023Decades of tag naming conventions, four systems of record that disagree, and connectivity that is genuinely intermittent. The problems are not the ones the vendor slides show.
Read →Jan 2023The term covers everything from a 3D render to a coupled simulation. The distinction that matters is how the data flows, and most things sold as twins do not close the loop.
Read →Nov 2022The hard part is not running in two regions. It is deciding what happens to a write when the regions cannot see each other, and that decision belongs to the product.
Read →Oct 2022A reconciliation loop over declared state is a genuinely good idea. It is not a distributed systems solution, and treating it as one produces confident, broken deployments.
Read →Sept 2022Deploy strategies get the attention, and most failed deploys I have seen failed because the outgoing version could not shut down cleanly or the schema change was not backward compatible.
Read →