Durable execution without running another service

#The problem with a long function

A workflow with six steps that each call an external service will fail somewhere in the middle. The process is restarted during a deploy, or the third call times out, and the work is half done with no record of where.

Retrying from the start repeats the first two steps, which is fine if they are idempotent and charges the customer twice if they are not [1].

#Checkpointing

Durable execution records the result of each step as it completes. A worker that dies resumes at the last completed step rather than the beginning, so completed work is not repeated and in-progress work is retried.

The constraint that makes this work is that a step must be deterministic given its inputs and its recorded outputs. A workflow that reads the current time or generates a random value inside a replayed region produces a different answer on resume. That constraint is easy to violate accidentally and the symptom is a workflow that behaves differently after a restart, which is unpleasant to debug.

#Why a library

Every self-hosted workflow engine is an operational unit with a deployment, an upgrade path and a version skew story against its workers. For a team running a handful of workflows, that exceeds the cost of the workflows.

1var SendEmail = job.NewDefinition("send_email",
2    func(ctx context.Context, in EmailInput) error {
3        return mailer.Send(ctx, in.To, in.Subject)
4    },
5)
6
7d, _ := dispatch.New(dispatch.WithStore(postgres.New(pool)))
8eng := engine.Build(d)
9engine.Register(eng, SendEmail)
Copy

The store is the database you already run, so the state lives where your other state lives and is backed up by the same mechanism [4].

#Leader election without a consensus service

Distributed cron needs exactly one worker to fire a schedule. The obvious approach is a consensus protocol [2], which means running another service.

For this workload a lease in the database is sufficient. A worker takes a row with an expiry, renews it while it holds leadership, and others take over when it lapses. That is weaker than consensus, and the failure mode is a brief overlap where two workers both believe they lead.

Making the scheduled work idempotent turns that overlap into a duplicate that is absorbed rather than an incident. Accepting a weaker guarantee and handling its consequence is cheaper than the stronger guarantee for anything short of financial ledgering.

#The dead letter queue is not optional

One job fails every time. Without a limit it retries forever and blocks its partition while doing so. A dead letter queue promotes it after exhausted retries so the rest of the work continues [3].

What matters is that entries can be inspected and replayed. A queue you can only drain is a queue nobody looks at, and the fault it holds is usually a deploy that fixed the underlying bug without reprocessing the work that failed before it.

#Where the model stops

Workers scale with the process they live in. Running the same binary in a worker-only mode covers most of it and is a workaround for a real limitation.

It is also Go only. If half your stack is in another language, the honest answer is a service, and a library like this is a reasonable way to build one.