Cluster

The cluster package enables multiple Dispatch instances to coordinate work. One instance is the cluster leader at any time; the others are followers.

How it works01

1. Every Dispatch instance registers a cluster.Worker record at startup with its hostname, queues, and concurrency.

2. Workers send heartbeats on HeartbeatInterval (default: 10s) to update LastSeen.

3. The leader is determined by AcquireLeadership using optimistic locking in the store. Leadership is renewed before it expires.

4. The leader periodically:

   • Fires due cron entries

   • Scans for stale workers (workers whose heartbeat has been missing longer than StaleJobThreshold)

   • Reclaims stale workers' in-flight jobs and re-enqueues them (work stealing)

Worker states02

Leadership03

Only the leader performs:

• Cron scheduling

• Stale-job recovery

• Work rebalancing

If leadership is lost mid-operation, dispatch.ErrLeadershipLost is returned. This is normal and safe to ignore — another worker will acquire leadership shortly.

Kubernetes consensus04

For Kubernetes deployments, use cluster/k8s which implements leader election via the Kubernetes API (client-go):

import "github.com/xraph/dispatch/cluster/k8s"

leaderElector := k8s.NewLeaderElector(kubeClient, namespace, leaseName)
eng := engine.Build(d,
    engine.WithLeaderElector(leaderElector),
)

This ensures correct behavior across pod restarts and rolling updates.

Configuration05