Blob GC & Reconcile

The byte store is external and referenced, not rebuilt (see ADR 0008 — the "referenced external blobs" model). That means fabriq never rebuilds it from events; instead a reconciler closes the gap between the blob_cas ledger and the object-storage layer on each GC cycle. BlobReconciler in adapters/trove owns all three checks: ref-count recompute, broken-row detection, and orphan-byte GC. In the worker it runs leader-elected under advisory lock 1004, so exactly one replica scans across every tenant.

Report type01

Reconcile returns a Report summarizing one tenant's pass:

// Report summarizes one tenant's reconcile pass.
type Report struct {
    RefsCorrected  int      // ledger rows whose ref_count disagreed with truth
    GCCount        int      // unreferenced+unpinned entries garbage-collected
    BytesFreed     int64    // bytes reclaimed by GC
    Broken         []string // referenced hashes whose bytes are missing
    OrphansDeleted int      // stored objects with no ledger row, removed
}

A non-empty Broken slice is an alert condition — these are live references pointing at missing bytes. RefsCorrected, GCCount, and OrphansDeleted are normal operational noise that converges to zero in a healthy system.

Reconcile(ctx, repair)02

func (r *BlobReconciler) Reconcile(ctx context.Context, repair bool) (Report, error)

ctx must carry a tenant ID stamped by tenant.WithTenant; the method returns tenant.ErrNoTenant (via tenant.Require) if absent.

With repair=false the call is a dry run: all three checks run and counts accumulate in the Report, but no SQL mutation and no byte deletion occurs. Use this for auditing or alerting without side effects.

With repair=true the reconciler mutates: it corrects blob_cas.ref_count, deletes GC-eligible byte objects then their ledger rows, and deletes orphan byte objects. All three operations are idempotent — a crash mid-cycle re-converges on the next run.

Check 1 — ref-count recompute

The reconciler counts live references from blob_objects (the command-authoritative catalog, written by every Blob Store call):

SELECT hash, COUNT(*) AS n FROM blob_objects GROUP BY hash

It compares each blob_cas ledger row's ref_count against the truth count. When they disagree:

• Report.RefsCorrected increments.

• With repair=true: UPDATE blob_cas SET ref_count = $truth WHERE hash = $h.

Check 1b — GC unreferenced entries past the grace window

When truth is 0, the entry is not pinned (blob_cas.pinned = false), and blob_cas.created_at is older than BlobGCGrace:

1. Report.GCCount and Report.BytesFreed accumulate.

2. With repair=true: byte object deleted from object storage, then the blob_cas row deleted. A fabriqerr.ErrNotFound on the byte delete is treated as already-gone and not surfaced as an error (idempotent).

The created_at column was added in migration 0015 (version 202606180015). Entries younger than the grace window are skipped, protecting an in-flight Store call whose ledger row commits after bytes land.

Check 2 — broken-row detection

For every ledger row with truth > 0, the reconciler calls a HEAD against object storage. If the object is absent the hash is appended to Report.Broken. Broken rows are reported, never deleted — deletion could destroy the only pointer to content that needs manual recovery.

A referenced hash that has no ledger row at all is also recorded as broken (the reverse case: catalog references a hash that was never written to the CAS).

Check 3 — orphan-byte GC

The reconciler lists the tenant bucket via the object-storage driver and compares keys against the ledger. A key with no matching ledger row and whose LastModified is older than BlobGCGrace is an orphan:

• Report.OrphansDeleted increments.

• With repair=true: byte object deleted. Not-found is treated as already-gone (idempotent).

The grace window prevents collecting bytes whose ledger row has not committed yet (a Store call that wrote bytes but not the ledger row).

If the tenant bucket does not exist yet, the list returns a ErrNotFound which the reconciler treats as zero orphans (no bucket = no bytes).

Worker (leader-elected)03

The blob GC runs inside the Forge extension worker. It is enabled when ReconcileInterval > 0 and Stores.CAS != nil. It shares the ReconcileInterval with the projection reconciler.

gcBlobAll falls back to a BlobGCGrace of 1h when the configured value is zero. It calls stores.BlobReconciler(grace) to build the reconciler, which requires Storage.EnableCas = true and a Postgres store; it returns an error otherwise (GC silently skips that cycle).

Stores.BlobReconciler accessor:

func (s *Stores) BlobReconciler(grace time.Duration) (*trovestore.BlobReconciler, error)

Returns fmt.Errorf("fabriq: blob reconciler needs storage with enableCas and postgres configured") when either s.CAS or s.Postgres is nil.

Configuration04

The blob GC shares ReconcileInterval with the projection reconciler and adds one knob: BlobGCGrace.

BlobGCGrace is a forgeext worker knob (forgeext.WithBlobGCGrace), not a fabriq.Config or config.yaml field. Configure it when embedding fabriq via the extension:

forgeext.New(reg,
    forgeext.WithWorker(true),
    forgeext.WithReconcileInterval(10 * time.Minute), // also governs blob GC tick
    forgeext.WithBlobGCGrace(2 * time.Hour),          // default 1h when zero
)

Metrics05

Five instruments from internal/metrics cover the GC cycle. All are reset-safe monotonic counters except fabriq_blob_gc_broken, which is set to the total broken count across all tenants at the end of each gcBlobAll call.

Alert on fabriq_blob_gc_broken > 0. Sustained growth in fabriq_blob_gc_ref_drift_corrected_total indicates a write-path bug in ref-count accounting. fabriq_blob_gc_bytes_freed_total and fabriq_blob_gc_collected_total growing at a healthy, bounded rate is expected in normal operation.

Relationship to the projection reconciler06

The blob reconciler follows the same "recompute from Postgres truth" discipline as the projection reconciler: blob_objects is the authoritative catalog (the command plane writes it), the ledger and byte store are derived, and the reconciler closes any gap. Both are leader-elected, run on the same interval, and repair idempotently. The blob GC extends this pattern into object storage.