Documents

The document plane backs KindDocument entities: collaborative documents — page-builder pages, annotations — where concurrent editing is normal and last-write-wins rows would destroy work. These entities are not written through the command plane (Exec rejects them). They converge through CRDT merges in an append-only log and only periodically materialize into ordinary rows. The plane is implemented in adapters/postgres (DocStore), folding the log through grove's canonical crdt.ApplyChange — referenced, never reimplemented.

The full grove type surface merges losslessly: LWW registers, PN-counters, OR-sets (exact observed-remove), RGA lists, nested documents, and the character-level text CRDT (formatting attributes, stable cursor positions, Quill-style deltas). Materialization projects each onto its column shape: counter totals, set/list element arrays, text strings, resolved document maps. @grove-js/crdt mirrors every type in the browser, with cross-engine convergence pinned by golden fixtures.

Declaring a document entity01

A document entity is Kind: KindDocument with a CRDTSpec:

type CRDTSpec struct {
	Engine        string        // engine reference, e.g. "grove-crdt"
	SnapshotEvery int           // compact after this many updates
	QuietWindow   time.Duration // idle window before materialization

	// ArchiveHistory offloads sealed update history to the blob plane on
	// Compact (latest state stays in the DB). nil = inherit the global
	// Config.Documents.ArchiveHistory default; a non-nil pointer overrides
	// it per entity. See "History offload" below.
	ArchiveHistory *bool
}

registry.EntitySpec{
	Name:  "page",
	Kind:  registry.KindDocument,
	Model: (*domain.Page)(nil),
	CRDT: &registry.CRDTSpec{
		Engine:        "grove-crdt",
		SnapshotEvery: 200,
		QuietWindow:   30 * time.Second,
	},
}

The registry still binds the relational shape (Model), because materialization writes an ordinary row for the entity. See Registry.

The Store port02

f.Document() returns the document.Store port:

type Store interface {
	ApplyUpdate(ctx context.Context, docID string, update []byte) error
	Sync(ctx context.Context, docID string, stateVector []byte) ([]byte, error)
	Snapshot(ctx context.Context, docID string) (Materialized, error)
	Compact(ctx context.Context, docID string) error
}

Document ids carry their entity: <entity>/<ulid>, e.g. page/01HZX…. The entity prefix is validated against the registry on every call (it must be a registered KindDocument), and the ulid identifies the document.

ApplyUpdate

ApplyUpdate appends one encoded CRDT update to the document's log and advances the per-document monotonic seq. Update blobs are JSON-encoded []crdt.ChangeRecord (the grove-crdt engine); an empty or malformed blob is rejected.

docID := "page/01HZX8N3QK8M4G7P2W5C0V9R6T"

update := encodeChanges(/* []crdt.ChangeRecord from the client editor */)
if err := f.Document().ApplyUpdate(ctx, docID, update); err != nil {
	return err
}

Sync (client/server reconciliation)

Sync is the diff protocol. The client passes its encoded state vector — an 8-byte big-endian last-seen seq (empty means "from the beginning") — and the server replies with everything the client is missing: the compacted snapshot (only when the client is behind it) plus every later update, and the new vector seq. The reply is bounded to 500 updates per page; a client far behind loops, advancing its vector each round, until it gets an empty page.

// First sync: empty state vector means "send me everything".
reply, err := f.Document().Sync(ctx, docID, nil)
if err != nil {
	return err
}
// reply is the JSON sync payload: {seq, snapshot?, updates[]}.
// Apply it client-side, then resume from reply.seq encoded as 8 big-endian bytes:
var sv [8]byte
binary.BigEndian.PutUint64(sv[:], uint64(seq))
reply, err = f.Document().Sync(ctx, docID, sv[:])

Snapshot

Snapshot returns the merged current state (compacted snapshot folded with the log tail) and the materialized aggregate version.

type Materialized struct {
	DocID    string
	Snapshot json.RawMessage // merged field values, column-keyed
	Version  int64           // aggregate version of the LAST materialization
}

Version only advances when a quiet-window materialization lands — not per update. A document with unmaterialized edits has a Snapshot newer than its Version.

Compact

Compact folds the update log into a snapshot row at the current high-water seq and trims log rows with seq <= last_seq, in one transaction. It changes storage shape only — never merge results — and bounds reconnect cost for long-lived documents. Cadence is governed by CRDTSpec.SnapshotEvery; in production it runs as the worker's leader-elected compactor job, not from request handlers.

By default the trimmed log tail is simply dropped — the compacted snapshot already carries its merged effect. With history offload enabled (below), that tail is instead sealed into a blob segment before it leaves the log, so the full raw edit history stays reconstructable without keeping it in Postgres.

History offload03

Long-lived documents accumulate a large raw update log. Compaction keeps the current state cheap, but the trimmed updates are gone. History offload keeps them — moved off Postgres into the blob plane as immutable, content-addressed segments, so full edit history remains available at object-store cost while the hot path stays small.

On Compact with offload enabled, the tail being trimmed (seq <= last_seq) is first sealed into one immutable segment: its updates are serialized and written to the blob store, and a single index row is recorded in fabriq_crdt_segments mapping the contiguous [seqLo, seqHi] range to the segment's blob key — then the log rows are deleted, all in the compaction transaction. Snapshot and Sync are unaffected: they read the compacted snapshot plus the live tail exactly as before, byte-for-byte identical whether offload is on or off.

Reading offloaded history

Offloading adds three optional, type-asserted capabilities on the store (consumers assert for them, mirroring blob.Presigner/Ranger). Stores that do not offload need not implement them:

// Reconstruct a raw update range, transparently spanning sealed blob
// segments and the still-in-DB tail.
type HistoryReader interface {
	ReadHistory(ctx context.Context, docID string, seqLo, seqHi int64) ([]HistoryUpdate, error)
}

// List a document's sealed segments (storage-shape metadata; blob keys
// are an internal detail and intentionally not exposed).
type SegmentLister interface {
	ListSegments(ctx context.Context, docID string) ([]SegmentInfo, error)
}

// Delete a document's offloaded history — segment blobs + index rows.
// The admin delete path purges history when a document entity is removed.
type HistoryPurger interface {
	DeleteHistory(ctx context.Context, docID string) error
}

type SegmentInfo struct {
	SegSeq      int64     // segment ordinal
	SeqLo       int64     // inclusive seq range start
	SeqHi       int64     // inclusive seq range end
	UpdateCount int64     // updates sealed in this segment
	ByteSize    int64     // segment payload size
	At          time.Time // seal time
}

ReadHistory returns every update with seqLo <= seq <= seqHi in seq order, unioning the sealed segments (fetched from the blob store and served through an in-process LRU cache) with any updates still in the live log. fabriq_crdt_segments is a tenant-scoped content table with the same scope-aware RLS as the CRDT update/snapshot tables.

Enabling it

Offload is off by default and opt-in at two levels:

f, _, err := fabriq.Open(ctx, reg, fabriq.Config{
	// ...
	Documents: fabriq.DocumentsConfig{ArchiveHistory: true}, // global default
})

Per entity, CRDTSpec.ArchiveHistory *bool overrides the global default (nil inherits it):

CRDT: &registry.CRDTSpec{
	Engine:         "grove-crdt",
	SnapshotEvery:  200,
	ArchiveHistory: ptr(true), // this entity offloads even if the global default is off
},

Live sync transport04

Bidirectional sync rides the subscription hub's connection layer with no conflation and no coalescing — CRDT frames must arrive complete and in order. The conflating delta path (Subscriptions) and the document sync path share connections, never semantics. The seam is the hub's raw channel pair, Hub.SubscribeRaw / Hub.PublishRaw, distinct from the conflated Subscribe / Publish.

Fabriq.SubscribeDocument is the application entry point: it attaches to a document's live frames over a RAW channel, resolves the channel server-side from the validated doc id and the context tenant (doc:{tenant}:{docID}), and runs the same authz hooks as Subscribe.

frames, err := f.SubscribeDocument(ctx, docID)
if err != nil {
	return err
}
for frame := range frames {
	// frame is a query.Delta: Payload is the update blob,
	// Version is the log seq. A gap in Version means
	// "call Document().Sync and resume".
}

On the write side, every ApplyUpdate fans its frame out on that channel as a best-effort live notification — the log is the truth, and clients heal any dropped frame through Sync (each frame carries the log seq as Version for gap detection).

Gateway endpoints

The gateway extension terminates the document plane at the edge (paths under its BasePath, default /api/v1/live):

Auth is the host app's: attach middleware via the gateway's RouteOptions; the handlers require a tenant on the request context.

Presence (awareness)

Cursors, selections and who's-online ride Fabriq.PublishDocumentPresence / SubscribeDocumentPresence — an ephemeral channel (docpresence:{tenant}:{docID}) on a capped Redis stream tailed from "now": never persisted, no delivery guarantees, exactly the semantics awareness wants. GET /docs/subscribe?id=…&presence=1 interleaves presence events into the SSE stream.

Materialization — the bridge back into the fabric05

After CRDTSpec.QuietWindow of silence on a document, the materializer runs:

1. Merge the log through grove's CRDT engine.

2. Post-merge validation. CRDTs converge but do not guarantee business validity. A validation hook inspects the merged values; on a violation the document is flagged for resolution (with the reason recorded) and nothing materializes.

3. Write one event. The merged state is written into the entity's relational row and exactly ONE ordinary versioned domain event (<entity>.updated, version+1) is appended through the transactional outbox — row, event, and the materialization watermark all in the same transaction, so a crash can never re-materialize.

Downstream — graph, search, audit, subscriptions — therefore sees a CRDT document as a perfectly normal entity; nothing knows the row was CRDT-merged. The materializer runs leader-elected in the worker (see Deployment), scanning for documents idle past their QuietWindow with updates beyond the last materialization.