The worker exposes Prometheus metrics and health on FABRIQ_HTTP_ADDR
(default :8081). Metrics are at /metrics (mounted by fabriq) and forge's
own /_/metrics; health is at the forge endpoints below.
Metrics01
Five instruments, defined in internal/metrics. The names are exact.
| Metric | Type | Meaning | Direction |
fabriq_outbox_backlog | gauge | Unpublished transactional-outbox rows. | Near zero is healthy. Sustained growth means the relay is down, has no leader, or Redis is unreachable. |
fabriq_projection_lag_events | gauge | Events between a projection's position and the stream head. Labels projection (graph/search) and tenant. | Near zero is healthy. Sustained growth means consumers cannot keep up or a projection is stalled. |
fabriq_tenant_hook_trips_total | counter | Tenant-guard backstop trips. | Must stay zero. Any non-zero value is a fabriq bug. |
fabriq_conflation_depth | gauge | Deltas buffered in subscription-hub conflation windows. | Low is healthy. Sustained growth means subscribers cannot drain. |
fabriq_relay_published_total | counter | Events published by the outbox relay. | Monotonic; flatlining while backlog grows confirms a stalled relay. |
fabriq_analytics_applied_total | counter | Envelopes successfully applied by the analytics sink's proj:analytics consumer. | Monotonic while analytics is configured and traffic flows; only emitted when Config.Analytics is set. |
fabriq_analytics_failures_total | counter | Envelopes the analytics consumer failed to apply (transient; left pending for redelivery). | Must stay near zero. Sustained growth means the analytics database is unreachable or rejecting writes. |
fabriq_analytics_lag_seconds | gauge | Worst-case analytics freshness: the stalest tenant's lag (now() minus that tenant's newest fact). Per-tenant under the hood, so one stalled tenant moves it — it is not masked by others still flowing. Sampled every 15s. | Low is healthy (sub-second under active traffic). A climb means at least one tenant's consumer is behind. |
fabriq_analytics_tenants_behind | gauge | Number of tenants whose analytics lag exceeds the alarm threshold (60s). | Zero is healthy. Non-zero names how many tenants are stalled; query fabriq_analytics_facts grouped by tenant_id to find which. |
fabriq_analytics_events_pruned_total | counter | Analytics history events deleted by the retention pruner (only when analytics.eventRetention is set). | Monotonic; flat means either retention is off or nothing is old enough to prune yet. |
fabriq_tenant_hook_trips_total is a correctness alarm, not a capacity one. A
non-zero value means a query reached an engine without tenant scoping — RLS
contained the blast radius, but the call site must be found and fixed. Page the
owning team. See Runbooks.
Planned, not yet built: a fabriq_analytics_backfill_rows_total counter
(backfill runs outside the worker's metrics loop today). See
Analytics sink.
How they are populated
fabriq_relay_published_totalincrements on every relay publish (an on-publish callback wired into the relay).The gauges are refreshed by a poller that runs every 15s while the worker leads. Each tick: counts unpublished
fabriq_outboxrows intofabriq_outbox_backlog; folds new backstop trips intofabriq_tenant_hook_trips_total; and reads consumer-group lag for thegraphandsearchprojections intofabriq_projection_lag_events. Lag is a group property, so the poller emits it under thetenantlabel_all.
Health endpoints02
Forge serves three health endpoints on the same address:
| Path | Use |
/_/livez | Liveness — process is up. Kubernetes liveness probe. |
/_/readyz | Readiness — ready to serve. Kubernetes readiness probe. |
/_/health | Aggregate health detail, including the worker's store ping. |
The worker's health check pings Postgres through grove; it reports unhealthy if the stores are not open.
curl localhost:8081/_/readyz
curl localhost:8081/_/health
curl localhost:8081/metricsTrace propagation03
Every command stamps the active W3C traceparent into the event envelope
by default (otel.TraceparentFromContext). The projection engine restores that
trace context when it applies the event, so a trace flows across the async hop
from command to projection — the write transaction, the relay publish, and the
downstream apply share one trace. The traceparent column on fabriq_outbox
persists it.
Scraping04
The Helm chart adds prometheus.io/scrape pod annotations by default
(/metrics on the http port). For a Prometheus Operator install, enable the
ServiceMonitor (metrics.serviceMonitor.enabled) to scrape the http port at
/metrics. See Deployment.
For lag-driven autoscaling of projection consumers, drive a KEDA scaler off
Redis stream lag or fabriq_projection_lag_events rather than CPU — the
singleton runners are leader-elected and do not scale with replica count, but
the consumers do.