--- title: Observability description: Prometheus metrics, structured access logging, and OpenTelemetry trace propagation. --- Bastion provides comprehensive observability through three pillars: Prometheus-compatible metrics, structured access logging, and OpenTelemetry trace propagation. Together they give full visibility into gateway traffic, upstream health, and request flows. ## Prometheus Metrics ### Configuration ```go bastion.WithMetrics(bastion.MetricsConfig{ Enabled: true, Prefix: "gateway", }) ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | `bool` | `true` | Enable or disable metrics collection | | `Prefix` | `string` | `"gateway"` | Prefix for all metric names | ### Exported Metrics The gateway exports the following metrics through the Forge metrics system: #### Request Metrics | Metric | Type | Description | |--------|------|-------------| | `gateway.requests_total` | Counter | Total number of proxied requests | | `gateway.request_duration_seconds` | Histogram | Request latency distribution | | `gateway.active_connections.` | Gauge | Active connections by protocol (http, websocket, sse) | #### Resilience Metrics | Metric | Type | Description | |--------|------|-------------| | `gateway.retry_total` | Counter | Total retry attempts | | `gateway.rate_limited_total` | Counter | Total rate-limited requests | | `gateway.circuit_breaker_state.` | Gauge | Circuit breaker state (0=closed, 1=half-open, 2=open) | #### Upstream Metrics | Metric | Type | Description | |--------|------|-------------| | `gateway.upstream_health.` | Gauge | Target health (1.0=healthy, 0.0=unhealthy) | | `gateway.discovery_routes.` | Gauge | Number of routes by source (manual, farp, discovery) | #### Cache Metrics | Metric | Type | Description | |--------|------|-------------| | `gateway.cache_hits_total` | Counter | Total cache hits | | `gateway.cache_misses_total` | Counter | Total cache misses | ### Custom Prefix Change the metric prefix to avoid collisions when running multiple gateways: ```go bastion.WithMetrics(bastion.MetricsConfig{ Enabled: true, Prefix: "api_gateway", }) // Metrics: api_gateway.requests_total, api_gateway.request_duration_seconds, etc. ``` ## Structured Access Logging ### Configuration ```go bastion.WithAccessLog(bastion.AccessLogConfig{ Enabled: true, RedactHeaders: []string{"Authorization", "Cookie", "Set-Cookie"}, IncludeBody: false, MaxBodyLogSize: 4096, }) ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | `bool` | `true` | Enable or disable access logging | | `RedactHeaders` | `[]string` | `["Authorization", "Cookie", "Set-Cookie"]` | Headers to redact in logs | | `IncludeBody` | `bool` | `false` | Include request/response body in logs | | `MaxBodyLogSize` | `int` | `4096` | Maximum body bytes to log | ### Log Fields Every proxied request produces a structured log entry with these fields: #### Request Fields | Field | Description | |-------|-------------| | `method` | HTTP method (GET, POST, etc.) | | `path` | Request path | | `status` | HTTP response status code | | `latency_ms` | Total request latency in milliseconds | | `client_ip` | Client IP address (from X-Forwarded-For, X-Real-IP, or RemoteAddr) | | `user_agent` | Client User-Agent header | | `request_id` | Value of X-Request-ID header (if present) | | `query` | Query string (truncated to 200 chars) | #### Route Fields | Field | Description | |-------|-------------| | `route_id` | Matched route ID | | `route_path` | Route pattern | | `route_protocol` | Route protocol (http, websocket, sse, grpc) | | `route_source` | Route source (manual, farp, discovery) | #### Upstream Fields | Field | Description | |-------|-------------| | `upstream_url` | Selected target URL | | `upstream_id` | Selected target ID | #### Safe Headers These headers are always included when present: - `Accept`, `Content-Type`, `Content-Length`, `Referer`, `Origin` Sensitive headers listed in `RedactHeaders` are logged as `[REDACTED]`. ### Log Levels The access logger uses different log levels based on the response status code: | Status Range | Log Level | |-------------|-----------| | 200-399 | `Info` | | 400-499 | `Warn` | | 500+ | `Error` | ### Admin Action Logging Administrative actions (route creation, config changes, cache invalidation) are logged separately: ```json { "msg": "gateway admin action", "action": "route_created", "resource": "route-abc123", "result": "success", "client_ip": "10.0.0.1", "user_agent": "curl/7.88.0" } ``` ## OpenTelemetry Trace Propagation ### Configuration ```go bastion.WithTracing(bastion.TracingConfig{ Enabled: true, PropagateFormat: "w3c", SampleRate: 1.0, }) ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | `bool` | `true` | Enable or disable trace propagation | | `PropagateFormat` | `string` | `"w3c"` | Trace context format to propagate | | `SampleRate` | `float64` | `1.0` | Sampling rate (0.0 to 1.0) | ### Trace Context Propagation When tracing is enabled, the gateway propagates trace context headers between the client and upstream: - **W3C Trace Context** (`"w3c"`): Propagates `traceparent` and `tracestate` headers. The gateway acts as a pass-through for trace context -- it does not create new spans but ensures that distributed tracing works end-to-end across the gateway boundary. ### Sample Rate The `SampleRate` controls what fraction of requests participate in tracing: | Value | Behavior | |-------|----------| | `1.0` | All requests are traced | | `0.5` | 50% of requests are traced | | `0.1` | 10% of requests are traced | | `0.0` | Tracing is effectively disabled | ## Gateway Stats The proxy engine maintains an aggregated stats structure available via the admin API: ```go type GatewayStats struct { TotalRequests int64 TotalErrors int64 ActiveConns int64 ActiveWSConns int64 ActiveSSEConns int64 AvgLatencyMs float64 P99LatencyMs float64 RequestsPerSec float64 CacheHits int64 CacheMisses int64 RateLimited int64 CircuitBreaks int64 RetryAttempts int64 TotalRoutes int HealthyUpstreams int TotalUpstreams int Uptime int64 StartedAt time.Time } ``` Access via the admin API: ``` GET /gateway/api/stats ``` ### Per-Route Stats Each route tracks its own metrics: ```go type RouteStats struct { RouteID string Path string TotalRequests int64 TotalErrors int64 AvgLatencyMs float64 P99LatencyMs float64 CacheHits int64 CacheMisses int64 RateLimited int64 } ``` Per-route stats are included in `GatewayStats.RouteStats` keyed by route ID. ## Dashboard Integration When the dashboard is enabled, all observability data feeds the real-time admin UI: ```go bastion.WithDashboard(bastion.DashboardConfig{ Enabled: true, BasePath: "/gateway", Realtime: true, }) ``` The dashboard displays: - Live request throughput and error rates - Latency percentiles (p50, p95, p99) - Active connections by protocol - Upstream health status per target - Circuit breaker states - Cache hit/miss ratios - Route-level traffic breakdown ## Audit Logging Administrative actions are recorded as audit events: ```go type AuditEvent struct { Action AuditAction Resource string Actor string Timestamp time.Time Details map[string]any } ``` Supported audit actions: | Action | Description | |--------|-------------| | `AuditRouteCreated` | A route was created | | `AuditRouteUpdated` | A route was updated | | `AuditRouteDeleted` | A route was deleted | | `AuditRouteToggled` | A route was enabled/disabled | | `AuditConfigChanged` | Gateway configuration was changed | | `AuditDiscoveryForced` | Discovery refresh was manually triggered | | `AuditCircuitReset` | A circuit breaker was manually reset | | `AuditCacheCleared` | Cache was cleared | Audit events can be consumed via the `AuditSink` interface for forwarding to external audit systems.