---
title: Observability
description: Monitor Herald with structured logging via slog, the message delivery log, and the event hook system.
---

Herald provides observability through three mechanisms: structured logging using Go's `log/slog`, a persistent message delivery log stored in the database, and event hooks that fire on key lifecycle events.

## Structured Logging with slog

Herald uses Go's standard `log/slog` package for all internal logging. Log messages are structured with key-value pairs, making them easy to filter, search, and aggregate in log management systems.

### Configuring the Logger

When using Herald directly, pass a logger with the `WithLogger` option:

```go
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

h, err := herald.New(
    herald.WithStore(myStore),
    herald.WithLogger(logger),
    // ... drivers
)
```

If no logger is provided, Herald defaults to `slog.Default()`.

### Log Messages

Herald emits structured log messages at key points in the delivery pipeline:

**Debug level** -- Provider resolution and delivery details:

```go
h.logger.Debug("herald: resolved provider via fallback",
    "provider", p.Name,
    "channel", channel,
    "app_id", appID,
)

h.logger.Debug("herald: user opted out",
    "user_id", req.UserID,
    "template", req.Template,
    "channel", channel,
)
```

**Warn level** -- Non-fatal issues during delivery:

```go
h.logger.Warn("herald: notify channel failed",
    "channel", ch,
    "template", req.Template,
    "error", err,
)

h.logger.Warn("herald: failed to seed template",
    "slug", tmpl.Slug,
    "channel", tmpl.Channel,
    "error", err,
)

h.logger.Warn("herald: invalid provider ID in scoped config",
    "provider_id", pidStr,
    "scope", scopeType,
    "scope_id", scopeID,
)
```

These structured fields make it easy to build queries in your logging platform:

- Filter by `channel` to see all email delivery events
- Filter by `template` to track a specific notification type
- Filter by `app_id` to isolate a single application
- Filter by `error` to find all delivery failures

## Message Delivery Log

Every notification delivery attempt is persisted in the messages table. This provides a complete audit trail of all notifications sent through Herald.

### Message Entity

```go
type Message struct {
    ID          id.MessageID      `json:"id"`
    AppID       string            `json:"app_id"`
    EnvID       string            `json:"env_id,omitempty"`
    TemplateID  string            `json:"template_id,omitempty"`
    ProviderID  string            `json:"provider_id,omitempty"`
    Channel     string            `json:"channel"`
    Recipient   string            `json:"recipient"`
    Subject     string            `json:"subject,omitempty"`
    Body        string            `json:"body,omitempty"`
    Status      Status            `json:"status"`
    Error       string            `json:"error,omitempty"`
    Metadata    map[string]string `json:"metadata,omitempty"`
    Async       bool              `json:"async"`
    Attempts    int               `json:"attempts"`
    SentAt      *time.Time        `json:"sent_at,omitempty"`
    DeliveredAt *time.Time        `json:"delivered_at,omitempty"`
    CreatedAt   time.Time         `json:"created_at"`
}
```

### What Gets Logged

For every recipient of every `Send` call, Herald creates a message record capturing:

- **Who**: `Recipient`, `AppID`, `EnvID`
- **What**: `Channel`, `TemplateID`, `Subject`, `Body` (truncated)
- **How**: `ProviderID`, `Attempts`
- **Result**: `Status`, `Error`, `SentAt`, `DeliveredAt`
- **Context**: `Metadata` (application-specific key-value pairs)

The `Body` field is truncated to `TruncateBodyAt` characters (default: 4096) to keep the delivery log compact while retaining enough content for debugging.

### Status Tracking

Messages progress through a lifecycle of states:

| Status | Description |
|--------|-------------|
| `queued` | Message created, pending delivery (async mode) |
| `sending` | Delivery attempt in progress |
| `sent` | Driver accepted the message |
| `delivered` | Delivery confirmed by the provider |
| `failed` | Delivery failed |
| `bounced` | Recipient address bounced |

The status is updated via `UpdateMessageStatus` as the delivery progresses:

```go
// Before sending
msg.Status = message.StatusSending
_ = h.store.CreateMessage(ctx, msg)

// After driver returns
if sendErr != nil {
    _ = h.store.UpdateMessageStatus(ctx, msg.ID, message.StatusFailed, sendErr.Error())
} else {
    _ = h.store.UpdateMessageStatus(ctx, msg.ID, message.StatusSent, "")
}
```

### Querying the Delivery Log

Messages can be queried through the store or API with filtering and pagination:

```go
messages, err := store.ListMessages(ctx, appID, message.ListOptions{
    Channel: "email",
    Status:  message.StatusFailed,
    Limit:   50,
    Offset:  0,
})
```

#### API Endpoints

| Method | Path | Operation |
|--------|------|-----------|
| `GET` | `/messages` | List messages (query: `app_id`, `channel`, `status`, `limit`, `offset`) |
| `GET` | `/messages/:id` | Get a single message by ID |

Example: list all failed email deliveries:

```
GET /herald/messages?app_id=app_01abc...&channel=email&status=failed&limit=20
```

### Store Interface

```go
type Store interface {
    CreateMessage(ctx context.Context, m *Message) error
    GetMessage(ctx context.Context, messageID id.MessageID) (*Message, error)
    UpdateMessageStatus(ctx context.Context, messageID id.MessageID, status Status, errMsg string) error
    ListMessages(ctx context.Context, appID string, opts ListOptions) ([]*Message, error)
}
```

## Monitoring Patterns

### Failed Delivery Alerting

Query the messages table periodically for failed deliveries to trigger alerts:

```go
failed, err := store.ListMessages(ctx, appID, message.ListOptions{
    Status: message.StatusFailed,
    Limit:  100,
})

if len(failed) > threshold {
    // Trigger alert
}
```

### Delivery Rate Tracking

Use the `CreatedAt` and `SentAt` timestamps to measure delivery latency:

```go
for _, msg := range messages {
    if msg.SentAt != nil {
        latency := msg.SentAt.Sub(msg.CreatedAt)
        // Record latency metric
    }
}
```

### Per-Channel Health

List messages filtered by channel and status to monitor the health of individual channels:

```go
// Check email delivery success rate
allEmail, _ := store.ListMessages(ctx, appID, message.ListOptions{Channel: "email", Limit: 1000})
failedEmail, _ := store.ListMessages(ctx, appID, message.ListOptions{Channel: "email", Status: message.StatusFailed, Limit: 1000})

successRate := 1.0 - float64(len(failedEmail)) / float64(len(allEmail))
```

### Per-Provider Tracking

Each message records the `ProviderID` that was used for delivery. This allows you to track provider-level reliability:

```go
msg, _ := store.GetMessage(ctx, messageID)
// msg.ProviderID tells you which provider handled this delivery
// msg.Error tells you what went wrong (if failed)
```

## Health Check

Herald exposes a `Health` method that pings the underlying store to verify database connectivity:

```go
err := h.Health(ctx)
if err != nil {
    // Store is unreachable
}
```

When running as a Forge extension, this is integrated into the Forge health check system automatically.