Architecture
Octopus is designed as a high-performance, modular, and extensible API gateway built in Rust.
System Overview01
┌────────────────────────────────────────────────────────────────────┐
│ Client Layer │
│ HTTP/HTTPS │ gRPC │ WebSocket │ SSE │ GraphQL │ WebTransport │
└────────────────────────────────┬───────────────────────────────────┘
│
┌────────────────────────────────▼───────────────────────────────────┐
│ Octopus API Gateway │
├────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Protocol Layer │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ HTTP │ │ gRPC │ │ WS │ │ GraphQL │ │ │
│ │ │ Handler │ │ Handler │ │ Handler │ │ Federation │ │ │
│ │ └─────┬────┘ └─────┬────┘ └─────┬────┘ └──────┬───────┘ │ │
│ └────────┼────────────┼────────────┼──────────────┼───────────┘ │
│ │ │ │ │ │
│ ┌────────▼────────────▼────────────▼──────────────▼───────────┐ │
│ │ Middleware Pipeline │ │
│ │ Auth │ RateLimit │ CORS │ Compression │ Scripting │ Metrics │ │
│ └───────────────────────────┬───────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼───────────────────────────────────┐ │
│ │ Router & Matcher │ │
│ │ - Trie-based path matching │ │
│ │ - Dynamic route registration from FARP │ │
│ │ - Load balancing (Round-robin, Least-conn, Weighted) │ │
│ └───────────────────────────┬───────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼───────────────────────────────────┐ │
│ │ Service Registry (FARP Client) │ │
│ │ - Watch service manifests │ │
│ │ - Fetch OpenAPI/AsyncAPI schemas │ │
│ │ - Generate federated docs │ │
│ │ - Health tracking │ │
│ └───────────────────────────┬───────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼───────────────────────────────────┐ │
│ │ Plugin System │ │
│ │ - Dynamic loading (libloading) │ │
│ │ - Lifecycle hooks │ │
│ │ - Admin dashboard extensions │ │
│ │ - Custom protocol handlers │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────┬───────────────────────────────────────┘
│
┌──────────────────────────────▼───────────────────────────────────────┐
│ Discovery Backends │
│ Consul │ Kubernetes │ etcd │ Eureka │ DNS │ Static Config │
└───────────────────────────────────────────────────────────────────────┘
│
┌──────────────────────────────▼───────────────────────────────────────┐
│ Upstream Services │
│ Microservices │ REST APIs │ gRPC Services │ GraphQL Servers │
└───────────────────────────────────────────────────────────────────────┘Design Philosophy02
1. Performance First
Octopus is designed for maximum throughput with minimal latency.
Zero-copy proxying: Stream data without unnecessary buffering
Async I/O: Built on Tokio for efficient concurrency
Connection pooling: Reuse connections to upstreams
Lock-free data structures: Use Arc and DashMap for concurrent access
SIMD optimizations: Fast path matching where applicable
2. Stateless by Default
Gateway instances share no state
Easy horizontal scaling
No session affinity required
Optional state via plugins (Redis, etc.)
3. Type Safety
Leverages Rust's type system
Compile-time guarantees
No null pointer exceptions
Memory safety without garbage collection
Zero-cost abstractions
4. Extensibility
Dynamic plugin system
Middleware pipeline
Protocol handlers
Admin dashboard extensions
Scripting support (Rhai)
5. Production Ready
Comprehensive error handling
Health checks and circuit breakers
Graceful shutdown
Observability built-in
Tested for failure scenarios
Crate Structure03
Octopus is organized as a Cargo workspace with focused crates:
octopus/
├── crates/
│ ├── octopus-core/ # Core types, traits, error handling
│ ├── octopus-runtime/ # Async runtime, lifecycle management
│ ├── octopus-router/ # Trie-based routing, matching
│ ├── octopus-proxy/ # HTTP proxy, connection pooling
│ ├── octopus-farp/ # FARP protocol client
│ ├── octopus-discovery/ # Service discovery backends
│ ├── octopus-protocols/ # Protocol handlers
│ ├── octopus-middleware/ # Core middleware
│ ├── octopus-auth/ # Authentication system
│ ├── octopus-plugins/ # Plugin system
│ ├── octopus-scripting/ # Scripting engine (Rhai)
│ ├── octopus-health/ # Health tracking
│ ├── octopus-admin/ # Admin API + Dashboard
│ ├── octopus-config/ # Configuration management
│ ├── octopus-metrics/ # Observability (Prometheus, OTLP)
│ └── octopus-tls/ # TLS termination
├── plugins/ # Built-in plugins
│ ├── auth-jwt/
│ ├── rate-limiter/
│ ├── cache-redis/
│ └── kafka-producer/
└── octopus-cli/ # CLI applicationCrate Responsibilities
octopus-core
Foundation crate with common types:
Types: Request, Response, Upstream, Route
Traits: Plugin, Middleware, ProtocolHandler
Error handling: Result types, error conversions
Utilities: Helpers and common functions
octopus-runtime
Server runtime and lifecycle:
Server: Hyper-based HTTP server
Lifecycle: Startup, shutdown, signal handling
Workers: Thread pool management
Graceful shutdown: Connection draining
octopus-router
Request routing:
Trie-based matching: O(k) lookup time
Path parameters: Extract
:idstyle parametersWildcards: Match
*patternsMethod routing: HTTP method matching
Priority: Route priority resolution
octopus-proxy
HTTP proxying:
Connection pooling: Reuse upstream connections
Request forwarding: Zero-copy where possible
Header management: X-Forwarded-* headers
Timeouts: Request and response timeouts
Retries: Automatic retry logic
octopus-farp
FARP protocol client:
Manifest watching: Monitor service manifests
Schema fetching: Download OpenAPI/AsyncAPI specs
Route generation: Create routes from schemas
Federation: Merge multiple schemas
Caching: Schema caching with TTL
octopus-discovery
Service discovery backends:
Kubernetes: Native K8s integration
Consul: Consul service discovery
etcd: etcd-based discovery
Eureka: Netflix Eureka support
DNS: DNS SRV records
Static: Configuration-based
octopus-protocols
Protocol-specific handlers:
HTTP: HTTP/1.1, HTTP/2, HTTP/3
gRPC: gRPC proxying and reflection
WebSocket: Bidirectional WebSocket proxy
SSE: Server-Sent Events
GraphQL: GraphQL federation
WebTransport: QUIC-based transport
octopus-middleware
The middleware catalogue — roughly two dozen implementations. Two are wired into the live gateway chain today (the auth gateway and compression); the rest are builder-only (CORS, rate limiting, security headers, WAF, caching, etc.). See Middleware for the full catalogue and wiring status:
Auth gateway: JWT, OIDC, API key, mTLS, forward-auth (via providers)
Compression: gzip, brotli, zstd (the wired global response compressor)
Builder-only: rate limiting, CORS, caching, retries, circuit breaker, and more
octopus-plugins
Plugin system:
Plugin trait: Common interface
Dynamic loading: libloading-based
Lifecycle: Load, configure, start, shutdown
Registry: Plugin management
Sandboxing: Security restrictions
Data Flow04
Request Path
The live request path runs the global middleware chain (today: optional compression, then the optional auth gateway), matches a route, selects an upstream instance, and proxies. For the source-verified step-by-step flow see Request Lifecycle.
sequenceDiagram
participant C as Client
participant S as Server
participant M as Middleware Chain
participant R as Router
participant LB as Load Balancer
participant U as Upstream
C->>S: HTTP Request (listener / TLS)
S->>R: Pre-match route, inject auth context
S->>M: Run chain (compression, auth gateway)
M->>R: Match route + rewrite prefix
R->>LB: Select instance for upstream
LB->>U: Forward Request
U-->>LB: Response
LB-->>M: Response
M-->>S: Compress on the way out
S-->>C: HTTP ResponseThe chain is assembled in code in
crates/octopus-runtime/src/server.rs and applied in handler.rs. It holds
at most two entries — compression (when gateway.compression.enabled) and the
auth gateway (when auth providers are configured or auth.global_enforce).
There is no middleware: array, and rate-limit / CORS are not in the live
chain. See Middleware.
Service Discovery Flow
sequenceDiagram
participant F as FARP Client
participant D as Discovery Backend
participant S as Service
participant R as Router
F->>D: Watch Services
D-->>F: Service Manifest
F->>S: Fetch Schema
S-->>F: OpenAPI/AsyncAPI
F->>F: Generate Routes
F->>R: Register Routes
R-->>F: Routes ActivePerformance Characteristics05
Router
Lookup Time: O(k) where k is path length
Memory: O(n) where n is number of routes
Thread Safety: Lock-free reads, write locks for updates
Connection Pool
Reuse Rate: 95%+ for keep-alive connections
Pool Size: Configurable per upstream
Eviction: Idle timeout and max age
Middleware
Overhead: < 1ms per request
Pipeline: Executed in order
Short-circuit: Early return on auth/rate limit failure
Memory
Baseline: < 100MB
Per Connection: ~4KB
Per Route: ~128 bytes
Scaling: Linear with connections and routes
Deployment Architecture06
Single Instance
┌─────────┐
│ Client │
└────┬────┘
│
┌────▼────┐
│ Octopus │
└────┬────┘
│
┌────▼──────────┐
│ Upstreams │
└───────────────┘Multiple Instances (Load Balanced)
┌─────────┐
│ Client │
└────┬────┘
│
┌────▼───────┐
│ L4 LB │
└──┬──┬──┬───┘
│ │ │
┌──▼──▼──▼───┐
│ Octopus │
│ Cluster │
└──┬──┬──┬───┘
│ │ │
┌──▼──▼──▼────┐
│ Upstreams │
└─────────────┘Multi-Region
┌─────────┐
│ DNS/CDN │
└─┬─────┬─┘
│ │
┌────▼─┐ ┌─▼────┐
│ US │ │ EU │
│Region│ │Region│
└─┬────┘ └───┬──┘
│ │
┌───▼───┐ ┌───▼───┐
│Octopus│ │Octopus│
│Cluster│ │Cluster│
└───┬───┘ └───┬───┘
│ │
┌───▼───┐ ┌───▼───┐
│ US │ │ EU │
│Svcs │ │Svcs │
└───────┘ └───────┘Scalability07
Horizontal Scaling
Stateless: Add more instances
Discovery: All instances see same services
Config: Shared or per-instance
Limits: Tested to 100+ instances
Vertical Scaling
CPU: Near-linear scaling with cores
Memory: Configurable limits
Connections: 10k+ per instance
Throughput: 100k+ RPS per instance
Next Steps08
Request Lifecycle - How a request is handled
Routing - Understand routing
Middleware - Learn about middleware
Plugins - Extend with plugins
FARP Protocol - Auto-discovery
Deployment - Deploy to production