--- title: Architecture description: Understanding the architecture and design of Octopus API Gateway --- # Architecture Octopus is designed as a high-performance, modular, and extensible API gateway built in Rust. ## System Overview ``` ┌────────────────────────────────────────────────────────────────────┐ │ Client Layer │ │ HTTP/HTTPS │ gRPC │ WebSocket │ SSE │ GraphQL │ WebTransport │ └────────────────────────────────┬───────────────────────────────────┘ │ ┌────────────────────────────────▼───────────────────────────────────┐ │ Octopus API Gateway │ ├────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Protocol Layer │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │ │ │ │ HTTP │ │ gRPC │ │ WS │ │ GraphQL │ │ │ │ │ │ Handler │ │ Handler │ │ Handler │ │ Federation │ │ │ │ │ └─────┬────┘ └─────┬────┘ └─────┬────┘ └──────┬───────┘ │ │ │ └────────┼────────────┼────────────┼──────────────┼───────────┘ │ │ │ │ │ │ │ │ ┌────────▼────────────▼────────────▼──────────────▼───────────┐ │ │ │ Middleware Pipeline │ │ │ │ Auth │ RateLimit │ CORS │ Compression │ Scripting │ Metrics │ │ │ └───────────────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼───────────────────────────────────┐ │ │ │ Router & Matcher │ │ │ │ - Trie-based path matching │ │ │ │ - Dynamic route registration from FARP │ │ │ │ - Load balancing (Round-robin, Least-conn, Weighted) │ │ │ └───────────────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼───────────────────────────────────┐ │ │ │ Service Registry (FARP Client) │ │ │ │ - Watch service manifests │ │ │ │ - Fetch OpenAPI/AsyncAPI schemas │ │ │ │ - Generate federated docs │ │ │ │ - Health tracking │ │ │ └───────────────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼───────────────────────────────────┐ │ │ │ Plugin System │ │ │ │ - Dynamic loading (libloading) │ │ │ │ - Lifecycle hooks │ │ │ │ - Admin dashboard extensions │ │ │ │ - Custom protocol handlers │ │ │ └────────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────┬───────────────────────────────────────┘ │ ┌──────────────────────────────▼───────────────────────────────────────┐ │ Discovery Backends │ │ Consul │ Kubernetes │ etcd │ Eureka │ DNS │ Static Config │ └───────────────────────────────────────────────────────────────────────┘ │ ┌──────────────────────────────▼───────────────────────────────────────┐ │ Upstream Services │ │ Microservices │ REST APIs │ gRPC Services │ GraphQL Servers │ └───────────────────────────────────────────────────────────────────────┘ ``` ## Design Philosophy ### 1. Performance First Octopus is designed for maximum throughput with minimal latency. - **Zero-copy proxying**: Stream data without unnecessary buffering - **Async I/O**: Built on Tokio for efficient concurrency - **Connection pooling**: Reuse connections to upstreams - **Lock-free data structures**: Use Arc and DashMap for concurrent access - **SIMD optimizations**: Fast path matching where applicable ### 2. Stateless by Default - Gateway instances share no state - Easy horizontal scaling - No session affinity required - Optional state via plugins (Redis, etc.) ### 3. Type Safety - Leverages Rust's type system - Compile-time guarantees - No null pointer exceptions - Memory safety without garbage collection - Zero-cost abstractions ### 4. Extensibility - Dynamic plugin system - Middleware pipeline - Protocol handlers - Admin dashboard extensions - Scripting support (Rhai) ### 5. Production Ready - Comprehensive error handling - Health checks and circuit breakers - Graceful shutdown - Observability built-in - Tested for failure scenarios ## Crate Structure Octopus is organized as a Cargo workspace with focused crates: ``` octopus/ ├── crates/ │ ├── octopus-core/ # Core types, traits, error handling │ ├── octopus-runtime/ # Async runtime, lifecycle management │ ├── octopus-router/ # Trie-based routing, matching │ ├── octopus-proxy/ # HTTP proxy, connection pooling │ ├── octopus-farp/ # FARP protocol client │ ├── octopus-discovery/ # Service discovery backends │ ├── octopus-protocols/ # Protocol handlers │ ├── octopus-middleware/ # Core middleware │ ├── octopus-auth/ # Authentication system │ ├── octopus-plugins/ # Plugin system │ ├── octopus-scripting/ # Scripting engine (Rhai) │ ├── octopus-health/ # Health tracking │ ├── octopus-admin/ # Admin API + Dashboard │ ├── octopus-config/ # Configuration management │ ├── octopus-metrics/ # Observability (Prometheus, OTLP) │ └── octopus-tls/ # TLS termination ├── plugins/ # Built-in plugins │ ├── auth-jwt/ │ ├── rate-limiter/ │ ├── cache-redis/ │ └── kafka-producer/ └── octopus-cli/ # CLI application ``` ### Crate Responsibilities #### octopus-core Foundation crate with common types: - **Types**: Request, Response, Upstream, Route - **Traits**: Plugin, Middleware, ProtocolHandler - **Error handling**: Result types, error conversions - **Utilities**: Helpers and common functions #### octopus-runtime Server runtime and lifecycle: - **Server**: Hyper-based HTTP server - **Lifecycle**: Startup, shutdown, signal handling - **Workers**: Thread pool management - **Graceful shutdown**: Connection draining #### octopus-router Request routing: - **Trie-based matching**: O(k) lookup time - **Path parameters**: Extract `:id` style parameters - **Wildcards**: Match `*` patterns - **Method routing**: HTTP method matching - **Priority**: Route priority resolution #### octopus-proxy HTTP proxying: - **Connection pooling**: Reuse upstream connections - **Request forwarding**: Zero-copy where possible - **Header management**: X-Forwarded-* headers - **Timeouts**: Request and response timeouts - **Retries**: Automatic retry logic #### octopus-farp FARP protocol client: - **Manifest watching**: Monitor service manifests - **Schema fetching**: Download OpenAPI/AsyncAPI specs - **Route generation**: Create routes from schemas - **Federation**: Merge multiple schemas - **Caching**: Schema caching with TTL #### octopus-discovery Service discovery backends: - **Kubernetes**: Native K8s integration - **Consul**: Consul service discovery - **etcd**: etcd-based discovery - **Eureka**: Netflix Eureka support - **DNS**: DNS SRV records - **Static**: Configuration-based #### octopus-protocols Protocol-specific handlers: - **HTTP**: HTTP/1.1, HTTP/2, HTTP/3 - **gRPC**: gRPC proxying and reflection - **WebSocket**: Bidirectional WebSocket proxy - **SSE**: Server-Sent Events - **GraphQL**: GraphQL federation - **WebTransport**: QUIC-based transport #### octopus-middleware The middleware catalogue — roughly two dozen implementations. Two are wired into the live gateway chain today (the auth gateway and compression); the rest are builder-only (CORS, rate limiting, security headers, WAF, caching, etc.). See [Middleware](/docs/octopus/middleware) for the full catalogue and wiring status: - **Auth gateway**: JWT, OIDC, API key, mTLS, forward-auth (via providers) - **Compression**: gzip, brotli, zstd (the wired global response compressor) - **Builder-only**: rate limiting, CORS, caching, retries, circuit breaker, and more #### octopus-plugins Plugin system: - **Plugin trait**: Common interface - **Dynamic loading**: libloading-based - **Lifecycle**: Load, configure, start, shutdown - **Registry**: Plugin management - **Sandboxing**: Security restrictions ## Data Flow ### Request Path The live request path runs the global middleware chain (today: optional compression, then the optional auth gateway), matches a route, selects an upstream instance, and proxies. For the source-verified step-by-step flow see [Request Lifecycle](/docs/octopus/concepts/request-lifecycle). ```mermaid sequenceDiagram participant C as Client participant S as Server participant M as Middleware Chain participant R as Router participant LB as Load Balancer participant U as Upstream C->>S: HTTP Request (listener / TLS) S->>R: Pre-match route, inject auth context S->>M: Run chain (compression, auth gateway) M->>R: Match route + rewrite prefix R->>LB: Select instance for upstream LB->>U: Forward Request U-->>LB: Response LB-->>M: Response M-->>S: Compress on the way out S-->>C: HTTP Response ``` The chain is assembled in code in `crates/octopus-runtime/src/server.rs` and applied in `handler.rs`. It holds at most two entries — compression (when `gateway.compression.enabled`) and the auth gateway (when auth providers are configured or `auth.global_enforce`). There is no `middleware:` array, and rate-limit / CORS are not in the live chain. See [Middleware](/docs/octopus/middleware). ### Service Discovery Flow ```mermaid sequenceDiagram participant F as FARP Client participant D as Discovery Backend participant S as Service participant R as Router F->>D: Watch Services D-->>F: Service Manifest F->>S: Fetch Schema S-->>F: OpenAPI/AsyncAPI F->>F: Generate Routes F->>R: Register Routes R-->>F: Routes Active ``` ## Performance Characteristics ### Router - **Lookup Time**: O(k) where k is path length - **Memory**: O(n) where n is number of routes - **Thread Safety**: Lock-free reads, write locks for updates ### Connection Pool - **Reuse Rate**: 95%+ for keep-alive connections - **Pool Size**: Configurable per upstream - **Eviction**: Idle timeout and max age ### Middleware - **Overhead**: < 1ms per request - **Pipeline**: Executed in order - **Short-circuit**: Early return on auth/rate limit failure ### Memory - **Baseline**: < 100MB - **Per Connection**: ~4KB - **Per Route**: ~128 bytes - **Scaling**: Linear with connections and routes ## Deployment Architecture ### Single Instance ``` ┌─────────┐ │ Client │ └────┬────┘ │ ┌────▼────┐ │ Octopus │ └────┬────┘ │ ┌────▼──────────┐ │ Upstreams │ └───────────────┘ ``` ### Multiple Instances (Load Balanced) ``` ┌─────────┐ │ Client │ └────┬────┘ │ ┌────▼───────┐ │ L4 LB │ └──┬──┬──┬───┘ │ │ │ ┌──▼──▼──▼───┐ │ Octopus │ │ Cluster │ └──┬──┬──┬───┘ │ │ │ ┌──▼──▼──▼────┐ │ Upstreams │ └─────────────┘ ``` ### Multi-Region ``` ┌─────────┐ │ DNS/CDN │ └─┬─────┬─┘ │ │ ┌────▼─┐ ┌─▼────┐ │ US │ │ EU │ │Region│ │Region│ └─┬────┘ └───┬──┘ │ │ ┌───▼───┐ ┌───▼───┐ │Octopus│ │Octopus│ │Cluster│ │Cluster│ └───┬───┘ └───┬───┘ │ │ ┌───▼───┐ ┌───▼───┐ │ US │ │ EU │ │Svcs │ │Svcs │ └───────┘ └───────┘ ``` ## Scalability ### Horizontal Scaling - **Stateless**: Add more instances - **Discovery**: All instances see same services - **Config**: Shared or per-instance - **Limits**: Tested to 100+ instances ### Vertical Scaling - **CPU**: Near-linear scaling with cores - **Memory**: Configurable limits - **Connections**: 10k+ per instance - **Throughput**: 100k+ RPS per instance ## Next Steps - [Request Lifecycle](/docs/octopus/concepts/request-lifecycle) - How a request is handled - [Routing](/docs/octopus/concepts/routing) - Understand routing - [Middleware](/docs/octopus/concepts/middleware) - Learn about middleware - [Plugins](/docs/octopus/concepts/plugins) - Extend with plugins - [FARP Protocol](/docs/octopus/concepts/farp) - Auto-discovery - [Deployment](/docs/octopus/deployment) - Deploy to production