This is the canonical configuration document for Spooky. It should answer these questions for every major configuration area:

• what the section is for
• what fields exist
• what values are allowed
• what the defaults are
• what runtime behavior the settings change
• what operators should be careful about

Use Configuration Defaults for the exhaustive default inventory and Configuration Examples for complete deployment patterns. Use this page when you need exact schema and semantics.

For distributed quota policy examples, Redis backend posture, migration from legacy scoped rate limiting, and operator interpretation, see Distributed Quota.

Scope Of This Reference

This page covers:

• schema shape
• precedence and normalization rules
• validation behavior
• runtime meaning of major knobs
• the boundary between raw YAML input and runtime-normalized policy objects

Default coverage now lives on Configuration Defaults so the full inventory can stay centralized and easier to audit against the code.

This page does not change the current product behavior:

• configuration hot reload is supported through the staged POST /admin/runtime/validate, POST /admin/runtime/preview, and POST /admin/runtime/activate flow. The legacy POST /admin/runtime/reload shortcut still exists, but it bypasses preview. Runtime-managed changes are re-read, validated, and applied through an atomic runtime swap (routes, upstreams, backends, timeouts, limits, resilience policies, and log.level). Only log format/file settings, tracing config, control-plane thread counts, and listener removal / bind-address changes still require a restart.
• certificate reload (POST /admin/runtime/reload-certs) covers new handshakes only
• backend transport is scheme-driven: https:// backends use HTTP/2, http:// backends use HTTP/1.1

Raw Config vs Runtime Interpretation

Spooky now has a clearer split between:

• raw configuration schema loaded from YAML
• normalized runtime configuration consumed by the rest of the system

The raw schema is defined by the config module and is what this page documents field-by-field.

The runtime model is defined by crates/config/src/runtime.rs and the domain interpreters under crates/config/src/runtime/policies/. Downstream crates should depend on that runtime model, not on the raw YAML shape.

Canonical runtime boundary

The important runtime outputs are:

• RuntimeConfig
• RuntimePolicySet
• RuntimeListenerPolicySet
• RuntimeUpstream
• RuntimeBackendEndpoint
• RuntimeLoadBalancingPolicy
• RuntimeAdmissionPolicy
• RuntimeAuthPolicy
• RuntimeTransportPolicy
• RuntimeTimeoutPolicy

These types are the validated, normalized forms that edge, transport, and lb actually execute against.

What normalization means in practice

Normalization is where Spooky resolves and validates things such as:

• listener selection precedence between listen and listeners
• per-upstream override precedence over global defaults
• trimming and rejection of empty strings where fields must be meaningful
• route host and method canonicalization
• backend endpoint parsing and transport-kind derivation
• timeout conversion from raw millisecond fields into runtime Durations
• cross-field validation for limits, inflight caps, and watchdog/retry policy
• route, auth, admission, backend, and load-balancing policy shaping

If the raw YAML is accepted, the rest of the system should not need to reinterpret those rules again.

Reading This Reference

• Start with Configuration Examples if you need a working template.
• Use Configuration Defaults when you need the effective baseline for omitted fields.
• Read TLS Setup before configuring production certificates or private trust roots.
• Read Production Readiness if you are deciding whether the current operational model fits your rollout requirements.

Configuration Reading Map

Use this quick map before diving into field tables:

Configuration File Format

Spooky uses YAML configuration loaded with:

spooky --config /path/to/config.yaml

If --config is omitted, Spooky attempts /etc/spooky/config.yaml.

Canonical Top-Level Shape

version: 1

listen:
  protocol: http3
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "/etc/spooky/certs/fullchain.pem"
    key: "/etc/spooky/certs/privkey.pem"

upstream_tls:
  verify_certificates: true
  strict_sni: true

upstream:
  default:
    route:
      path_prefix: "/"
    backends:
      - id: "backend1"
        address: "backend.internal.example:8443"
        weight: 100

log:
  level: info
  format: plain

Top-Level Keys At A Glance

Common Configuration Patterns

Use these snippets as starting points before moving to the field-by-field sections below.

Pattern Comparison

Minimal Public Edge

version: 1

listen:
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "/etc/spooky/certs/fullchain.pem"
    key: "/etc/spooky/certs/privkey.pem"

upstream:
  app:
    route:
      path_prefix: "/"
    backends:
      - id: "app-1"
        address: "https://app.internal.example:8443"

Local Development With Cleartext Backend

version: 1

listen:
  address: "127.0.0.1"
  port: 9889
  tls:
    cert: "certs/localhost.crt"
    key: "certs/localhost.key"

upstream:
  local_app:
    route:
      path_prefix: "/"
    backends:
      - id: "app-local"
        address: "http://127.0.0.1:8080"

Private CA Upstream

upstream_tls:
  verify_certificates: true
  strict_sni: true
  ca_file: "/etc/spooky/pki/internal-ca.pem"

upstream:
  internal_api:
    route:
      host: "api.example.internal"
      path_prefix: "/"
    backends:
      - id: "internal-api-1"
        address: "https://api.internal.example:9443"

Common Top-Level Mistakes

• configuring listeners[] and then expecting the top-level listen block to stay active at runtime
• using backend shorthand when cleartext http:// was intended
• exposing the Control API on a non-loopback address without strong access controls
• turning off upstream certificate verification without treating it as an explicit break-glass choice
• increasing inflight, body, or queue limits without validating backend and host capacity

Runtime Normalization And Precedence

Spooky normalizes configuration into a single runtime model before it serves traffic.

Precedence and interpretation rules:

1. If listeners[] is non-empty, it is the only effective listener set.
2. The top-level listen block is used only when listeners[] is absent or empty.
3. Per-upstream TLS settings override global upstream_tls.
4. Per-upstream load-balancing settings override the top-level load_balancing fallback.
5. Certificate reload updates listener TLS material for future handshakes; it does not rewrite the already-running route or upstream model.

Runtime Interpretation Domains

The runtime interpreter is now decomposed by policy domain. That split is important because it defines where runtime behavior is shaped and validated.

Listener and listener-TLS interpretation

The listener interpreter resolves:

• whether listen or listeners[] is authoritative
• listener source identity
• listener bind conflicts
• default TLS identity and SNI identities
• listener-scoped TLS reload inventory

This produces runtime listener types such as:

• RuntimeListener
• RuntimeListenerTls
• ListenerRuntimeConfig

Timeout interpretation

Timeout interpretation converts raw timeout fields into the canonical runtime timeout policy:

• backend request timeout
• backend connect timeout
• backend body idle and total timeouts
• shutdown drain timeout
• client body idle timeout
• backend DNS refresh interval
• QUIC idle timeout

Cross-field ordering is validated here rather than by data-plane callers.

Transport interpretation

Transport interpretation shapes:

• worker and control-plane thread counts
• shard layout
• queue capacities
• UDP buffer sizing
• inflight limits
• backend connection reuse policy
• DNS refresh enablement
• body-size and prebuffer limits

This produces:

• RuntimeTransportPolicy
• RuntimeConnectionLimits
• RuntimeBackendConnectionPolicy

Auth interpretation

Auth interpretation shapes:

• API key auth policy
• JWT auth policy
• external auth policy
• external auth failure mode
• external auth request-header shaping

This produces runtime auth types such as:

• RuntimeAuthPolicy
• RuntimeApiKeyAuth
• RuntimeJwtAuth
• RuntimeExternalAuth

Admission and rate-limit interpretation

Admission interpretation shapes:

• brownout policy
• overload and route queue policy
• scoped rate-limit rules
• watchdog-related admission policy

This produces:

• RuntimeAdmissionPolicy
• RuntimeRateLimitPolicy
• RuntimeScopedRateLimitPolicy
• RuntimeBrownoutPolicy

Backend interpretation

Backend interpretation shapes:

• canonical backend endpoint
• authority host and port
• hostname vs IP-literal classification
• runtime backend transport kind
• backend TLS policy
• backend DNS policy
• backend health-check policy

This produces:

• RuntimeBackendEndpoint
• RuntimeBackendTlsPolicy
• RuntimeBackendDnsPolicy
• RuntimeBackendHealthCheck

Load-balancing interpretation

Load-balancing interpretation shapes:

• canonical strategy
• request-key extraction spec
• alternate-backend behavior

This produces:

• RuntimeLoadBalancingPolicy
• RuntimeLoadBalancingStrategy
• RuntimeRequestKeySpec

Resilience and watchdog interpretation

Resilience interpretation shapes:

• retry budget policy
• hedge policy
• circuit breaker policy
• watchdog runtime policy

This produces:

• RuntimeRetryBudgetPolicy
• RuntimeHedgingPolicy
• RuntimeCircuitBreakerPolicy
• RuntimeWatchdogPolicy

How To Read Field Semantics

For each setting on this page, keep the following distinction in mind:

• raw schema semantics tell you what can be written in YAML
• runtime semantics tell you what the interpreter will actually execute after normalization

Examples:

• a backend address string is raw input; RuntimeBackendEndpoint is the executed form
• a timeout field in milliseconds is raw input; RuntimeTimeoutPolicy is the executed form
• a load_balancing.key string is raw input; RuntimeRequestKeySpec is the executed form
• auth and admission nested objects are raw input; RuntimeAuthPolicy and RuntimeAdmissionPolicy are the executed forms

Production-Safe Defaults

The configuration model is intentionally safe-by-default in several important areas:

• native ingress defaults to HTTP/3
• HTTPS upstreams verify certificates by default
• upstream SNI is enabled by default
• bootstrap listener TLS is always tied to configured listener identity
• request and response paths are bounded by explicit timeout and size controls

Treat the following settings as high-risk when changed:

• upstream_tls.verify_certificates: false
• broad increases to inflight or body-size limits without capacity validation
• enabling public exposure of the control API
• route or listener changes that rely on restart without a drain-and-rollback plan

Complete Example Configurations

For complete examples, use Configuration Examples.

Top-Level Configuration

version

Configuration schema version.

• Current version: 1
• Supported versions: 1
• Backward-compatibility policy: unsupported versions are rejected at load time, and version-specific migration hooks are used when introducing future schema versions.

listen

Server listening configuration. Defines the protocol, address, and port for incoming client connections. Used as the single listener when listeners is absent or empty.

Runtime interpretation:

• lowered into RuntimeListener plus RuntimeListenerTls
• then wrapped into ListenerRuntimeConfig with normalized timeout and transport policy
• ignored for runtime listener selection when listeners[] is non-empty

listeners

Optional multi-listener array. When set, overrides the top-level listen block. Each entry is an independent listener with its own address, port, and TLS identity. Spooky spawns a separate QUIC worker group and bootstrap TLS listener per entry.

Runtime interpretation:

• becomes the authoritative listener set when non-empty
• each entry is normalized independently
• duplicate bind combinations are rejected before startup or reload commit

Runtime Normalization And Precedence

Spooky normalizes configuration into one canonical runtime model before any listener starts.

Precedence rules:

1. listeners[] is the only effective listener set when it is non-empty.
2. The top-level listen block is only used when listeners[] is empty.
3. Listener TLS fallback order is:
4. exact SNI match in listen.tls.certificates
5. legacy listen.tls.cert + listen.tls.key when configured
6. otherwise the first listen.tls.certificates[] entry becomes the default identity
7. Upstream TLS precedence is:
8. upstream.<name>.tls
9. global upstream_tls
10. Listener certificate reload updates listener TLS material for new handshakes through observability.control_api.reload_certs_path without restarting the process. Existing QUIC connections and existing bootstrap TLS sessions keep the certificate and client-auth state that they already negotiated.

Startup rejects ambiguous or contradictory combinations, including duplicate effective listener binds, duplicate normalized route matchers, partial legacy listener cert/key pairs, invalid or duplicate SNI server_name entries, host_policy.host outside mode: rewrite, and CONNECT routing/policy conflicts.

upstream

Named upstream pool definitions. Each key represents a unique upstream pool with its own routing rules, load balancing strategy, and backend servers.

Runtime interpretation:

• lowered into RuntimeUpstream
• route matching becomes RuntimeRouteMatchPolicy
• backend entries become RuntimeBackend plus RuntimeBackendEndpoint
• effective upstream TLS, auth, admission, and load-balancing policy are resolved here

load_balancing

Optional global fallback for upstream load balancing. If an upstream omits upstream.<name>.load_balancing, the top-level load_balancing value is applied to that upstream during config load.

Runtime interpretation:

• global fallback only
• each effective upstream receives a canonical RuntimeLoadBalancingPolicy
• request key strings are parsed into RuntimeRequestKeySpec

log

Logging configuration. Controls log level and output formatting.

Runtime interpretation:

• log.level participates in live reload
• log sink shape such as file output and format is treated as startup-owned and may require restart

Default Values

Spooky has a large number of defaults spread across helper functions and Default implementations. The central inventory now lives on Configuration Defaults.

Use that page when you need:

• the full list of fields that may be omitted
• the exact value applied for omitted fields
• the difference between null, empty collections, empty strings, and structured section defaults

This reference page keeps the schema and semantics, while Configuration Defaults owns the exhaustive default matrix.

Validation Model

Validation happens during runtime interpretation, not lazily in downstream crates.

Important validation categories include:

• invalid listener bind combinations
• invalid or contradictory TLS identity configuration
• duplicate normalized route matchers
• invalid backend endpoint addresses
• unsupported load-balancing strategies or key specs
• zero or out-of-range timeout and limit values
• illegal cross-field timeout ordering
• unsupported watchdog or auth policy combinations

The expected downstream contract is:

• if RuntimeConfig::from_config(...) succeeds, the runtime receives canonical and validated policy objects
• data-plane and control-plane crates should consume those objects rather than repeat raw-schema validation

Listen Configuration

Configures the listening interface for incoming client connections. HTTP/3 requires TLS configuration.

Use this section when you need to decide:

• where Spooky binds
• which TLS identity it serves
• whether one listener or multiple listeners are needed

Properties

Protocol Values

• http3: HTTP/3 over QUIC (recommended)

Spooky also exposes a TLS bootstrap ingress for HTTP/1.1 and HTTP/2 clients. This compatibility path is primarily used for browser interoperability and advertising Alt-Svc so clients can upgrade to HTTP/3. Backend selection on the bootstrap path uses the same route-resolution, load-balancing strategy, and health-aware eligibility rules as the native QUIC ingress.

TLS Configuration

Certificate selection order:

1. Exact SNI match in listen.tls.certificates.
2. Fallback to listen.tls.cert/listen.tls.key when configured.
3. If legacy pair is not configured, fallback to the first entry in listen.tls.certificates.

Operational notes:

• If SNI is missing or unmatched, Spooky serves the default identity rather than rejecting the handshake.
• listen.tls.certificates[].server_name must be covered by the mapped certificate SANs or startup fails.
• Spooky exports downstream certificate expiry gauges:
• spooky_downstream_tls_certificate_not_after_seconds
• spooky_downstream_tls_certificate_days_remaining
• Certificate reload affects new QUIC and bootstrap TLS handshakes only. Existing connections continue with the TLS session they already negotiated.
• Downstream TLS metrics also include:
• spooky_downstream_tls_handshake_failure_total{listener,reason}
• spooky_downstream_tls_certificate_selection_total{listener,selection}
• spooky_downstream_tls_alpn_total{listener,protocol}
• Important reason labels are:
• missing_client_cert
• invalid_client_cert
• expired_client_cert
• unknown_issuer
• alpn
• handshake

Examples

# Standard HTTP/3 configuration
listen:
  protocol: http3
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "/etc/spooky/certs/server.crt"
    key: "/etc/spooky/certs/server.key"

# Localhost-only development
listen:
  protocol: http3
  address: "127.0.0.1"
  port: 9889
  tls:
    cert: "certs/localhost.crt"
    key: "certs/localhost.key"

# Multi-domain SNI certificates with legacy fallback
listen:
  protocol: http3
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "/etc/spooky/certs/default.crt"
    key: "/etc/spooky/certs/default.key"
    certificates:
      - server_name: "api.example.com"
        cert: "/etc/spooky/certs/api.crt"
        key: "/etc/spooky/certs/api.key"
      - server_name: "www.example.com"
        cert: "/etc/spooky/certs/www.crt"
        key: "/etc/spooky/certs/www.key"

Multi-Listener Configuration

Use listeners instead of listen when you need multiple independent listeners — for example, a public-facing port and a private/internal port with different TLS identities.

listeners and listen share the same per-entry schema. When listeners is set, the top-level listen block is ignored for runtime listener selection and listener validation.

# Single listener — use the top-level listen block (default)
listen:
  protocol: http3
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "/etc/spooky/certs/fullchain.pem"
    key: "/etc/spooky/certs/privkey.pem"

# Multi-listener — independent public and internal listeners
listeners:
  - protocol: http3
    address: "0.0.0.0"
    port: 9889
    tls:
      cert: "/etc/spooky/certs/public-fullchain.pem"
      key: "/etc/spooky/certs/public-privkey.pem"
  - protocol: http3
    address: "10.0.0.1"
    port: 9890
    tls:
      cert: "/etc/spooky/certs/internal-fullchain.pem"
      key: "/etc/spooky/certs/internal-privkey.pem"

Each listener entry shares the same upstream routing table — route matching, load balancing, and health checks are global across all listeners.

Common Mistakes

• assuming listeners[] adds to the top-level listen block instead of replacing it at runtime
• providing only cert or only key in the legacy pair
• expecting certificate reload to change active connections rather than only future handshakes
• forgetting that listener changes can still cross restart boundaries depending on bind topology

Upstream Configuration

Upstreams define groups of backends with routing rules and load-balancing strategies. Each upstream is identified by a unique name and contains route criteria, load-balancing configuration, and backend definitions.

Use this section when you need to decide:

• how requests match a route
• which upstream handles which traffic
• how backend addresses and health checks are defined
• how host and forwarded-header policy are applied

Structure

upstream:
  pool_name:
    load_balancing: <LoadBalancing>
    route: <RouteMatch>
    backends: [<Backend>]

Properties

Route Matching

Route matching determines which upstream handles a request. Routes are evaluated by longest-prefix matching across all configured upstreams, selecting the route with the most specific (longest) path prefix.

RouteMatch Properties

Route matching rules:

1. If host is specified:
2. Exact form: request Host must match exactly (case-insensitive after normalization)
3. Wildcard form: *.example.com matches subdomains like api.example.com, but not the bare apex example.com
4. If path_prefix is specified, the request path must start with the prefix
5. If both are specified, both conditions must match
6. Routes are evaluated by longest-prefix matching - the route with the most specific (longest) path prefix is selected
7. For equal-length prefixes, ties are deterministic:
8. host-specific routes win over host-agnostic routes
9. exact-host matches win over wildcard-host matches
10. among wildcard matches, longer suffixes win (*.a.example.com beats *.example.com)
11. method-specific routes win over method-agnostic routes
12. then lexicographically smaller upstream name wins

Route Examples

# Host-based routing
upstream:
  api_pool:
    route:
      host: "api.example.com"
    backends: [...]

  web_pool:
    route:
      host: "www.example.com"
    backends: [...]

# Wildcard host routing
upstream:
  tenant_pool:
    route:
      host: "*.example.com"
      path_prefix: "/api"
    backends: [...]

# Path-based routing
upstream:
  api_pool:
    route:
      path_prefix: "/api"
    backends: [...]

  admin_pool:
    route:
      path_prefix: "/admin"
    backends: [...]

  default_pool:
    route:
      path_prefix: "/"
    backends: [...]

# Combined host and path routing
upstream:
  api_v2_pool:
    route:
      host: "api.example.com"
      path_prefix: "/v2"
    backends: [...]

  api_v1_pool:
    route:
      host: "api.example.com"
      path_prefix: "/v1"
    backends: [...]

Backend Configuration

Each backend represents an upstream server that can handle requests.

Backend Properties

Backend Address Forms

Address format notes: - host:port or host — shorthand, treated as https://host:port (port defaults to 443) - https://host[:port] — TLS upstream; port defaults to 443 if omitted - http://host[:port] — cleartext HTTP/1.1 upstream; port defaults to 80 if omitted. Mixed http:// and https:// backends are supported within the same upstream.

Health Check Configuration

Health checks monitor backend availability and automatically remove unhealthy backends from the pool.

Health check behavior:

1. Health checks are performed at the specified interval
2. A backend is marked unhealthy after failure_threshold consecutive failures
3. An unhealthy backend enters cooldown for cooldown_ms milliseconds
4. After cooldown, health checks resume
5. A backend is marked healthy after success_threshold consecutive successes

Backend Examples

# Minimal backend — no health check (backend stays permanently healthy)
backends:
  - id: "backend1"
    address: "https://example.com"

# Minimal backend with health check
backends:
  - id: "backend1"
    address: "10.0.1.10:8080"
    health_check:
      path: "/health"

# Weighted backend with custom health checks
backends:
  - id: "backend1"
    address: "10.0.1.10:8080"
    weight: 100
    health_check:
      path: "/api/health"
      interval: 10000
      timeout_ms: 2000
      failure_threshold: 5
      success_threshold: 3
      cooldown_ms: 10000

  - id: "backend2"
    address: "10.0.1.11:8080"
    weight: 200
    health_check:
      path: "/api/health"
      interval: 10000

# Multiple backends with different health endpoints
backends:
  - id: "primary"
    address: "10.0.1.10:8080"
    weight: 150
    health_check:
      path: "/status"
      interval: 5000

  - id: "secondary"
    address: "10.0.1.11:8080"
    weight: 100
    health_check:
      path: "/healthz"
      interval: 5000

Host Policy

Controls how the Host / :authority header is set on requests forwarded to the upstream.

Modes

Examples

upstream:
  # Pass client host through as-is (default)
  api_pool:
    host_policy:
      mode: pass-through
    backends: [...]

  # Rewrite to a static host
  legacy_pool:
    host_policy:
      mode: rewrite
      host: "legacy-origin.internal.example"
    backends: [...]

  # Use the backend's own hostname
  direct_pool:
    host_policy:
      mode: upstream
    backends: [...]

Forwarded Headers Policy

Controls how X-Forwarded-For and related forwarding headers are set on upstream requests.

Modes

Use append in multi-hop deployments where the full client IP chain must be preserved. Use overwrite (default) when spooky is the first edge and inbound forwarded headers should not be trusted.

Examples

upstream:
  # First edge — overwrite inbound XFF with real client IP (default)
  public_pool:
    forwarded_headers:
      mode: overwrite
    backends: [...]

  # Behind another trusted proxy — append to the existing chain
  internal_pool:
    forwarded_headers:
      mode: append
    backends: [...]

  # Pass the inbound chain through unchanged
  passthrough_pool:
    forwarded_headers:
      mode: preserve
    backends: [...]

Per-Upstream TLS Policy

Each upstream can optionally override the global upstream_tls settings with its own TLS profile. When tls is omitted, the global upstream_tls block applies.

This is useful when backends have heterogeneous trust requirements — for example, one upstream uses a private internal CA while another uses a public CA.

Verification semantics:

• Hostname backends verify the upstream certificate against the configured backend hostname.
• IP-literal backends verify against the configured IP identity.
• strict_sni: false disables only the SNI extension; verification still remains enabled unless verify_certificates: false.
• verify_certificates: false disables upstream certificate validation entirely.

Global vs Per-Upstream TLS Overrides

Operational Implications

• Route specificity matters more than declaration order. The longest matching path prefix wins.
• Backend address scheme changes runtime transport behavior. https:// selects HTTP/2 transport; http:// selects HTTP/1.1 transport.
• Health checks are optional. If you omit them, a backend stays eligible unless passive health signals or other runtime behavior remove it.
• host_policy and forwarded_headers directly affect what upstream applications see.

Common Mistakes

• forgetting the http:// prefix for local or cleartext backends and accidentally opting into HTTPS defaults
• creating overlapping routes without understanding the longest-prefix and tie-break rules
• treating backend id as cosmetic only even though it appears in logs, metrics, and runtime views
• using forwarded_headers.preserve on untrusted edge traffic

Examples

upstream_tls:
  verify_certificates: true   # global default
  strict_sni: true

upstream:
  # Uses global upstream_tls — no override needed
  public_pool:
    route:
      path_prefix: "/api"
    backends: [...]

  # Override: trust a private CA for this upstream only
  internal_pool:
    tls:
      verify_certificates: true
      strict_sni: true
      ca_file: "/etc/spooky/certs/internal-ca.pem"
    route:
      path_prefix: "/internal"
    backends: [...]

  # Override: disable verification for a trusted dev upstream
  dev_pool:
    tls:
      verify_certificates: false
      strict_sni: false
    route:
      path_prefix: "/dev"
    backends: [...]

Load Balancing Configuration

Load balancing determines how requests are distributed across healthy backends within an upstream. Each upstream configures its own strategy independently.

Properties

Supported Algorithms

random

Selects a backend randomly from all healthy backends. Weight values are currently ignored.

upstream:
  my_pool:
    load_balancing:
      type: "random"

round-robin

Distributes requests evenly across all healthy backends in sequential order. Weight values are currently ignored.

upstream:
  my_pool:
    load_balancing:
      type: "round-robin"

consistent-hash

Routes requests using consistent hashing. By default it hashes request authority (if present), otherwise request path, otherwise HTTP method. Set load_balancing.key to override key derivation.

upstream:
  my_pool:
    load_balancing:
      type: "consistent-hash"
      key: "header:x-user-id"

least-connections

Selects the healthy backend with the fewest active requests. Ties are deterministic by backend index order.

upstream:
  my_pool:
    load_balancing:
      type: "least-connections"

latency-aware

Selects healthy backends using a latency score built from EWMA backend latency and active request pressure. Unsampled backends are probed first to avoid cold-start bias.

upstream:
  my_pool:
    load_balancing:
      type: "latency-aware"

sticky-cid

Uses consistent hashing keyed by QUIC connection ID for connection-level stickiness. The same CID is routed to the same backend while healthy membership is stable.

upstream:
  my_pool:
    load_balancing:
      type: "sticky-cid"

Algorithm Selection

• Use random for simple stateless load distribution
• Use round-robin for even distribution across backends
• Use consistent-hash when session affinity or request consistency is required
• Use least-connections when backend load varies significantly across requests
• Use latency-aware when you want faster backends to absorb more traffic
• Use sticky-cid for QUIC-connection affinity without application-level stickiness keys

Operational Implications

• round-robin and random are the easiest to reason about for initial rollouts.
• consistent-hash and sticky-cid improve affinity but make membership changes more visible to clients.
• least-connections and latency-aware depend more heavily on live runtime signals and should be paired with good observability.

Common Mistakes

• picking consistent-hash without a stable key that matches application behavior
• expecting backend weight to influence algorithms that currently ignore weights
• using sticky-cid to solve application-layer affinity problems that should use explicit request keys

Examples

upstream:
  api_pool:
    load_balancing:
      type: "consistent-hash"
    route:
      path_prefix: "/api"
    backends: [...]

  default_pool:
    load_balancing:
      type: "round-robin"
    route:
      path_prefix: "/"
    backends: [...]

Logging Configuration

Controls logging output, verbosity, and destination.

Properties

Log Levels

Log levels in order of increasing verbosity:

• silence: No logging output
• poltergeist: Error messages only
• scream: Warnings and errors
• spooky: Informational messages, warnings, and errors
• haunt: Debug information
• whisper: Trace-level debugging

Standard log level mapping:

• silence = off
• poltergeist = error
• scream = warn
• spooky = info
• haunt = debug
• whisper = trace

Examples

# stderr only (default)
log:
  level: info
  format: plain

# Write to file
log:
  level: info
  format: plain
  file:
    enabled: true
    path: /var/log/spooky/spooky.log

# Structured JSON logs (recommended for log pipelines)
log:
  level: info
  format: json

# Development — debug to stderr
log:
  level: haunt  # debug level
  format: plain

# Troubleshooting — trace to file
log:
  level: whisper  # trace level
  format: json
  file:
    enabled: true
    path: /tmp/spooky-trace.log

Operational Implications

• log.level reloads live, but log sink shape such as file output and format remains startup-owned.
• json is the safer default for production log pipelines.
• file logging adds local disk-management responsibility; stderr or journald avoids that at the cost of external collection requirements.

Common Mistakes

• enabling file logging without rotation
• using trace-level logging for sustained production traffic
• assuming format changes apply through live runtime activation

Performance Configuration

Controls resource limits, tuning knobs, and connection-flood protection. All fields are optional and fall back to sane defaults.

Properties

Connection flood protection

new_connections_per_sec and new_connections_burst implement a token-bucket rate limiter on new QUIC connection accepts. The bucket starts full so legitimate burst traffic at startup is never penalised. Packets for existing connections are never affected by this limit — only unknown Initial packets that would create a new connection state entry are gated.

max_active_connections is a separate hard guardrail for total connection state. Use it to enforce deterministic memory limits under sustained handshake floods even when token-bucket limits allow temporary bursts.

performance:
  new_connections_per_sec: 2000   # refill rate: 2 k new conns/sec
  new_connections_burst: 500      # allow a burst of up to 500 above the rate
  max_active_connections: 20000   # hard ceiling for concurrently tracked connections

Set new_connections_burst to 1 and new_connections_per_sec to a low value to aggressively throttle connection floods at the cost of rejecting legitimate concurrent handshakes.

Examples

# Single-worker, conservative limits
performance:
  worker_threads: 1
  global_inflight_limit: 1024
  new_connections_per_sec: 500
  new_connections_burst: 100

# High-throughput multi-worker setup
performance:
  worker_threads: 8
  reuseport: true
  pin_workers: true
  global_inflight_limit: 16384
  per_upstream_inflight_limit: 4096
  per_backend_inflight_limit: 256
  new_connections_per_sec: 10000
  new_connections_burst: 2000

Operational Implications

• worker_threads, reuseport, and shard settings shape how ingress work spreads across cores.
• inflight limits, timeouts, and body caps define overload behavior as much as raw performance.
• DNS refresh and connection-pool settings affect how quickly backend changes are observed.

Common Mistakes

• raising inflight limits without validating backend capacity and timeout posture
• setting very high body caps without thinking about memory pressure
• enabling aggressive multi-worker tuning before baseline observability is in place

Resilience Configuration

Controls retry budgets, circuit breaking, hedging, adaptive admission, brownout shedding, route queuing, protocol policy, and the worker watchdog. All fields are optional and fall back to production-tuned defaults.

Use this section when you need to decide:

• how Spooky protects itself and its backends under pressure
• which retry and hedge behaviors are allowed
• what request-shape rules are enforced before backend execution

adaptive_admission

Dynamically adjusts the global in-flight request limit based on observed backend latency.

circuit_breaker

Tracks consecutive failures per backend and opens the circuit to stop sending requests to a failing backend.

retry_budget

Limits retried requests as a fraction of primary requests to prevent retry amplification.

hedging

Fires a speculative second request to an alternate backend when the primary is slow.

brownout

Brownout is a load-shedding mode that activates when the proxy is near capacity. When active, every incoming request whose upstream pool is not in core_routes is immediately rejected with 503 Service Unavailable and a Retry-After header. Requests on core routes continue to be processed normally.

How it works

1. After each request is routed, Spooky samples the current global in-flight percent (active requests ÷ global limit × 100).
2. If the sample reaches trigger_inflight_percent, brownout activates.
3. Brownout stays active until the sample falls to or below recover_inflight_percent. The gap between the two thresholds is hysteresis — it prevents rapid oscillation when load is right at the boundary.
4. While active, spooky_brownout_active gauge is 1 and spooky_overload_shed_by_reason_total{reason="brownout"} increments for every shed request.

Choosing core_routes

core_routes is a list of upstream pool names (the id field under upstreams[].pool). Routes not in this list are shed during brownout.

• If core_routes is empty (the default), all routes are shed during brownout. This is safe but means brownout effectively becomes a full-stop — no requests get through.
• List only the routes that must keep working during a partial outage: authentication, payments, health checks. Avoid listing high-volume non-critical routes or you defeat the purpose of shedding.
• A route shed during brownout receives a 503 with the body brownout active, non-core route shed and a Retry-After hint. Clients that respect Retry-After will back off automatically.

Interaction with other overload mechanisms

Brownout runs after routing but before adaptive admission and circuit breakers. The order is:

1. Brownout — shed non-core routes immediately (no backend resource consumed)
2. Adaptive admission — dynamically cap total in-flight based on observed latency
3. Per-upstream / per-backend inflight limits — static caps per pool and backend
4. Circuit breaker — stop sending to a specific failing backend

If brownout is active and shedding load, adaptive admission will also begin to recover (inflight drops → limit rises). Once the in-flight percent falls to recover_inflight_percent, brownout deactivates and full traffic resumes. Set recover_inflight_percent at least 20–30 points below trigger_inflight_percent to give the system time to recover before re-admitting full traffic.

Alerting

Alert on spooky_brownout_active == 1 for more than a brief window — sustained brownout means backends are under-provisioned or a downstream dependency is slow:

- alert: SpookyBrownoutActive
  expr: spooky_brownout_active == 1
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "Spooky brownout active on {{ $labels.instance }}"
    description: "Non-core routes are being shed. Check backend latency and inflight metrics."

route_queue

Per-route and global caps on queued (waiting) requests.

protocol

Request validation and early-data policy.

Request-shape rules enforced by the runtime:

• HTTP/3 requests are rejected when :authority and Host differ and enforce_authority_host_match is enabled.
• CONNECT requires :authority/Host in host:port form and must also satisfy the CONNECT allowlists when enabled.
• Native HTTP/3 ingress rejects Upgrade / Connection: upgrade style requests. WebSocket-style upgrades are only supported on the bootstrap HTTP/1.1 compatibility path, not on native H3.
• HEAD responses terminate after headers even if the upstream attempted to send a body.

watchdog

Monitors worker health and triggers a restart command when error rates or stall conditions exceed thresholds.

Startup Validation Errors

The following resilience configurations are rejected at startup with a descriptive error:

Example

resilience:
  adaptive_admission:
    enabled: true
    min_limit: 64
    max_limit: 4096
    high_latency_ms: 500

  circuit_breaker:
    enabled: true
    failure_threshold: 3
    open_ms: 30000
    half_open_max_probes: 1

  retry_budget:
    enabled: true
    ratio_percent: 10

  hedging:
    enabled: false
    delay_ms: 100

  brownout:
    enabled: true
    trigger_inflight_percent: 90
    recover_inflight_percent: 60
    core_routes:
      - "auth_pool"
      - "payments_pool"

Operational Implications

• adaptive_admission, brownout, route_queue, and inflight caps interact as one overload-control surface.
• retry, hedging, and circuit breaking can protect latency or amplify backend pressure depending on how they are tuned.
• quota and scoped rate limiting are policy-contract features; they should not be interpreted as overload behavior.

Common Mistakes

• leaving brownout.core_routes empty and unintentionally shedding all routes during brownout
• enabling hedging without understanding replay safety and backend amplification
• treating retry budgets as a substitute for backend reliability work
• mixing quota expectations with overload tuning

Observability Endpoint Hardening

When enabling observability.metrics or observability.control_api, keep endpoints on loopback unless you intentionally expose them behind network controls.

Use this section when you need to decide:

• where metrics and the Control API should bind
• how much runtime control to expose
• how strongly the admin surface must be protected

Metrics Endpoint

Key fields:

• observability.metrics.max_connections (default: 512): concurrent connection cap.
• observability.metrics.connection_timeout_ms (default: 30000): per-connection lifetime timeout.

Control API Endpoint

Key fields:

• observability.control_api.auth_token: bearer token required for runtime, reload, reload-certs, and restart endpoints (Authorization: Bearer <token>).
• observability.control_api.reload_path (default: /admin/runtime/reload): authenticated POST endpoint that re-reads the config file and applies the full configuration via an atomic runtime swap (routes, upstreams, backends, timeouts, limits, resilience policies). Startup-owned settings and listener bind/removal changes are rejected and still require a restart.
• observability.control_api.reload_certs_path: authenticated POST endpoint that reloads listener certificate and client-auth CA material for new handshakes.
• observability.control_api.max_connections (default: 256): concurrent connection cap.
• observability.control_api.connection_timeout_ms (default: 30000): per-connection lifetime timeout.

If observability.control_api.address is non-loopback, observability.control_api.auth_token is required.

Routing Transparency

observability.routing enables explicit route-decision logging.

Watchdog Restart Hook

Use structured command execution:

• resilience.watchdog.restart_command: array, where index 0 is executable and remaining entries are arguments.

Legacy resilience.watchdog.restart_hook is deprecated and rejected by validation.

Common Mistakes

• exposing the Control API on 0.0.0.0 for convenience
• assuming health and readiness endpoints are enough protection for admin-plane exposure
• using the legacy reload shortcut when staged validate/preview/activate flows are the safer operational path

Configuration Validation

Spooky validates configuration at startup and reports errors before attempting to start the server.

Common Validation Errors

1. Missing required fields
2. Neither listen.tls.cert/key nor listen.tls.certificates specified
3. Backend address or ID missing

4. Route configuration empty

5. Invalid file paths

6. TLS certificate file not found or not readable
7. TLS key file not found or not readable

8. Incorrect file permissions

9. Invalid values

10. Port number out of range (1-65535)
11. Invalid IP address format
12. Invalid backend address format (accepted: host:port, https://host:port, http://host:port, or bare host; scheme-default port is inferred when omitted)

13. Duplicate backend IDs within a pool

14. Configuration conflicts

15. Port already in use
16. Duplicate upstream pool names
17. Overlapping or ambiguous route definitions
18. Brownout recover_inflight_percent ≥ trigger_inflight_percent
19. adaptive_admission.min_limit set to 0
20. retry_budget.ratio_percent > 100
21. hedging.enabled with delay_ms = 0

Testing Configuration

Validate configuration without starting the server:

spooky --config <path>

The command exits with status 0 if configuration is valid, or prints detailed error messages and exits with non-zero status if invalid.

Complete Working Example

version: 1

listen:
  protocol: http3
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "certs/proxy-fullchain.pem"
    key: "certs/proxy-key-pkcs8.pem"

upstream:
  api_pool:
    load_balancing:
      type: "consistent-hash"

    route:
      path_prefix: "/api"

    backends:
      - id: "backend1"
        address: "https://127.0.0.1:7001"
        weight: 100
        health_check:
          path: "/health"
          interval: 5000

      - id: "backend2"
        address: "https://127.0.0.1:7002"
        weight: 50
        health_check:
          path: "/status"
          interval: 10000

  default_pool:
    load_balancing:
      type: "round-robin"

    route:
      path_prefix: "/"

    backends:
      - id: "auth1"
        address: "https://127.0.0.1:8001"
        weight: 100
        health_check:
          path: "/health"
          interval: 5000

log:
  level: debug
  format: plain