Use this page to look up control-plane endpoints, roles, request fields, and response semantics.

Open These First

Use this table when you need the fastest runtime-introspection path:

Endpoint Index

Common Control API Flows

Read Current Runtime State

curl -k --http1.1 \
  -H "Authorization: Bearer <token>" \
  https://127.0.0.1:9902/admin/runtime

Validate, Preview, and Activate a Candidate

curl -k --http1.1 -X POST \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  https://127.0.0.1:9902/admin/runtime/validate \
  -d '{"config_path":"/etc/spooky/candidate.yaml","requested_by":"ops","reason":"preflight"}'

curl -k --http1.1 -X POST \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  https://127.0.0.1:9902/admin/runtime/preview \
  -d '{"config_path":"/etc/spooky/candidate.yaml","requested_by":"ops","reason":"preview"}'

curl -k --http1.1 -X POST \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  https://127.0.0.1:9902/admin/runtime/activate \
  -d '{"config_path":"/etc/spooky/candidate.yaml","expected_generation":12,"requested_by":"ops","reason":"deploy"}'

Roll Back to a Retained Generation

curl -k --http1.1 \
  -H "Authorization: Bearer <token>" \
  https://127.0.0.1:9902/admin/runtime/history

curl -k --http1.1 -X POST \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  https://127.0.0.1:9902/admin/runtime/rollback \
  -d '{"target_generation":11,"expected_active_generation":12,"requested_by":"ops","reason":"rollback"}'

Protocol

The Control API uses HTTP/1.1 over TLS. HTTP/2 is not supported.

When using curl, pass --http1.1 explicitly — curl negotiates h2 by default when connecting to a TLS endpoint and the server will reject the connection:

curl -k --http1.1 https://<address>:<port>/...

The -k flag skips certificate verification for self-signed certs.

Authentication

Supported authentication shapes:

• bearer token only
• mTLS only
• mTLS + bearer token

Authentication and authorization are separate concerns:

• authentication proves who the caller is
• authorization decides whether the caller has viewer, operator, or admin

Bearer-token form:

Authorization: Bearer <token>

Compatibility note:

• observability.control_api.auth_token remains supported as the legacy single-token admin credential
• the legacy token is mapped internally to a static admin identity so existing operators keep current restart/reload privileges during migration
• new deployments should prefer observability.control_api.auth.bearer_tokens[] with explicit roles
• compatibility boundary: a new Spooky binary accepts legacy auth_token configs, but an older binary will reject configs that use the newer nested control-plane fields because ControlApi uses strict deny_unknown_fields

Role model:

• viewer: runtime snapshot and history reads
• operator: viewer plus validate, preview, activate, rollback, reload, and cert reload
• admin: operator plus restart and future destructive admin actions

Route Access Rules

Route families:

• /health and /ready: unauthenticated or separately protected
• /admin/runtime* reads: viewer
• runtime mutation routes except restart: operator
• /admin/runtime/restart: admin

Contract rules:

• viewer is the minimum privileged read role
• operator is the minimum non-restart mutation role
• admin is required for restart
• implementation should distinguish invalid authentication from insufficient role

Response Contract

Privileged routes distinguish authentication failure from authorization failure:

• 401 Unauthorized: missing authentication or invalid authentication
• 403 Forbidden: authenticated caller is under-scoped, or the source-address policy rejected the request

Representative reasons returned in JSON payloads:

• missing_authentication
• invalid_bearer_token
• insufficient_role
• source_ip_not_allowed

When control API mTLS is configured as required, missing or invalid client certificates are rejected during the TLS handshake before HTTP routing. That failure does not produce an HTTP 401 or 403 response.

Configuration Patterns

Bearer-Only Local Dev

Use this for loopback-only development or local automation.

observability:
  control_api:
    enabled: true
    address: "127.0.0.1"
    port: 9890
    auth_token: "change-me-local-dev"

mTLS Optional With Viewer Token

Use this when you want to accept client certificates without making them mandatory yet.

observability:
  control_api:
    enabled: true
    address: "127.0.0.1"
    port: 9902
    tls:
      client_auth:
        mode: optional
        ca_file: "/etc/spooky/pki/admin-ca.pem"
    auth:
      bearer_tokens:
        - token: "viewer-token"
          role: viewer
          actor_id: "ops-readonly"

mTLS Required With Operator/Admin Identities

Use this for hardened production admin-plane access.

observability:
  control_api:
    enabled: true
    required: true
    address: "10.0.10.5"
    port: 9902
    tls:
      client_auth:
        mode: required
        ca_file: "/etc/spooky/pki/admin-ca.pem"
    auth:
      bearer_tokens:
        - token: "operator-token"
          role: operator
          actor_id: "ops-automation"
        - token: "admin-token"
          role: admin
          actor_id: "platform-admin"
      identity_source:
        kind: "mtls_subject_cn"
        role_attribute: "OU"
    ip_allowlist:
      cidrs:
        - "10.0.10.0/24"
    audit:
      enabled: true
      format: json
      sink: log

Endpoints

GET /health

Purpose:

• liveness check
• watchdog state visibility

Expected use:

• load balancer or platform liveness probe
• operator sanity check

GET /ready

Purpose:

• readiness state for serving traffic

Expected use:

• deployment orchestration
• maintenance and rollout checks

GET /admin/runtime

Purpose:

• runtime snapshot for operators

Minimum role:

• viewer

Typical contents include:

• worker and runtime state
• key counters
• admission state
• backend health summary
• quota backend health summary
• observability package metadata
• recent admin actions when available
• dashboard and documentation references for the shipped operator bundle

Expected use:

• debugging
• rollout validation
• incident response

The observability block is the packaged runtime-introspection entry point for operators. The high-signal fields are:

• contract_version
• audit_schema_version
• current_generation
• dashboard_packages
• documentation
• backend_health_summary
• quota_backend_health_summary
• recent_admin_actions

Example:

{
  "generation": 12,
  "readiness": "ready",
  "observability": {
    "contract_version": "v1",
    "audit_schema_version": "v1",
    "current_generation": 12,
    "backend_health_summary": {
      "healthy": 7,
      "unhealthy": 1
    },
    "quota_backend_health_summary": {
      "backend_mode": "redis",
      "availability": "available"
    }
  }
}

POST /admin/runtime/validate

Purpose:

• parse and validate a candidate config, and report whether it could be activated — without touching the running runtime

Returns 200 with a plan describing the candidate generation, a per-domain diff, and any rejected changes. A config that cannot be activated still returns 200; inspect rejected_changes and candidate_status rather than relying on the status code.

Accepts the same optional body fields as /admin/runtime/reload.

Expected use:

• CI gating on config changes before a deploy
• confirming a config is loadable before scheduling a maintenance window

Minimum role:

• operator

Example:

curl -k --http1.1 -X POST https://127.0.0.1:9902/admin/runtime/validate \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  -d '{"config_path":"/etc/spooky/candidate.yaml","requested_by":"ops","reason":"preflight"}'

POST /admin/runtime/preview

Purpose:

• same planning work as validate, recorded in generation history as an operator preview

Returns 200 with the same plan shape as validate. Neither endpoint mutates the active generation.

Expected use:

• operator dry-run immediately before an activation, when you want the attempt in the audit trail

Minimum role:

• operator

POST /admin/runtime/activate

Purpose:

• stage and commit a config change, returning the structured activation result

Returns 202 on success. Failures are classified rather than collapsed into 500:

Accepts the same optional body fields as /admin/runtime/reload.

Expected use:

• the preferred activation path — prefer this over the legacy /reload shortcut, since it returns the full diff, rejection detail, and generation history entry

Minimum role:

• operator

Example:

curl -k --http1.1 -X POST https://127.0.0.1:9902/admin/runtime/activate \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  -d '{"config_path":"/etc/spooky/candidate.yaml","expected_generation":12,"requested_by":"ops","reason":"deploy"}'

POST /admin/runtime/rollback

Purpose:

• restore a previously retained runtime generation

Required request body:

Returns 202 on success. Failures:

Use GET /admin/runtime/history first to pick a target whose rollback_candidate is true.

Minimum role:

• operator

Example:

curl -k --http1.1 -X POST https://127.0.0.1:9902/admin/runtime/rollback \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  -d '{"target_generation": 3}'

GET /admin/runtime/history

Purpose:

• list retained runtime generations and the recorded history of control-plane operations

Response shape:

Each entry in retained_generations:

status: failed_prepare records a candidate generation that was never successfully prepared; it has has_bundle: false and carries the failure reason in note.

Expected use:

• choosing a safe rollback target
• auditing who changed runtime config, when, and from which config source
• diagnosing why a staged activation never committed
• correlating runtime operations with audit and observability views

Minimum role:

• viewer

Example:

curl -k --http1.1 \
  -H "Authorization: Bearer <token>" \
  https://127.0.0.1:9902/admin/runtime/history

GET /admin/runtime/history/{generation}

Purpose:

• the retained-generation record and operation entries for a single generation

Returns 200 with generation, a single retained_generation object (same shape as above), and the entries recorded against it. Returns 404 if that generation is not retained.

Minimum role:

• viewer

POST /admin/runtime/reload

Legacy shortcut. Prefer POST /admin/runtime/activate, which returns the full diff and rejection detail.

Purpose:

• reload the full config from disk and apply changes to upstreams, backends, policies, timeouts, and log.level

Config source:

• with no request body, the reload re-reads the currently active runtime config source
• on a freshly started process that source is the path passed at startup, but activating an alternate config_path makes that file the active source for every later reload
• pass config_path in the body to read a different file; a successful activation makes that path the new active source

Important scope note:

• listener bind addresses, control API bind, and metrics bind cannot change without a restart
• log format/file settings, tracing config (observability.tracing.*), and performance.control_plane_threads also require a restart (a reload changing them is rejected); log.level, however, is applied live
• in-flight requests on the old config complete normally; new requests use the new config immediately

Expected use:

• adding or removing backends
• changing load balancing, timeouts, resilience, or routing policy at runtime

Minimum role:

• operator

Optional request body:

Example:

curl -k --http1.1 -X POST https://127.0.0.1:9902/admin/runtime/reload \
  -H "Authorization: Bearer <token>"

Activating an alternate config file:

curl -k --http1.1 -X POST https://127.0.0.1:9902/admin/runtime/reload \
  -H "Authorization: Bearer <token>" \
  -H "content-type: application/json" \
  -d '{"config_path": "/etc/spooky/canary.yaml"}'

POST /admin/runtime/reload-certs

Purpose:

• reload listener certificate and related trust material for new handshakes

Important scope note:

• this is not full config hot reload
• existing sessions keep their already-negotiated certificate and auth state

Expected use:

• listener certificate rotation
• listener trust-material refresh

Minimum role:

• operator

POST /admin/runtime/restart

Purpose:

• request a controlled restart/drain workflow through the watchdog coordinator

Expected use:

• operational restart requests
• orchestrated maintenance flow

Minimum role:

• admin

Audit Configuration And Event Shape

The control API audit stream is the operator history surface for admin-plane actions.

Example:

observability:
  control_api:
    audit:
      enabled: true
      format: json
      sink: log

Current audit schema version:

• v1

The stable top-level event fields are:

• schema_version
• event_id
• event_type
• time_unix_ms
• request_id
• trace_id
• span_id
• listener
• actor
• action
• target
• generation
• result
• reason
• failure_class
• peer_addr
• authn

Use audit for actor attribution, authn and authz failure history, and attempt-versus-result correlation for runtime operations.

Related Pages

• Metrics Reference
• Control Plane
• Operations Runbook