Guide for configuring TLS certificates for HTTP/3 connections in Spooky.

Overview

HTTP/3 uses QUIC as its transport protocol, which requires TLS 1.3 for encryption and authentication. Spooky requires valid TLS certificates to establish secure connections with clients.

Requirements

Protocol Requirements

• TLS 1.3 (required for QUIC/HTTP3)
• ALPN (Application-Layer Protocol Negotiation) support
• SNI (Server Name Indication) support

Supported Formats

• Certificates: PEM-encoded X.509 certificates
• Private Keys: PEM-encoded PKCS#8 format (recommended) or traditional RSA/ECDSA formats
• Key Types: RSA (2048-bit minimum) or ECDSA (P-256, P-384)

Certificate Generation

Development: Self-Signed Certificates with mkcert

For local development, mkcert generates locally-trusted certificates:

# Install mkcert
# Ubuntu/Debian
sudo apt install mkcert

# macOS
brew install mkcert

# Install local CA
mkcert -install

# Generate certificate for localhost
mkdir -p certs
cd certs
mkcert -key-file server.key -cert-file server.crt localhost 127.0.0.1 ::1

# Verify generation
ls -lh server.crt server.key

Configuration:

listen:
  protocol: http3
  port: 9889
  address: "127.0.0.1"
  tls:
    cert: "certs/server.crt"
    key: "certs/server.key"

Development: Self-Signed Certificates with OpenSSL

For environments where mkcert is not available:

# Create certificate directory
mkdir -p certs
cd certs

# Generate private key (RSA 2048-bit)
openssl genrsa -out server.key 2048

# Generate certificate signing request
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Development/CN=localhost"

# Generate self-signed certificate (valid 365 days)
openssl x509 -req -in server.csr -signkey server.key \
  -out server.crt -days 365 -sha256

# Convert key to PKCS#8 format (recommended)
openssl pkcs8 -topk8 -nocrypt -in server.key -out server-pkcs8.key

# Verify certificate
openssl x509 -in server.crt -text -noout

# Clean up CSR
rm server.csr

Configuration:

listen:
  protocol: http3
  port: 9889
  address: "127.0.0.1"
  tls:
    cert: "certs/server.crt"
    key: "certs/server-pkcs8.key"

Production: Let's Encrypt

For production deployments with public domains:

# Install certbot
sudo apt update
sudo apt install certbot

# Option 1: Standalone mode (requires port 80 available)
sudo certbot certonly --standalone \
  -d example.com \
  -d www.example.com

# Option 2: DNS challenge (no port requirements)
sudo certbot certonly --manual \
  --preferred-challenges dns \
  -d example.com

# Certificates are saved to:
# Certificate: /etc/letsencrypt/live/example.com/fullchain.pem
# Private Key: /etc/letsencrypt/live/example.com/privkey.pem

Configuration:

listen:
  protocol: http3
  port: 9889
  address: "0.0.0.0"
  tls:
    cert: "/etc/letsencrypt/live/example.com/fullchain.pem"
    key: "/etc/letsencrypt/live/example.com/privkey.pem"

Production: ECDSA Certificates

ECDSA certificates offer better performance than RSA:

# Generate ECDSA private key (P-256)
openssl ecparam -genkey -name prime256v1 -out server-ec.key

# Convert to PKCS#8 format
openssl pkcs8 -topk8 -nocrypt -in server-ec.key -out server-ec-pkcs8.key

# Generate CSR
openssl req -new -key server-ec-pkcs8.key -out server-ec.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=example.com"

# Generate self-signed certificate (or send CSR to CA)
openssl x509 -req -in server-ec.csr -signkey server-ec-pkcs8.key \
  -out server-ec.crt -days 365 -sha256

Certificate Configuration

Basic Configuration

Minimal TLS configuration for HTTP/3:

listen:
  protocol: http3
  port: 9889
  address: "0.0.0.0"
  tls:
    cert: "/path/to/certificate.pem"
    key: "/path/to/private-key.pem"

Path Specifications

Paths can be absolute or relative:

# Absolute paths (recommended for production)
tls:
  cert: "/etc/spooky/certs/fullchain.pem"
  key: "/etc/spooky/certs/privkey.pem"

# Relative paths (relative to working directory)
tls:
  cert: "certs/server.crt"
  key: "certs/server.key"

Multi-Domain Certificates

For certificates covering multiple domains (SAN certificates):

# Generate certificate with Subject Alternative Names
openssl req -new -x509 -key server.key -out server.crt -days 365 \
  -subj "/CN=example.com" \
  -addext "subjectAltName=DNS:example.com,DNS:www.example.com,DNS:api.example.com"

Configuration remains the same:

tls:
  cert: "/etc/spooky/certs/multi-domain.crt"
  key: "/etc/spooky/certs/multi-domain.key"

Multi-Certificate SNI Configuration

Use listen.tls.certificates when one listener must present different certificates by hostname:

listen:
  protocol: http3
  address: "0.0.0.0"
  port: 9889
  tls:
    cert: "/etc/spooky/certs/default-fullchain.pem"
    key: "/etc/spooky/certs/default-privkey.pem"
    certificates:
      - server_name: "api.example.com"
        cert: "/etc/spooky/certs/api-fullchain.pem"
        key: "/etc/spooky/certs/api-privkey.pem"
      - server_name: "www.example.com"
        cert: "/etc/spooky/certs/www-fullchain.pem"
        key: "/etc/spooky/certs/www-privkey.pem"

Selection order:

1. Exact server_name match in listen.tls.certificates
2. Fallback listen.tls.cert + listen.tls.key
3. If no legacy fallback pair is configured, the first certificates[] entry becomes the default identity

Fallback behavior details:

• Both the native QUIC/HTTP/3 listener and the bootstrap TLS listener use the same selection order.
• server_name matching is exact after hostname normalization. There is no wildcard SNI certificate lookup here; wildcard behavior must come from the certificate SANs themselves, not from the listener map.
• If the client sends no SNI, Spooky always serves the default identity.
• If the client sends an SNI hostname that is not present in listen.tls.certificates, Spooky serves the default identity rather than rejecting the handshake.
• Startup rejects any listen.tls.certificates[].server_name mapping whose configured certificate SANs do not cover that hostname.

File Permissions and Security

Recommended Permissions

Restrict access to certificate files:

# Create dedicated certificate directory
sudo mkdir -p /etc/spooky/certs
sudo chown spooky:spooky /etc/spooky/certs
sudo chmod 700 /etc/spooky/certs

# Set certificate permissions
sudo chmod 644 /etc/spooky/certs/server.crt
sudo chmod 600 /etc/spooky/certs/server.key

# Verify permissions
ls -l /etc/spooky/certs/

Expected output:

drwx------ 2 spooky spooky 4096 Dec 15 10:00 .
-rw-r--r-- 1 spooky spooky 1234 Dec 15 10:00 server.crt
-rw------- 1 spooky spooky 1704 Dec 15 10:00 server.key

Security Best Practices

1. Private Key Protection
2. Never commit private keys to version control
3. Use restrictive file permissions (600)
4. Store keys on encrypted filesystems

5. Consider using hardware security modules (HSM) for production

6. Certificate Chain Validation

7. Use complete certificate chains (fullchain.pem with Let's Encrypt)
8. Include intermediate certificates

9. Verify chain with openssl verify

10. Certificate Monitoring

11. Monitor expiration dates
12. Set up renewal automation for Let's Encrypt
13. Implement alerting for certificates expiring within 30 days
14. Scrape:
    • spooky_downstream_tls_certificate_not_after_seconds
    • spooky_downstream_tls_certificate_days_remaining

Certificate Validation

Verify Certificate and Key Match

Ensure certificate and private key are paired correctly:

# Extract modulus from certificate
cert_modulus=$(openssl x509 -noout -modulus -in server.crt | md5sum)

# Extract modulus from private key
key_modulus=$(openssl rsa -noout -modulus -in server.key | md5sum)

# Compare (should be identical)
echo "Certificate: $cert_modulus"
echo "Private Key: $key_modulus"

For ECDSA keys:

# Verify ECDSA private key
openssl ec -in server-ec.key -check

# Verify certificate
openssl x509 -in server-ec.crt -text -noout

Verify Certificate Properties

Check certificate details:

# Display certificate information
openssl x509 -in server.crt -text -noout

# Check expiration date
openssl x509 -in server.crt -noout -enddate

# Check subject and issuer
openssl x509 -in server.crt -noout -subject -issuer

# Verify certificate chain
openssl verify -CAfile ca.crt server.crt

Test Configuration

Verify Spooky can load certificates:

# Test configuration validity
spooky --config config.yaml

# Run in debug mode to see TLS initialization
# Set log level in config.yaml (log.level) or via RUST_LOG=debug
spooky --config config.yaml

Certificate Rotation and Renewal

Let's Encrypt Automatic Renewal

Let's Encrypt certificates are valid for 90 days. Set up automatic renewal:

# Test renewal process
sudo certbot renew --dry-run

# Enable automatic renewal (certbot installs systemd timer)
sudo systemctl status certbot.timer

# Manually renew certificates
sudo certbot renew

# Reload listener certificates for new handshakes
curl -X POST \
  -H "Authorization: Bearer ${SPOOKY_CONTROL_API_TOKEN}" \
  https://127.0.0.1:9902/admin/runtime/reload-certs

Manual Certificate Rotation

For manually-managed certificates:

# Backup current certificates
sudo cp /etc/spooky/certs/server.crt /etc/spooky/certs/server.crt.backup
sudo cp /etc/spooky/certs/server.key /etc/spooky/certs/server.key.backup

# Install new certificates
sudo cp new-server.crt /etc/spooky/certs/server.crt
sudo cp new-server.key /etc/spooky/certs/server.key

# Set permissions
sudo chmod 644 /etc/spooky/certs/server.crt
sudo chmod 600 /etc/spooky/certs/server.key

# Reload listener certificates for new handshakes
curl -X POST \
  -H "Authorization: Bearer ${SPOOKY_CONTROL_API_TOKEN}" \
  https://127.0.0.1:9902/admin/runtime/reload-certs

# Verify new certificates are loaded
openssl s_client -connect localhost:9889 -servername localhost < /dev/null 2>/dev/null | openssl x509 -noout -dates

Reload behavior:

• New QUIC and bootstrap TLS handshakes use the updated certificate material immediately after reload succeeds.
• Existing connections are not interrupted or re-handshaken.
• Existing QUIC connections keep the certificate and client-auth policy that were negotiated when their handshake completed. The new certificate material is only visible to later QUIC Initial packets and later bootstrap TCP+TLS accepts.
• Existing HTTP/2 streams multiplexed over an already-established bootstrap TLS session are also unaffected. Only brand-new bootstrap TLS sessions observe the reloaded certificate set.

Downstream TLS Metrics

Spooky exposes downstream TLS observability through Prometheus:

• spooky_downstream_tls_handshake_failure_total{listener,reason}
• spooky_downstream_tls_certificate_selection_total{listener,selection}
• spooky_downstream_tls_alpn_total{listener,protocol}
• spooky_downstream_tls_certificate_not_after_seconds{listener,server_name}
• spooky_downstream_tls_certificate_days_remaining{listener,server_name}

Important label values:

• reason=missing_client_cert: mTLS listener required a client cert and none was presented
• reason=invalid_client_cert: client cert was present but rejected for a generic certificate validation reason
• reason=expired_client_cert: client cert was expired or not yet valid
• reason=unknown_issuer: client cert chain was not rooted in the configured CA set
• reason=alpn: handshake failed because no acceptable application protocol could be negotiated
• reason=handshake: fallback bucket for other downstream TLS handshake failures

Certificate-selection labels:

• selection=exact_sni: exact SNI match in listen.tls.certificates
• selection=fallback_unmatched_sni: client sent SNI, but no configured mapping matched, so Spooky served the default identity
• selection=fallback_no_sni: client sent no SNI and Spooky served the default identity while additional SNI identities existed
• selection=default_only: listener had only one effective identity, so that certificate was always served

Monitoring Certificate Expiry

Check certificate expiration:

# Check days until expiry
openssl x509 -in /etc/spooky/certs/server.crt -noout -enddate

# Calculate days remaining
days_left=$(( ($(date -d "$(openssl x509 -in /etc/spooky/certs/server.crt -noout -enddate | cut -d= -f2)" +%s) - $(date +%s)) / 86400 ))
echo "Certificate expires in $days_left days"

# Alert if less than 30 days
if [ $days_left -lt 30 ]; then
  echo "WARNING: Certificate expires soon!"
fi

Troubleshooting

Common Issues

Certificate File Not Found

Error: failed to read certificate file: No such file or directory

Solution:

# Verify file exists
ls -l /etc/spooky/certs/server.crt

# Check path in configuration
cat config.yaml | grep -A2 tls

# Use absolute paths
realpath certs/server.crt

Permission Denied

Error: failed to read certificate file: Permission denied

Solution:

# Check file permissions
ls -l /etc/spooky/certs/

# Fix permissions
sudo chown spooky:spooky /etc/spooky/certs/server.{crt,key}
sudo chmod 644 /etc/spooky/certs/server.crt
sudo chmod 600 /etc/spooky/certs/server.key

# Verify Spooky user can read files
sudo -u spooky cat /etc/spooky/certs/server.crt > /dev/null

Invalid Certificate Format

Error: failed to parse certificate: invalid PEM format

Solution:

# Verify PEM format
openssl x509 -in server.crt -text -noout

# Check file encoding
file server.crt

# Convert DER to PEM if needed
openssl x509 -inform DER -in server.der -out server.pem

Certificate and Key Mismatch

Error: certificate and private key do not match

Solution:

# Verify certificate and key match (RSA)
openssl x509 -noout -modulus -in server.crt | md5sum
openssl rsa -noout -modulus -in server.key | md5sum

# Verify ECDSA key
openssl ec -in server.key -pubout -out server-pub.pem
openssl x509 -in server.crt -pubkey -noout -out cert-pub.pem
diff server-pub.pem cert-pub.pem

PKCS#8 Format Required

Some systems require PKCS#8 format:

# Convert traditional RSA to PKCS#8
openssl pkcs8 -topk8 -nocrypt -in server.key -out server-pkcs8.key

# Update configuration to use PKCS#8 key

Testing TLS Connections

Test with OpenSSL

# Test TLS 1.3 connection
echo -e "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" | \
  openssl s_client -connect localhost:9889 -servername localhost -tls1_3

# Display certificate chain
openssl s_client -connect localhost:9889 -servername localhost -showcerts < /dev/null

# Check ALPN negotiation
openssl s_client -connect localhost:9889 -servername localhost -alpn h3 < /dev/null

Test with cURL (HTTP/3 Support)

If curl is built with HTTP/3 support:

# Test HTTP/3 connection
curl --http3 https://localhost:9889/

# Verbose output for debugging
curl --http3 -v https://localhost:9889/

# Test with self-signed certificate
curl --http3 -k https://localhost:9889/

Debug Logging

Enable debug logging to troubleshoot TLS issues:

log:
  level: debug

Look for log entries related to:

• Certificate loading
• TLS handshake
• QUIC connection establishment
• ALPN negotiation

Common Error Messages

Reference

Configuration Schema

listen:
  tls:
    cert: string    # Path to PEM certificate file (required)
    key: string     # Path to PEM private key file (required)

Supported Key Algorithms

• RSA 2048-bit (minimum)
• RSA 4096-bit (recommended for long-term use)
• ECDSA P-256 (secp256r1)
• ECDSA P-384 (secp384r1)

Certificate Requirements

• PEM encoding
• X.509 format
• Valid date range (not expired, not future-dated)
• Subject Alternative Names (SAN) for multi-domain support
• Complete certificate chain (including intermediates)