Clustering & KV Store Architecture

This document covers the in-memory KV store, gossip-based clustering, hash % nodes key ownership, large-value replication, and all features that depend on the KV layer (ACME cert coordination, PHP response cache, session storage).

Doc status: this is a broad design document written ahead of implementation. Sections describing the KV store, gossip transport security, key ownership, and replication have been reconciled with the shipped code (crates/ephpm-cluster/). Some later sections (auto-instrumentation, admin UI wiring, and the illustrative session-handler C listing) still describe the intended design and may not match the code line-for-line — those spots are flagged inline.

Why Build Our Own KV Store

No competitor (FrankenPHP, RoadRunner, Swoole) has built-in multi-node KV clustering. They all require external Redis or Memcached. ePHPm’s integrated KV store is a headline differentiator.

We evaluated embedding existing databases (Redis, Dragonfly, KeyDB, Garnet) and rejected them:

Requirements

Single-Node KV Store

The foundation. Must be implemented and stable before any clustering work begins.

Core Data Structure

The core is a DashMap — a sharded, lock-free concurrent hashmap. Millions of ops/sec with zero contention on reads and fine-grained write locks.

use dashmap::DashMap;
use std::time::Instant;

struct KvEntry {
    value: KvValue,
    expires_at: Option<Instant>,
}

/// Strings and hashes are implemented. Covers cache, sessions, object cache.
/// Lists, sets, sorted sets added later if needed (see Data Structure Roadmap below).
/// Note: in the actual implementation, strings use `DashMap<String, Entry>` and
/// hashes use a separate `DashMap<String, HashEntry>` for type safety.
enum KvValue {
    String(Vec<u8>),
    Hash(HashMap<Vec<u8>, Vec<u8>>),
}

struct KvStore {
    data: DashMap<Vec<u8>, KvEntry>,
    memory_limit: usize,
    eviction_policy: EvictionPolicy,
}

enum EvictionPolicy {
    NoEviction,    // return error when full
    AllKeysLru,    // evict least recently used
    VolatileLru,   // evict LRU among keys with TTL
    AllKeysRandom,
}

Operations

impl KvStore {
    fn get(&self, key: &[u8]) -> Option<Vec<u8>> {
        let entry = self.data.get(key)?;
        // Lazy expiry: check TTL on access
        if let Some(exp) = entry.expires_at {
            if Instant::now() > exp {
                drop(entry);
                self.data.remove(key);
                return None;
            }
        }
        match &entry.value {
            KvValue::String(v) => Some(v.clone()),
            _ => None, // type mismatch
        }
    }

    fn set(&self, key: Vec<u8>, value: Vec<u8>, ttl: Option<Duration>) {
        let entry = KvEntry {
            value: KvValue::String(value),
            expires_at: ttl.map(|d| Instant::now() + d),
        };
        self.data.insert(key, entry);
    }
}

TTL Expiry

Dual strategy: lazy expiry on access (check TTL on every get) + active expiry via background sweeper:

async fn expiry_sweeper(store: Arc<KvStore>, interval: Duration) {
    let mut ticker = tokio::time::interval(interval);
    loop {
        ticker.tick().await;
        let now = Instant::now();
        store.data.retain(|_, entry| {
            entry.expires_at.map_or(true, |exp| now < exp)
        });
    }
}

This prevents memory leaks from keys that are set-and-forgotten.

Crate Dependencies

Configuration

[kv]
memory_limit = "256MB"
eviction_policy = "allkeys-lru"  # noeviction, allkeys-lru, volatile-lru, allkeys-random

PHP Access

Three ways for PHP to access the KV store, each serving different use cases. All three work transparently with clustering — local keys are fast-pathed, remote keys are routed automatically.

1. Redis Protocol (RESP) — Zero Code Changes

Priority: implement first. This gives instant compatibility with every PHP app that already uses Redis. WordPress, Laravel, Symfony, Drupal — they all have Redis session and cache drivers. No code changes, no new dependencies.

ePHPm exposes a RESP (Redis Serialization Protocol) listener that PHP’s existing Redis clients connect to as if it were a real Redis server. Two transport options:

TCP (default) — drop-in replacement, zero config changes for most apps:

[kv.redis_compat]
enabled = true
listen = "127.0.0.1:6379"   # looks like Redis to PHP

Unix socket — eliminates TCP overhead (no handshake, no Nagle, no loopback routing). ~2-5x faster than TCP for high-frequency small operations like session reads. Requires a one-line config change in the PHP app:

[kv.redis_compat]
enabled = true
socket = "/run/ephpm/redis.sock"   # unix socket path
# listen = "127.0.0.1:6379"       # TCP and socket can run simultaneously

Both can be enabled at the same time — TCP for backward compatibility and tooling (redis-cli), Unix socket for performance-sensitive paths.

Transport Comparison

Unix socket is the sweet spot for apps that want better performance without rewriting code. Just change the Redis connection string:

// Laravel — config/database.php
'redis' => [
    'default' => [
        'scheme' => 'unix',
        'path' => '/run/ephpm/redis.sock',
        // or for phpredis: 'host' => '/run/ephpm/redis.sock'
    ],
],

// WordPress — wp-config.php
define('WP_REDIS_SCHEME', 'unix');
define('WP_REDIS_PATH', '/run/ephpm/redis.sock');

// Symfony — config/packages/cache.yaml
framework:
    cache:
        default_redis_provider: 'redis:///run/ephpm/redis.sock'

Supported PHP Clients

Both major PHP Redis clients support TCP and Unix sockets out of the box:

• phpredis — C extension, best performance for RESP path. Pre-compiled into many PHP distributions. Supports Unix sockets natively via host parameter.
• predis/predis — Pure PHP, most popular Composer package. No extension required. Supports Unix sockets via scheme: unix parameter.

Framework Integration (Zero Changes)

Laravel — point the Redis connection at localhost:

// config/database.php — the only change needed
'redis' => [
    'client' => env('REDIS_CLIENT', 'phpredis'), // or 'predis'
    'default' => [
        'host' => '127.0.0.1',
        'port' => 6379,
        'database' => 0,
    ],
],

// Cache, sessions, queues — all work automatically
Cache::put('key', 'value', 3600);
// config/session.php: 'driver' => 'redis'

WordPress — with the Redis Object Cache plugin:

// wp-config.php
define('WP_REDIS_HOST', '127.0.0.1');
define('WP_REDIS_PORT', 6379);
// Plugin handles the rest — object cache, transients, etc.

Symfony — configure the cache adapter:

# config/packages/cache.yaml
framework:
    cache:
        default_redis_provider: 'redis://127.0.0.1:6379'
        pools:
            app.cache:
                adapter: cache.adapter.redis

RESP Commands to Implement

The RESP protocol is simple text-based. ~500-800 lines of Rust to implement the commands PHP clients actually use.

What PHP apps actually need: WordPress, Laravel, and Symfony Redis usage is almost entirely strings and hashes — GET/SET for cache and sessions, HGET/HSET for WordPress object cache groups, INCR/DECR for rate limiting. This covers 95%+ of real-world usage. Lists, sets, and sorted sets are niche — anyone needing those probably has a real Redis anyway.

MVP — Strings + Hashes (Phase 1)

These commands cover cache, sessions, object cache, and rate limiting:

This is enough for:

• WordPress with Redis Object Cache plugin
• Laravel cache, sessions, and rate limiting
• Symfony cache adapter
• Any app using Redis as a simple key-value cache

Phase 2 — Lists (if needed)

Lists are used primarily by Laravel queues (LPUSH/BRPOP). However, Laravel also supports database and SQS queue drivers, and we may want a native ephpm queue backed by the KV store instead of emulating Redis lists. Add lists only if there’s real demand:

Phase 3 — Sets + Sorted Sets (low priority)

Niche usage — unique tracking (SADD/SISMEMBER), leaderboards (ZADD/ZRANGE). Most apps that need these are using Redis as a primary data store, not just a cache, and should keep using a dedicated Redis instance:

Not Implementing

Unsupported commands return -ERR unknown command so clients handle them gracefully. This matches how Redis itself handles unknown commands — PHP clients and frameworks already have fallback paths.

2. Session Handler — Automatic

Priority: implement second. Register a custom PHP session.save_handler so session_start() uses the KV store with zero application changes. This uses the FFI path internally for maximum performance.

[php]
ini_overrides = [
    ["session.save_handler", "ephpm"],
    ["session.save_path", ""],           # not needed, ephpm manages storage
]

From PHP’s perspective, sessions just work:

session_start();                    // → ephpm_kv_get("session:<id>")
$_SESSION['cart'] = ['item1'];
// On request end:                  // → ephpm_kv_set("session:<id>", data, ttl: gc_maxlifetime)

The session handler implements PHP’s SessionHandlerInterface at the C level:

With clustering, sessions are accessible from any node — no sticky sessions required at the load balancer. Ownership is a deterministic hash(session_id) % alive_nodes, so within a stable cluster reads-after-writes for a session hit the same owner (read-your-writes for free). Large session blobs are additionally replicated to replication_factor nodes, so a session survives owner loss.

KV v1 caveats: session_destroy() and lazy-write TTL refresh are not replicated cluster-wide — see the clustered KV v2 roadmap and the sessions doc’s clustered-mode caveats.

3. Direct SAPI Functions — Maximum Performance

Priority: implement third. For applications that want the absolute fastest KV access — 1,000-10,000x faster than RESP for local keys. Requires ephpm-specific PHP code.

C-level functions exposed to PHP via the ePHPm SAPI:

// String operations
ephpm_kv_set("user:123:profile", $json, ttl: 3600);
$json = ephpm_kv_get("user:123:profile");
ephpm_kv_del("user:123:profile");

// Atomic operations
$count = ephpm_kv_incr("page:views");
$exists = ephpm_kv_exists("cache:key");

// With clustering, this is transparent:
// - If key is local → direct memory access, ~100ns
// - If key is on another node → internal network hop, ~0.5-2ms
// PHP code doesn't know or care which node owns the key

Composer Package (ephpm/ephpm)

Ship a Composer package that wraps the SAPI functions in standard PHP interfaces, with a fallback to Redis for local development without ephpm:

// composer require ephpm/ephpm

use Ephpm\Cache\KvStore;
use Ephpm\Cache\KvCachePool;       // PSR-6 CacheItemPoolInterface
use Ephpm\Cache\KvSimpleCache;     // PSR-16 CacheInterface

// PSR-16 SimpleCache
$cache = new KvSimpleCache();
$cache->set('key', 'value', 3600);
$value = $cache->get('key');

// PSR-6 Cache Pool
$pool = new KvCachePool();
$item = $pool->getItem('key');

// Laravel cache driver registration
// In a ServiceProvider:
Cache::extend('ephpm', function ($app, $config) {
    return new EphpmCacheStore();
});

// config/cache.php
'stores' => [
    'ephpm' => ['driver' => 'ephpm'],
],

The Composer package detects whether it’s running inside ephpm (SAPI functions available) or standalone (falls back to Redis/Memcached). This means the same application code works in development with a local Redis and in production on ephpm with zero-overhead KV.

Access Path Comparison

The RESP paths have overhead from IPC + RESP serialization, but are still 100-1,000x faster than external Redis since there’s no network hop — just in-process communication. Unix sockets cut the RESP overhead roughly in half vs TCP by eliminating the TCP/IP stack.

Performance Expectations

Clustering

Overview

Three components: gossip-based peer discovery, hash % nodes key ownership for routing, and replication for fault tolerance.

   ┌─────────────────┐   gossip    ┌─────────────────┐   gossip    ┌─────────────────┐
   │ Node A          │ ◄─────────► │ Node B          │ ◄─────────► │ Node C          │
   │ KvStore         │             │ KvStore         │             │ KvStore         │
   │ (local shard)   │             │ (local shard)   │             │ (local shard)   │
   └────────┬────────┘             └────────┬────────┘             └────────┬────────┘
            │                               │                               │
            └────────────── gossip ─────────┴───────────────────────────────┘
            │                               │                               │
            │      all three share one sorted alive-node list (gossip)      │
            ▼                               ▼                               ▼
            ┌───────────────────────────────────────────────────────────────┐
            │            key ownership: hash(key) % alive_nodes             │
            │              (key → owner + replica-set mapping)              │
            └───────────────────────────────────────────────────────────────┘

1. Gossip-Based Peer Discovery

Nodes find each other via a gossip protocol (chitchat — Quickwit’s SWIM-based gossip library for Rust, or a custom SWIM implementation).

   Node A                    Node B                    Node C
   alive · gen=5             alive · gen=3             alive · gen=7
   load=45%                  load=62%                  load=38%
       ▲                         ▲                         ▲
       │                         │                         │
       └─────── gossip ──────────┴─────── gossip ──────────┘
       │                                                   │
       └────────────────────── gossip ─────────────────────┘

Each node broadcasts:

• Its identity (address, port)
• Heartbeat generation (monotonically increasing)
• KV store metadata (memory usage, key count)
• Health status

Gossip handles node join, node leave, and failure detection automatically. No external coordination service (no etcd, no ZooKeeper, no Consul).

Configuration is minimal:

[cluster]
enabled = true
bind = "0.0.0.0:7946"
join = ["10.0.1.2:7946", "10.0.1.3:7946"]  # seed nodes

Kubernetes Service Discovery

In Kubernetes, hardcoded IPs don’t work — pods are ephemeral. Instead, use a headless Service which returns individual pod IPs via DNS (A records) rather than a single cluster IP:

apiVersion: v1
kind: Service
metadata:
  name: ephpm-headless
spec:
  clusterIP: None          # headless — returns pod IPs directly
  selector:
    app: ephpm
  ports:
    - name: gossip
      port: 7946
      protocol: UDP
    - name: data
      port: 7947
      protocol: TCP
    # Only needed with experimental CDC replication (cdc_experimental):
    # - name: cluster-channel
    #   port: 7948
    #   protocol: TCP

The gossip config uses the service DNS name instead of IPs:

[cluster]
enabled = true
join = ["ephpm-headless.default.svc.cluster.local"]

On startup, the node resolves the DNS name to multiple A records (one per pod), and treats each as a gossip seed. The resolution logic:

1. Initial join: Resolve join entries — if a value contains no :port suffix, append the default gossip port. If resolution returns multiple IPs, use all of them as seeds.
2. Periodic re-resolution: Re-resolve DNS every 30s so newly scaled pods are discovered even if all original seeds have been replaced. This handles rolling deployments where every pod eventually gets a new IP.
3. Self-filtering: Skip our own pod IP from the seed list to avoid self-connection.

The cluster secret comes from a Kubernetes Secret via env var override:

apiVersion: v1
kind: Secret
metadata:
  name: ephpm-cluster
type: Opaque
data:
  secret: <base64-encoded-32-byte-key>
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ephpm
spec:
  serviceName: ephpm-headless
  template:
    spec:
      containers:
        - name: ephpm
          env:
            - name: EPHPM_CLUSTER__SECRET
              valueFrom:
                secretKeyRef:
                  name: ephpm-cluster
                  key: secret
            - name: EPHPM_CLUSTER__ENABLED
              value: "true"
            - name: EPHPM_CLUSTER__JOIN
              value: "ephpm-headless.default.svc.cluster.local"

A StatefulSet is preferred over a Deployment for the KV store — pods get stable network identities (ephpm-0, ephpm-1, etc.) which makes debugging and log correlation easier, though gossip works with either.

2. Inter-Node Security

Cluster communication has two always-on channels plus one opt-in channel, and all of them are secured by the same symmetric pre-shared key ([cluster] secret) — there is no TLS or mTLS anywhere in the cluster transport. From the shared secret, ePHPm derives domain-separated ChaCha20-Poly1305 keys via HKDF-SHA256 (crates/ephpm-cluster/src/secure_transport.rs):

• gossip UDP: HKDF info "ephpm-gossip-v1"
• KV data plane TCP: HKDF info "ephpm-kv-data-v1"
• cluster channel handshake: HKDF info "ephpm-cluster-channel-v1"
• cluster channel session frames: HKDF info "ephpm-cluster-channel-session-v1" (salted per-session from the handshake transcript)

Because each plane derives a different key, a ciphertext valid on one plane is meaningless on the others.

Gossip Channel (UDP)

The SWIM gossip protocol uses UDP for high-frequency heartbeats and metadata exchange. Every datagram is sealed with the gossip key (nonce + Poly1305 tag). Datagrams that fail authentication are silently dropped, so a node without the matching secret — or one sending plaintext — is invisible to the cluster rather than an error. This prevents unauthorized nodes from joining and protects metadata (node addresses, health, KV stats) in transit.

Data Plane (TCP + symmetric PSK)

KV operations between nodes (remote get/set and large-value replication) use TCP. When [cluster] secret is set, each request and response is sealed as a single ChaCha20-Poly1305 frame with the data-plane key:

[frame_len: u32 BE][nonce: 12 bytes][ciphertext + 16-byte tag]

This is symmetric pre-shared-key authentication, not mutual TLS. There are no certificates, no CA, no fingerprint exchange, and no tls_cert/tls_key/tls_ca cluster options. Both sides prove possession of the same secret by being able to seal/open frames; a peer with the wrong secret cannot read values or inject writes (frames fail authentication and the connection is dropped).

Fail-closed on an empty secret. Because an unset secret means unauthenticated plaintext gossip and data plane, ePHPm refuses to start clustering (enabled = true) when secret is empty. To run without a secret on a trusted private network, opt in explicitly with allow_insecure_no_auth = true; a loud warning is logged. This closes the previous fail-open behavior where an empty secret only produced a startup warning.

Cluster Channel (TCP, opt-in)

Opt-in cluster features — today, experimental CDC-native Turso replication and its snapshot bootstrap ([db.sqlite.replication] cdc_experimental = true) — share one additional multiplexed TCP listener, the cluster channel. It binds only when a feature asks for it (default port: gossip port + 2, so 7948; override with [cluster.channel] listen) — configs without such a feature open no extra port and are byte-identical to before.

Security is one step stronger than the KV data plane: before any payload flows, both sides complete a mutual challenge/response handshake (each proves possession of the secret by sealing the other’s challenge), and every post-handshake byte is sealed under a per-session key derived from the handshake transcript — so a recorded session cannot be replayed into a new one. Inbound connections are additionally checked against live gossip membership before the handshake is attempted. Like the other planes this is symmetric PSK, not mTLS: there is no per-node identity, and any holder of the secret is a full peer. [cluster.channel] secret can override the key for this plane alone; firewall guidance: open TCP 7948 between nodes only when cdc_experimental is enabled.

The shared secret is the only security config — one value to set across all nodes, covering every plane.

Security Summary

                    ┌─────────────────────────────────┐
                    │         Cluster Secret           │
                    │   (symmetric, pre-shared key)    │
                    └──────────┬──────────────────────┘
                               │  HKDF-SHA256 (domain-separated)
               ┌───────────────┼───────────────┐
               │               │               │
               ▼               ▼               ▼
          ┌─────────┐    ┌─────────┐    ┌─────────┐
          │ Node A  │    │ Node B  │    │ Node C  │
          └────┬────┘    └────┬────┘    └────┬────┘
               │              │              │
  Gossip (UDP) │◄─sealed─────►│◄─sealed─────►│
  ChaCha20     │  (gossip key)│              │
               │              │              │
  Data (TCP)   │◄─sealed─────►│◄─sealed─────►│
  ChaCha20     │  (data key)  │              │
  (PSK, no TLS)│              │              │

3. Key Ownership (hash % nodes)

Large-tier keys are distributed across nodes by a plain modulo of the sorted alive-node list — there is no consistent hash ring, no virtual nodes, and no hashring crate dependency (see crates/ephpm-cluster/src/clustered_store.rs):

owner = alive_nodes_sorted[ hash(key) % alive_nodes_sorted.len() ]

            ┌───────────────────────────┐
            │  sorted alive nodes:       │
            │  [ Node A, Node B, Node C ]│
            │                            │
            │  Key "session:abc"         │
            │  hash % 3 = 0 → Node A     │
            └───────────────────────────┘

The trade-off vs a consistent hash ring: when membership changes (a node joins, leaves, or dies), ownership remaps broadly — most keys change owner, not the ~1/N a ring would give. For ePHPm’s cache-oriented workload (values that can be refetched or recomputed) this is acceptable; a consistent-hashing scheme may replace it later if remapping cost becomes a problem.

struct ClusterRouter {
    alive_nodes: RwLock<Vec<NodeId>>,   // sorted; from gossip membership
    local_node: NodeId,
}

impl ClusterRouter {
    /// Primary owner of a key.
    fn owner(&self, key: &[u8]) -> NodeId {
        let nodes = self.alive_nodes.read();
        nodes[(hash(key) as usize) % nodes.len()].clone()
    }
}

Crate dependencies:

(Key ownership is a few lines of arithmetic over the gossip membership list — no external hashing crate.)

4. Replication

Large-tier writes go to a replica set: the primary owner plus the next replication_factor - 1 distinct nodes on the sorted alive-node ring (wrapping around, clamped to the cluster size). This provides fault tolerance — if the owner dies, a replica can serve reads. Small (gossip-tier) values are separately replicated to every node and ignore replication_factor.

Write large "cart:abc" = <8 KiB>, replication_factor = 2
       │
       ▼
   hash(key) % nodes → Node A (primary owner)
       │
       ├──► write to Node A (primary)
       └──► replicate to Node B (next node on the ring)

Read "cart:abc":  try Node A first → on miss/owner-down, fall back to Node B

Replication modes ([cluster.kv] replication_mode):

• async (default) — the client write returns after the primary/local copy is written; the remaining replicas are updated in the background (fire-and-forget, failures logged and counted). Lower latency, eventual replica convergence.
• sync — the write also awaits every reachable replica before returning. This is best-effort read-your-writes durability against live peers: a replica that is down is logged but does not fail the write. It is not a quorum or consensus protocol (no Raft, no rollback).

Read path: reads try the primary owner first, then fall back to the other replicas in ring order until one returns the value. A large value therefore survives up to replication_factor - 1 node failures.

Membership caveat (v1): replication is write-time only. There is no active anti-entropy or rebalancing — a node that was down during a write does not receive the key until it is rewritten or fetched-through, and existing keys are not re-replicated when membership changes.

For sessions and cache data, async replication is the right default. PHP apps doing $_SESSION['cart'] = ... on one request and reading it on the next hit the same owner (ownership is a deterministic function of the session id and the current membership), so within a stable cluster they get read-your-writes for free.

[cluster.kv]
replication_factor = 2        # copies of each large value (owner + N-1 peers)
replication_mode = "async"    # async (default) or sync

5. Node Join / Leave / Failure

Because ownership is hash % alive_nodes and replication is write-time only (no active rebalancing), membership changes remap ownership but do not move existing data:

Features Built on the KV Store

The KV layer is a foundation for several higher-level features. Each section below depends on the single-node KV store existing first, and the clustering sections depend on gossip being operational.

TLS Certificate Management (Clustered)

Single-node ACME is already implemented using filesystem-based DirCache. The clustered KV store replaces it with KvCache, enabling zero-config HTTPS across multiple nodes. See http.md for single-node ACME details.

Problem 1: Cert Issuance Race Condition

Without coordination, multiple nodes detect no cert exists and all request one from Let’s Encrypt simultaneously. This wastes ACME quota (50 certs/domain/week) and may trigger rate-limiting bans.

Solution: Distributed lock via KV store.

Node A: KvStore.lock("acme:lock:example.com", node_id="A", ttl=300s)
  → Lock acquired. Node A proceeds with ACME flow.

Node B: KvStore.lock("acme:lock:example.com", node_id="B", ttl=300s)
  → Lock held by Node A. Node B waits.

Node A completes issuance:
  KvStore.set("certs:example.com:cert", cert_pem)
  KvStore.set("certs:example.com:key", key_pem)
  KvStore.unlock("acme:lock:example.com")
       │
       ├──► replicated to Node B (gossip)
       └──► replicated to Node C (gossip)

Node B: lock released, checks KvStore → cert exists → done.

Lock has a TTL (default 5 minutes) to prevent deadlock if a node crashes mid-issuance.

Problem 2: ACME Challenge Routing

In a multi-node setup behind a load balancer, Let’s Encrypt’s challenge request may hit any node — not the one that initiated the ACME order.

Solution: Challenge token propagation via KV store.

Node A initiates ACME order for example.com
       │
       ▼
   LE returns challenge: token=abc123, response=xyz789
       │
       ▼
   KvStore.set("acme:challenge:abc123", "xyz789", ttl=600s)
       │
       ├──► replicated to Node B (gossip, ~100ms)
       └──► replicated to Node C (gossip, ~100ms)

LE requests: GET http://example.com/.well-known/acme-challenge/abc123
       │
       ▼ (load balancer routes to Node B)
   Node B: KvStore.get("acme:challenge:abc123") → "xyz789"
   Node B: responds 200 OK → challenge passed ✓

Works for both HTTP-01 and TLS-ALPN-01 challenges. Gossip replication (~100-200ms) completes well within the ACME protocol’s built-in polling delay.

Problem 3: Renewal Stampede

All nodes notice the cert expires soon. Without coordination, all attempt renewal simultaneously.

Solution: Leader election for cert renewal.

KvStore.set("acme:leader", node_id="A", ttl=60s)
  → Node A is the cert renewal leader
  → Node A refreshes TTL every 30s (heartbeat)
  → Node A checks all certs, renews any expiring within 30 days

If Node A dies:
  → TTL expires after 60s
  → Node B or C acquires leadership
  → New leader picks up renewal duties

Only the leader initiates renewals. All other nodes receive renewed certs via KV replication.

Full Cert Lifecycle

1. First HTTPS request arrives for example.com
2. Node checks KvStore for "certs:example.com:cert"
   ├── Found → use it, serve request with TLS
   └── Not found ↓
3. Acquire lock: KvStore.lock("acme:lock:example.com")
   ├── Lock held by another node → wait, then goto 2
   └── Lock acquired ↓
4. Create ACME order (Let's Encrypt)
5. Store challenge token in KV (replicated to all nodes)
6. LE verifies challenge → any node can respond → passes
7. Download issued cert
8. Store cert + key in KV (replicated to all nodes)
9. Release lock
10. All nodes now have the cert locally → zero-latency TLS handshakes

No external cert store, lock service, or coordination service needed.

The rustls-acme crate has a pluggable Cache trait — swap DirCache for a KvCache implementation. Zero changes to the ACME logic itself.

Phase 1 (single-node, implemented):  AcmeConfig → DirCache (filesystem)
Phase 2 (clustered):                 AcmeConfig → KvCache (gossip-replicated KV store)

PHP Response Cache (Clustered)

Intercept PHP-generated ETag headers and short-circuit repeat requests across all nodes without executing PHP. This is distinct from static file ETags (which are handled at the server level today).

Flow

1. First request: /blog/hello
   → PHP executes, returns response with ETag: "abc123"
   → Server stores in KV: cache:<url_key> → { etag, headers, body }
   → Response sent to client

2. Repeat request: /blog/hello + If-None-Match: "abc123"
   → Server checks KV for cache:<url_key>
   → ETag matches → return 304 Not Modified immediately
   → No PHP execution, no mutex contention

3. Works across all nodes via gossip replication

Design Decisions

Impact

Most WordPress page views are anonymous and return identical content. Skipping PHP entirely for repeat visitors eliminates the single-threaded PHP mutex bottleneck and lets the async HTTP server handle cached responses at full throughput across all nodes.

Session Storage

PHP sessions stored in the KV store instead of the filesystem. With clustering, sessions are accessible from any node — no sticky sessions required at the load balancer (but note the KV v1 caveats: session_destroy() and lazy-write TTL refresh are not replicated — see the clustered KV v2 roadmap).

How PHP Sessions Work

PHP’s session system is callback-driven. The engine calls a set of handler functions at specific points in the session lifecycle. By default, PHP uses the files handler (/tmp/sess_<id>). Redis and Memcached replace this with their own handlers. We do the same, implementing the callbacks in C (via ephpm_wrapper.c) backed by Rust’s KV store.

Request Lifecycle

PHP: session_start()
       │
       ▼
  1. open(save_path, session_name)
       → No-op. KV store is always available, no connection to establish.
       │
       ▼
  2. read(session_id)
       → C: ephpm_session_read(id)
       → Rust: kv_store.get("session:{id}")
       → Returns serialized $_SESSION data (or "" for new session)
       │
       ▼
  3. PHP deserializes data into $_SESSION superglobal
       │
       ▼
  4. Script runs, modifies $_SESSION freely
       │
       ▼
  5. Request ends (or explicit session_write_close())
       │
       ▼
  6. write(session_id, serialized_data)
       → C: ephpm_session_write(id, data, gc_maxlifetime)
       → Rust: kv_store.set("session:{id}", data, ttl: gc_maxlifetime)
       │
       ▼
  7. close()
       → Release session lock (see Locking below)

C Handler Implementation

The session handler is registered during SAPI initialization via php_session_register_module(). All functions go through ephpm_wrapper.c with zend_try/zend_catch bailout protection.

TODO verify / does not match the shipped code: the C listing below is the original design sketch. The actual implementation in crates/ephpm-php/ephpm_wrapper.c does not use the ephpm_kv_session_lock(const char *id, size_t id_len) / ephpm_kv_session_unlock(...) FFI signatures shown here. Locking is done inside the wrapper via a KV op table (g_kv_ops.set_nx / g_kv_ops.del) and internal static helpers ephpm_session_lock_acquire(sid, sid_len) / ephpm_session_lock_release(void), with the acquired sid tracked in thread-local state (session_lock:<sid>, 30s TTL, spin-with-backoff). See the Sessions feature doc for the authoritative behaviour.

// ephpm_wrapper.c

#include "ext/session/php_session.h"

// Forward declarations — these call into Rust via FFI
extern int ephpm_kv_session_read(const char *id, size_t id_len,
                                  char **data, size_t *data_len);
extern int ephpm_kv_session_write(const char *id, size_t id_len,
                                   const char *data, size_t data_len,
                                   int gc_maxlifetime);
extern int ephpm_kv_session_destroy(const char *id, size_t id_len);
extern int ephpm_kv_session_lock(const char *id, size_t id_len);
extern int ephpm_kv_session_unlock(const char *id, size_t id_len);

PS_OPEN_FUNC(ephpm) {
    // No-op — KV store is always available in-process
    return SUCCESS;
}

PS_CLOSE_FUNC(ephpm) {
    // Release session lock
    ephpm_kv_session_unlock(/* current session id */);
    return SUCCESS;
}

PS_READ_FUNC(ephpm) {
    // Acquire session lock first (blocking)
    int rc = ephpm_kv_session_lock(key->val, key->len);
    if (rc != SUCCESS) {
        php_error_docref(NULL, E_WARNING,
            "ephpm: session lock timeout for %s, proceeding without lock",
            key->val);
    }

    char *data = NULL;
    size_t data_len = 0;
    rc = ephpm_kv_session_read(key->val, key->len, &data, &data_len);
    if (rc == SUCCESS && data != NULL) {
        *val = zend_string_init(data, data_len, 0);
        free(data);
    } else {
        *val = zend_string_init("", 0, 0);  // new session
    }
    return SUCCESS;
}

PS_WRITE_FUNC(ephpm) {
    int maxlifetime = INI_INT("session.gc_maxlifetime");
    return ephpm_kv_session_write(
        key->val, key->len,
        val->val, val->len,
        maxlifetime
    );
}

PS_DESTROY_FUNC(ephpm) {
    return ephpm_kv_session_destroy(key->val, key->len);
}

PS_GC_FUNC(ephpm) {
    // No-op — TTL-based expiry handles garbage collection automatically.
    // PHP calls this probabilistically (session.gc_probability / session.gc_divisor),
    // but we don't need it since every session key has a TTL.
    *nrdels = 0;
    return SUCCESS;
}

// Register the handler module
ps_module ps_mod_ephpm = {
    PS_MOD_UPDATE_TIMESTAMP(ephpm)
};

void ephpm_register_session_handler(void) {
    php_session_register_module(&ps_mod_ephpm);
}

Rust Side

// In ephpm-php/src/lib.rs (or a new session.rs module)

/// Called from C: read session data from KV store.
#[no_mangle]
pub extern "C" fn ephpm_kv_session_read(
    id: *const c_char, id_len: usize,
    data: *mut *mut c_char, data_len: *mut usize,
) -> c_int {
    let id = unsafe { std::slice::from_raw_parts(id as *const u8, id_len) };
    let key = format!("session:{}", std::str::from_utf8(id).unwrap_or(""));

    let store = KV_STORE.get().expect("KV store not initialized");
    match store.get(key.as_bytes()) {
        Some(value) => {
            let ptr = unsafe { libc::malloc(value.len()) as *mut c_char };
            unsafe {
                std::ptr::copy_nonoverlapping(
                    value.as_ptr(), ptr as *mut u8, value.len()
                );
                *data = ptr;
                *data_len = value.len();
            }
            0 // SUCCESS
        }
        None => {
            unsafe {
                *data = std::ptr::null_mut();
                *data_len = 0;
            }
            0 // SUCCESS (empty session is not an error)
        }
    }
}

/// Called from C: write session data to KV store with TTL.
#[no_mangle]
pub extern "C" fn ephpm_kv_session_write(
    id: *const c_char, id_len: usize,
    data: *const c_char, data_len: usize,
    gc_maxlifetime: c_int,
) -> c_int {
    let id = unsafe { std::slice::from_raw_parts(id as *const u8, id_len) };
    let data = unsafe { std::slice::from_raw_parts(data as *const u8, data_len) };
    let key = format!("session:{}", std::str::from_utf8(id).unwrap_or(""));
    let ttl = Duration::from_secs(gc_maxlifetime as u64);

    let store = KV_STORE.get().expect("KV store not initialized");
    store.set(key.into_bytes(), data.to_vec(), Some(ttl));
    0 // SUCCESS
}

Session Locking

This is the most important design decision. Without locking, concurrent requests sharing a session ID can clobber each other:

Request A: session_start() → reads $_SESSION = {cart: [item1]}
Request B: session_start() → reads $_SESSION = {cart: [item1]}
Request A: $_SESSION['cart'][] = 'item2'; session_write_close()
           → writes {cart: [item1, item2]}
Request B: $_SESSION['cart'][] = 'item3'; session_write_close()
           → writes {cart: [item1, item3]}  ← item2 is LOST

Current model (ZTS, concurrent PHP requests): This is a real concern. Multiple threads can execute PHP concurrently, so concurrent session access from the same user (e.g. AJAX requests) can cause lost updates.

Future model (external PHP workers, multi-process): Locking is essential.

Implementation: Lock acquired during read(), released during close(). The lock is a KV key with TTL to prevent deadlocks:

KV key: "session_lock:{session_id}"
Value:  "{request_id}"
TTL:    30s (safety net if request crashes without releasing)

Lock acquisition strategy:

• Spin with backoff: Try to acquire lock, sleep 10ms, retry up to session.lock_wait_timeout (default 10s). Same approach as the Redis session handler.
• If timeout exceeded: Proceed without lock (same behavior as phpredis redis.session.lock_retries exhausted). Log a warning.

/// Acquire session lock with spin + backoff.
fn session_lock(session_id: &str, timeout: Duration) -> bool {
    let lock_key = format!("session_lock:{session_id}");
    let lock_ttl = Duration::from_secs(30);
    let deadline = Instant::now() + timeout;
    let mut backoff = Duration::from_millis(10);

    loop {
        // SET NX — only succeeds if key doesn't exist
        if store.set_nx(lock_key.as_bytes(), request_id, Some(lock_ttl)) {
            return true;  // lock acquired
        }
        if Instant::now() >= deadline {
            return false;  // timeout
        }
        std::thread::sleep(backoff);
        backoff = (backoff * 2).min(Duration::from_millis(200));
    }
}

With clustering, the lock key session_lock:{id} is small and rides the gossip tier (it is well under small_key_threshold), so it is replicated to every node — the SETNX and DEL operate on the local gossip-backed view. (In the shipped implementation the lock is node-local; see the Sessions feature doc for the exact cross-node semantics.)

Serialization

PHP has a built-in session serializer (session.serialize_handler). By default it uses php format, but php_serialize (standard serialize()) is also common. We don’t touch serialization at all — PHP handles it before calling our write() and after our read(). The KV store just sees opaque bytes.

Configuration

[php]
ini_overrides = [
    # Enable ephpm session handler (set automatically when [kv] is configured)
    ["session.save_handler", "ephpm"],

    # Standard PHP session settings still apply
    ["session.gc_maxlifetime", "1440"],   # TTL in seconds (default 24 min)
    ["session.cookie_lifetime", "0"],      # browser session cookie
    ["session.cookie_secure", "1"],        # HTTPS only
    ["session.cookie_httponly", "1"],       # no JS access
    ["session.cookie_samesite", "Lax"],    # CSRF protection
]

When [kv] is configured in ephpm.toml, the session handler is registered automatically. If the user hasn’t explicitly set session.save_handler, ephpm defaults to the KV handler. If they’ve set it to files or redis explicitly, we respect that.

Clustering Behavior

With clustering, sessions work transparently across nodes:

Client → Load Balancer → Pod 0: session_start()
  │                              → kv_store.get("session:abc123")
  │                              → key hashes to Pod 2 (owner)
  │                              → internal network hop to Pod 2
  │                              → Pod 2 returns session data
  │                              → PHP deserializes, script runs
  │                              → session_write_close()
  │                              → kv_store.set("session:abc123", data)
  │                              → routed to Pod 2 (owner)
  │                              → Pod 2 writes locally + replicates
  │
  ├── Next request → Pod 1: session_start()
  │                          → same flow, key routes to Pod 2
  │                          → session data available immediately

No sticky sessions needed at the load balancer. Any node can read/write any session. Ownership (hash(session_id) % alive_nodes) keeps a session on the same owner while membership is stable, so there’s no replication-lag concern for read-your-writes within a single session.

If the owner node (Pod 2) crashes, a surviving node recomputes ownership and reads the session from a replica (large blobs are replicated to replication_factor nodes; small blobs live on every node via gossip). The session survives up to replication_factor - 1 simultaneous owner-side failures. Note the write-time-only caveat: there is no background re-replication, so after a crash the surviving copies are not automatically topped back up to replication_factor until the session is rewritten.

Performance vs Other Session Handlers

For the common case (session key local to the node), ephpm is 1,000-10,000x faster than Redis or Memcached. For the clustered case (key on another node), it’s comparable — but with zero external infrastructure.

Distributed Locking

General-purpose distributed locks for PHP applications, built on the atomic ephpm_kv_setnx() primitive plus a TTL:

// setnx() returns true only if the key did not already exist.
if (ephpm_kv_setnx("deploy:mutex", getmypid())) {
    ephpm_kv_expire("deploy:mutex", 30);  // never hold it forever
    try {
        // critical section
    } finally {
        ephpm_kv_del("deploy:mutex");
    }
}

ePHPm uses the same KV + TTL pattern internally to elect a single ACME certificate requester across the cluster, but that coordination lives in Rust (crates/ephpm-server/src/acme.rs) — it is not exposed as a PHP lock API.

Rate Limiting

Per-IP or per-key rate limiting using KV counters with TTL:

Request from 1.2.3.4:
  key = "ratelimit:1.2.3.4"
  count = KvStore.incr(key)
  if count == 1: KvStore.expire(key, window_seconds)
  if count > limit: return 429

With clustering, rate limits are cluster-wide — an attacker can’t bypass limits by hitting different nodes.

Node API Endpoints

The Node API exposes KV and cluster state for monitoring and debugging:

Implementation Order

The KV store and clustering should be built incrementally:

Phases 1-5 (single-node) and Phase 6 (gossip) can be developed in parallel. Everything after Phase 7 depends on key ownership being operational. RESP data structure phases (13-14) can be added at any time after Phase 2.

Configuration Reference

Planned

[kv]
memory_limit = "256MB"                 # max memory for KV data
eviction_policy = "allkeys-lru"        # noeviction, allkeys-lru, volatile-lru, allkeys-random

[kv.redis_compat]
enabled = false                        # RESP protocol listener
listen = "127.0.0.1:6379"             # TCP listener (default, zero-config for existing apps)
socket = "/run/ephpm/redis.sock"       # Unix socket (~2-5x faster than TCP)
# Both can be enabled simultaneously — TCP for tooling, socket for performance

[cluster]
enabled = false
bind = "0.0.0.0:7946"                 # gossip listen address
join = ["10.0.1.2:7946"]              # seed nodes
secret = "REPLACE_ME"                  # shared secret -- a ChaCha20-Poly1305 PSK that
                                       # seals BOTH gossip UDP and the KV data plane TCP
                                       # (no TLS/mTLS; there are no tls_cert/tls_key/tls_ca
                                       # cluster options). REQUIRED when enabled = true:
                                       # an empty secret is a hard startup error unless
                                       # allow_insecure_no_auth = true (not recommended).
# allow_insecure_no_auth = false       # opt in to empty-secret plaintext clustering

[cluster.kv]
replication_factor = 2                 # copies of each large value (owner + N-1 peers); clamped to cluster size
replication_mode = "async"             # async (default) or sync — see Replication above