Skip to content
Strategy

Testing Architecture

This document covers the end-to-end testing strategy, from single-node validation through multi-node cluster testing and high availability verification.


Current State

Test Layers

LayerToolSpeedWhat it covers
Unitcargo nextestSecondsConfig parsing, routing logic, static file serving, path traversal, glob matching
Integrationcargo nextest (ignored without libphp)SecondsPHP FFI calls, request/response mapping, superglobal population
E2Eephpm-e2e crate in Kind clusterMinutesFull HTTP lifecycle: PHP execution, static serving, version/SAPI validation
Benchmarkscargo benchMinutesThroughput measurement

Infrastructure

  • Kind cluster (ephpm-dev) with single control-plane node
  • Tilt orchestration for image build, deploy, and test execution
  • GitHub Actions CI matrix: PHP 8.4 + 8.5 x Linux + macOS
  • xtask commands: e2e-install, e2e, e2e-up, e2e-down

Current E2E Coverage

The ephpm-e2e crate covers the following areas. Each row is one test file.

FileAreaTests
basic.rsCore lifecycle404, PHP render, static file
phpinfo.rsPHP version + SAPIversion string, health check
http.rsHTTP protocolHEAD, POST, Content-Type, ETag/304, gzip, 413, Cache-Control, X-Forwarded-For, fallback chain
php.rsPHP execution$_GET, $_SERVER, exit(0), custom status, $_COOKIE, php://input, header()
errors.rsPHP error recoveryfatal error → 500, OOM → 500, syntax error → 500, server survives each
kv.rsKV store PHP bridgeset/get, TTL, incr, del/exists, pttl, incr_by, expire, setnx, mset/mget
concurrency.rsConcurrent load20 parallel GETs, 20 concurrent KV increments
security.rsAccess controldotfile 403, PHP source not exposed, blocked_paths glob, path traversal

Remaining Single-Node Gaps

Planned but not yet implemented (see detailed tables in the sections below):

  • PHP: $_FILES upload (multipart parsing), session_start() / $_SESSION, output buffering, SCRIPT_NAME/SCRIPT_FILENAME after fallback rewrite
  • HTTP: If-Modified-Since / Last-Modified, large file streaming, PHP-initiated redirect (header("Location: ..."))
  • Security: allowed_php_paths whitelist enforcement, trusted_hosts 421 response
  • Config: X-Forwarded-ForREMOTE_ADDR rewrite when trusted_proxies is set, $_SERVER['HTTPS'] flag

Multi-node cluster and HA tests remain out of scope until clustering is implemented (see below).


Single-Node Test Plan

These tests run against a single ephpm instance in Kind. They should all be implemented in the ephpm-e2e crate as async Rust tests using reqwest.

PHP Execution

TestMethodValidates
php_hello_worldGET /index.php200, response body contains greeting
php_versionGET /index.phpResponse contains expected PHP version
php_sapi_nameGET /index.phpSAPI is embed
phpinfo_rendersGET /info.php200, body contains <html, phpinfo() output
php_get_paramsGET /test.php?foo=bar&baz=123$_GET['foo'] == 'bar', $_GET['baz'] == '123'
php_post_formPOST /test.php (form-urlencoded)$_POST contains submitted values
php_post_jsonPOST /test.php (application/json)php://input contains raw JSON body
php_post_multipartPOST /test.php (multipart/form-data)$_FILES populated, $_POST fields present
php_cookiesGET /test.php with Cookie: foo=bar$_COOKIE['foo'] == 'bar'
php_request_uri_preservedGET /some/path?q=1 (via fallback to index.php)$_SERVER['REQUEST_URI'] == '/some/path?q=1'
php_script_name_after_rewriteGET /blog/hello (fallback rewrite)$_SERVER['SCRIPT_NAME'] == '/index.php'
php_content_type_headerGET /test.phpContent-Type set by PHP script
php_custom_status_codeGET (script calls http_response_code(404))Response status is 404
php_custom_headersGET (script calls header('X-Custom: value'))Response has X-Custom: value
php_large_outputGET (script outputs >1MB)Full body received, Content-Length correct
php_exit_with_outputGET (script calls echo 'hello'; exit;)200, body is hello
php_error_handlingGET (script triggers E_WARNING)Server doesn’t crash, response returned

Static File Serving

TestMethodValidates
static_htmlGET /test.html200, Content-Type: text/html, body matches file
static_cssGET /style.css200, Content-Type: text/css
static_jsGET /app.js200, Content-Type: application/javascript
static_imageGET /image.png200, Content-Type: image/png, binary body matches
static_content_lengthGET /test.htmlContent-Length header matches file size
static_unknown_extensionGET /data.xyz200, Content-Type: application/octet-stream
static_missing_fileGET /nonexistent.txt404
static_nested_pathGET /subdir/file.html200, correct body

ETag and Caching

TestMethodValidates
etag_presentGET /test.htmlResponse has ETag header (weak format W/"...")
etag_304_on_matchGET /test.html with If-None-Match: <etag>304 Not Modified, empty body
etag_200_on_mismatchGET /test.html with If-None-Match: "wrong"200, full body
etag_star_matchesGET /test.html with If-None-Match: *304
etag_comma_listGET /test.html with If-None-Match: "a", <real>, "b"304
etag_consistentGET /test.html twiceSame ETag both times
cache_control_headerGET /test.html (with cache_control configured)Cache-Control header present

Compression

TestMethodValidates
gzip_html_responseGET /test.html with Accept-Encoding: gzipContent-Encoding: gzip, body decompresses to original
gzip_php_responseGET /info.php with Accept-Encoding: gzipContent-Encoding: gzip on large phpinfo output
no_gzip_without_headerGET /test.html (no Accept-Encoding)No Content-Encoding header
no_gzip_small_bodyGET small file with Accept-Encoding: gzipNo compression (below min size)
no_gzip_imageGET /image.png with Accept-Encoding: gzipNo compression (non-compressible type)
vary_header_presentGET with Accept-Encoding: gzipVary: Accept-Encoding header

Security

TestMethodValidates
dotfile_blockedGET /.env403 Forbidden
dotdir_blockedGET /.git/config403 Forbidden
htaccess_blockedGET /.htaccess403 Forbidden
path_traversal_blockedGET /../../../etc/passwd403 or 404
blocked_path_exactGET /wp-config.php (when in blocked_paths)403
blocked_path_wildcardGET /vendor/autoload.php (when /vendor/* blocked)403
php_allowlist_blocksGET /uploads/shell.php (when allowed_php_paths set)403
php_allowlist_allowsGET /index.php (in allowed_php_paths)200
body_size_limitPOST with body exceeding max_body_size413 Payload Too Large
trusted_host_validGET with Host: allowed.example.com200
trusted_host_invalidGET with Host: evil.example.com421 Misdirected Request
trusted_host_with_portGET with Host: allowed.example.com:8080200 (port stripped for comparison)

Custom Response Headers

TestMethodValidates
custom_header_staticGET /test.htmlConfigured custom headers present
custom_header_phpGET /index.phpConfigured custom headers present on PHP responses
hsts_headerGET any pageStrict-Transport-Security header if configured
cors_headersGET any pageAccess-Control-Allow-Origin etc. if configured

Fallback / URL Resolution

TestMethodValidates
uri_literal_fileGET /test.htmlServes static file directly
uri_directory_indexGET /Resolves to /index.php via index_files
uri_subdirectory_indexGET /subdir/Resolves to /subdir/index.html
fallback_to_index_phpGET /nonexistent/pathFalls through to /index.php
fallback_preserves_queryGET /path?key=valFallback to /index.php?key=val
fallback_404_configGET /missing (with =404 fallback)404 Not Found

Trusted Proxies

TestMethodValidates
xff_trusted_proxyGET with X-Forwarded-For from trusted IP$_SERVER['REMOTE_ADDR'] is the client IP from XFF
xff_untrusted_proxyGET with X-Forwarded-For from untrusted IP$_SERVER['REMOTE_ADDR'] is the connecting IP (XFF ignored)
xfp_https_detectionGET with X-Forwarded-Proto: https from trusted proxy$_SERVER['HTTPS'] == 'on'

TLS (Manual Certs)

TestMethodValidates
tls_serves_httpsHTTPS GET /index.php200, valid TLS handshake
tls_redirect_httpHTTP GET (with redirect_http = true)301 redirect to HTTPS
tls_server_varHTTPS GET /test.php$_SERVER['HTTPS'] == 'on', $_SERVER['SERVER_PORT'] == '443'
tls_invalid_cert_rejectedHTTPS GET with strict clientHandshake fails if cert doesn’t match

Timeouts and Limits

TestMethodValidates
request_timeoutGET (PHP script sleeps beyond server.timeouts.request)Connection closed or 504
idle_timeoutOpen connection, send nothing for > idle timeoutConnection closed
max_header_sizeSend request with oversized headers431 or connection closed

Graceful Shutdown

TestMethodValidates
inflight_request_completesStart slow PHP request, send SIGTERMResponse received before shutdown
new_requests_rejectedSend SIGTERM, then new requestConnection refused or 503
readiness_probe_failsSend SIGTERMKubernetes readiness probe fails, pod removed from service

Single-Node Test Infrastructure

Test Config Variants

Different features need different ephpm.toml configurations. Use Kubernetes ConfigMaps to inject test-specific configs:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ephpm-security-test-config
data:
  ephpm.toml: |
    [server]
    listen = "0.0.0.0:8080"
    document_root = "/var/www/html"

    [server.security]
    blocked_paths = ["/wp-config.php", "/vendor/*"]
    allowed_php_paths = ["/index.php", "/test.php", "/info.php"]
    trusted_proxies = ["10.0.0.0/8"]

    [server.request]
    max_body_size = 1024
    trusted_hosts = ["ephpm", "ephpm.default.svc.cluster.local"]

    [server.response]
    headers = [
        ["X-Frame-Options", "DENY"],
        ["Strict-Transport-Security", "max-age=63072000"],
    ]

Test Docroot Fixtures

Expand the test docroot with purpose-built PHP scripts:

tests/docroot/
  index.php          # greeting + version + SAPI (existing)
  info.php           # phpinfo() (existing)
  test.php           # server vars dump (existing)
  test.html          # static file (existing)
  style.css          # CSS MIME type test
  app.js             # JS MIME type test
  image.png          # binary static file test
  large_output.php   # outputs >1MB for compression/body tests
  custom_status.php  # http_response_code(404)
  custom_headers.php # header('X-Custom: value')
  exit_test.php      # echo 'hello'; exit;
  sleep.php          # sleep($seconds) for timeout tests
  error_test.php     # triggers E_WARNING
  post_echo.php      # echoes $_POST, $_FILES, php://input
  cookie_echo.php    # echoes $_COOKIE
  server_vars.php    # JSON dump of $_SERVER for precise assertions
  subdir/
    index.html       # directory index test

E2E Crate Structure

Organize tests by feature area using Rust test modules:

crates/ephpm-e2e/
  src/
    lib.rs           # shared helpers (HTTP client, assertions, env vars)
  tests/
    php_execution.rs  # PHP lifecycle tests
    static_files.rs   # static serving + MIME types
    etag.rs           # ETag + 304 tests
    compression.rs    # gzip tests
    security.rs       # dotfiles, blocked paths, allowlist, body limits
    fallback.rs       # URL resolution / try_files
    headers.rs        # custom response headers, trusted hosts
    proxy.rs          # X-Forwarded-For, X-Forwarded-Proto
    tls.rs            # HTTPS, redirects, $_SERVER['HTTPS']
    timeouts.rs       # request timeout, idle timeout, header size
    shutdown.rs       # graceful shutdown behavior

Each test file reads EPHPM_URL from the environment and issues HTTP requests. Tests that need specific config (security, TLS) use separate ephpm deployments with their own ConfigMaps.

Tilt Orchestration

Extend the Tiltfile to manage multiple ephpm deployments with different configs:

# Default ephpm instance (standard config)
k8s_yaml('k8s/base/ephpm-single.yaml')

# Security-focused instance (blocked paths, allowlist, trusted hosts)
k8s_yaml('k8s/tests/ephpm-security.yaml')

# TLS instance (manual certs)
k8s_yaml('k8s/tests/ephpm-tls.yaml')

# Timeout instance (short timeouts for testing)
k8s_yaml('k8s/tests/ephpm-timeouts.yaml')

E2E tests target the appropriate instance via different EPHPM_URL env vars.


Multi-Node Cluster Test Plan

These tests validate the KV store, gossip protocol, and clustering features. They require multiple ephpm instances running simultaneously.

Infrastructure

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ephpm-cluster
spec:
  serviceName: ephpm-headless
  replicas: 3
  template:
    spec:
      containers:
        - name: ephpm
          env:
            - name: EPHPM_CLUSTER__ENABLED
              value: "true"
            - name: EPHPM_CLUSTER__JOIN
              value: "ephpm-headless.default.svc.cluster.local"
            - name: EPHPM_CLUSTER__SECRET
              valueFrom:
                secretKeyRef:
                  name: ephpm-cluster
                  key: secret
---
apiVersion: v1
kind: Service
metadata:
  name: ephpm-headless
spec:
  clusterIP: None
  selector:
    app: ephpm-cluster
  ports:
    - name: http
      port: 8080
    - name: gossip
      port: 7946
      protocol: UDP
    - name: data
      port: 7947
---
# Load-balanced service for client-facing requests
apiVersion: v1
kind: Service
metadata:
  name: ephpm-cluster-lb
spec:
  selector:
    app: ephpm-cluster
  ports:
    - name: http
      port: 8080

Cluster Formation

TestMethodValidates
cluster_formsQuery Node API on all 3 podsAll pods see 3 members
cluster_gossip_metadataQuery /api/kv/cluster on each podEach pod reports memory usage, key count, health for all peers
cluster_node_identityQuery each podEach reports a unique node ID
cluster_join_timeMeasure time from pod ready to cluster membershipJoins within 5 seconds

KV Store — Single Node

TestMethodValidates
kv_set_getPHP: ephpm_kv_set("key", "val") then ephpm_kv_get("key")Value returned correctly
kv_delSet, delete, getReturns null after delete
kv_ttl_expirySet with TTL=2s, wait 3s, getReturns null (expired)
kv_ttl_not_expiredSet with TTL=60s, get immediatelyReturns value
kv_overwriteSet key twice with different valuesSecond value returned
kv_hash_operationsHSET, HGET, HGETALL, HDELAll hash operations work
kv_incr_decrSET to “10”, INCR, DECRCorrect arithmetic
kv_large_valueSet 1MB value, getFull value returned
kv_binary_valueSet value with null bytes, getBinary-safe storage

KV Store — RESP Protocol

TestMethodValidates
resp_pingRedis client PINGReturns PONG
resp_set_getRedis client SET/GETRound-trip works
resp_delRedis client DELKey removed
resp_expire_ttlRedis client EXPIRE/TTLTTL counts down
resp_mget_msetRedis client MGET/MSETBatch operations work
resp_incrRedis client INCR/DECRAtomic counters
resp_hash_opsRedis client HSET/HGET/HGETALLHash operations via RESP
resp_unknown_commandRedis client sends unsupported commandReturns ERR, doesn’t crash
resp_predis_compatLaravel app using predis/predisCache and session drivers work
resp_phpredis_compatLaravel app using phpredis extensionCache and session drivers work

KV Store — Cross-Node (Clustered)

TestMethodValidates
kv_write_read_same_nodeSet on pod-0, get on pod-0Fast path works
kv_write_read_different_nodeSet on pod-0, get on pod-1Cross-node routing works
kv_hash_ring_routingSet many keys, check distribution via Node APIKeys distributed across all nodes (not all on one)
kv_replication_existsSet on pod-0, check replica count via Node APIKey replicated to N additional nodes
kv_ttl_cross_nodeSet with TTL on pod-0, wait, get on pod-1TTL respected across nodes
kv_large_keyspaceSet 10,000 keys via load balancerEven distribution across nodes (±20%)

Session Continuity (Clustered)

TestMethodValidates
session_createPOST login to pod-0Session cookie returned
session_read_same_nodeGET with session cookie to pod-0Session data present
session_read_other_nodeGET with session cookie to pod-1Session data present (cross-node)
session_update_propagatesUpdate session on pod-0, read on pod-2Updated value visible
session_expiryCreate session, wait beyond gc_maxlifetimeSession expired on all nodes

High Availability Tests

These tests deliberately break things to verify the cluster recovers correctly.

Node Failure

TestMethodValidates
node_crash_cluster_continuesKill pod-1, query pod-0 and pod-2Remaining nodes report 2 members, continue serving
node_crash_keys_availableKill pod-1, read keys that were on pod-1Replicas serve the keys (no data loss)
node_crash_detection_timeKill pod-1, measure time until other nodes detect failureDetected within gossip failure timeout (~10-30s)
node_crash_sessions_surviveKill pod-1 with active sessionsSessions accessible via surviving nodes
node_rejoin_rebalanceKill pod-1, restart itPod-1 rejoins cluster, keys rebalanced back

Rolling Restart

TestMethodValidates
rolling_restart_no_downtimeRestart pods one at a timeHTTP requests succeed continuously (zero failed requests)
rolling_restart_sessions_persistCreate session, rolling restart all podsSession still accessible after all pods restarted
rolling_restart_kv_intactSet 1000 keys, rolling restart all podsAll keys still accessible after restart

Scale Up / Down

TestMethodValidates
scale_up_joinsScale from 3 to 5 replicasNew pods join cluster, all 5 visible in gossip
scale_up_rebalancesScale up, check key distributionKeys redistributed to include new nodes
scale_down_gracefulScale from 5 to 3 replicasDeparting pods transfer keys before shutdown
scale_down_keys_intactScale down, verify all keysNo data loss after scale-down

Network Partition (Advanced)

These tests require network policy manipulation to simulate partitions:

TestMethodValidates
partition_both_sides_serveIsolate pod-0 from pod-1,2Both partitions continue serving local keys
partition_heal_reconcileCreate partition, write to both sides, healLWW conflict resolution merges correctly
partition_no_split_brain_writesPartition, write same key on both sides, healOne value wins (deterministic)

ACME Certificate HA (Clustered)

TestMethodValidates
acme_single_issuer3 nodes, trigger cert issuanceOnly one node contacts Let’s Encrypt (check logs)
acme_leader_failoverKill ACME leader nodeNew leader elected, takes over renewal duties
acme_cert_propagationIssue cert on one nodeAll nodes serve HTTPS with the cert (check via TLS handshake to each pod)
acme_challenge_any_nodeInitiate ACME on pod-0Challenge token servable from pod-1 and pod-2

PHP Response Cache HA

TestMethodValidates
cache_miss_executes_phpGET /blog/hello (first time)PHP executes, response cached in KV
cache_hit_skips_phpGET /blog/hello with matching If-None-Match304, no PHP execution (verify via access log or metric)
cache_hit_any_nodeCache on pod-0, request with ETag to pod-1304 from pod-1 (cache replicated)
cache_invalidationPurge cache entry, request againPHP re-executes, new ETag generated
cache_bypass_authGET with auth cookieCache bypassed, PHP always executes

Test Infrastructure for Cluster Tests

Kind Cluster Configuration

Multi-node Kind cluster for realistic HA testing:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

Spreading ephpm pods across workers via pod anti-affinity ensures node failure tests are meaningful (killing a Kind worker takes down the ephpm pod on it).

Network Policy for Partition Tests

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: partition-pod-0
spec:
  podSelector:
    matchLabels:
      statefulset.kubernetes.io/pod-name: ephpm-cluster-0
  policyTypes:
    - Ingress
    - Egress
  ingress: []   # block all inbound from other pods
  egress: []    # block all outbound to other pods

Apply to simulate partition, delete to heal. The e2e test applies/removes these policies programmatically via the Kubernetes API.

Test Execution Phases

Cluster tests must run in a specific order since some tests are destructive:

Phase 1: Cluster formation (non-destructive)
  → cluster_forms, cluster_gossip_metadata, cluster_node_identity

Phase 2: KV operations (non-destructive)
  → kv_set_get, kv_cross_node, kv_hash_ring, kv_sessions

Phase 3: HA — node failure (destructive, pods restarted)
  → node_crash_*, rolling_restart_*

Phase 4: HA — scaling (destructive, replica count changes)
  → scale_up_*, scale_down_*

Phase 5: HA — network partition (destructive, network policies)
  → partition_*

Each phase waits for the cluster to be fully healthy before proceeding.

E2E Crate — Cluster Test Modules

crates/ephpm-e2e/
  tests/
    # Single-node (existing + expanded)
    php_execution.rs
    static_files.rs
    etag.rs
    compression.rs
    security.rs
    fallback.rs
    headers.rs
    proxy.rs
    tls.rs
    timeouts.rs
    shutdown.rs

    # Cluster tests (new)
    cluster_formation.rs    # gossip, membership, metadata
    kv_single_node.rs       # local KV operations
    kv_resp.rs              # Redis protocol compatibility
    kv_cross_node.rs        # cross-node routing + replication
    kv_sessions.rs          # PHP session storage
    ha_node_failure.rs      # pod crash + recovery
    ha_rolling_restart.rs   # zero-downtime restarts
    ha_scaling.rs           # scale up/down
    ha_partition.rs         # network partition + heal
    ha_acme.rs              # certificate HA
    cache_response.rs       # PHP response cache

Metrics and Assertions

Cluster tests need richer assertions than simple HTTP status checks. The Node API provides the data:

/// Query the Node API for cluster state.
async fn cluster_state(pod_url: &str) -> ClusterState {
    let resp = reqwest::get(format!("{pod_url}/api/kv/cluster")).await.unwrap();
    resp.json::<ClusterState>().await.unwrap()
}

/// Assert all pods see the expected number of cluster members.
async fn assert_cluster_size(pods: &[&str], expected: usize) {
    for pod in pods {
        let state = cluster_state(pod).await;
        assert_eq!(state.members.len(), expected, "pod {pod} sees wrong member count");
    }
}

/// Assert a key is accessible from a specific pod.
async fn assert_key_readable(pod_url: &str, key: &str, expected_value: &str) {
    let resp = reqwest::get(format!("{pod_url}/test-kv-get.php?key={key}")).await.unwrap();
    assert_eq!(resp.status(), 200);
    assert_eq!(resp.text().await.unwrap(), expected_value);
}

/// Wait for cluster to reach target size with timeout.
async fn wait_for_cluster(pods: &[&str], target_size: usize, timeout: Duration) {
    let deadline = Instant::now() + timeout;
    loop {
        let mut all_ready = true;
        for pod in pods {
            let state = cluster_state(pod).await;
            if state.members.len() != target_size {
                all_ready = false;
                break;
            }
        }
        if all_ready { return; }
        assert!(Instant::now() < deadline, "cluster did not reach size {target_size} within {timeout:?}");
        tokio::time::sleep(Duration::from_secs(1)).await;
    }
}

CI Pipeline

# .github/workflows/e2e.yml (extended)
jobs:
  e2e-single:
    strategy:
      matrix:
        php: ["8.4", "8.5"]
    steps:
      - uses: actions/checkout@v4
      - run: cargo xtask e2e-install
      - run: cargo xtask e2e --php-version ${{ matrix.php }} --suite single

  e2e-cluster:
    needs: e2e-single
    strategy:
      matrix:
        php: ["8.5"]   # cluster tests on latest PHP only
    steps:
      - uses: actions/checkout@v4
      - run: cargo xtask e2e-install
      - run: cargo xtask e2e --php-version ${{ matrix.php }} --suite cluster --workers 3

Cluster tests only run on the latest PHP version to keep CI time reasonable. Single-node tests run on the full PHP matrix.


Implementation Order

PhaseScopePriority
1. Expand single-node E2EPHP execution, static files, security, fallback, compression, ETag, headers, timeoutsNow — validates everything we’ve already built
2. Test docroot fixturesAdd missing PHP scripts and static filesNow — required for Phase 1
3. Multi-config Tilt setupMultiple ephpm deployments with different configsNow — needed for security/TLS/timeout tests
4. TLS E2EManual cert tests in KindAfter TLS is stable
5. KV single-node testsSAPI functions, RESP protocol, sessionsAfter KV Phase 1-4
6. Cluster formation testsGossip, membership, Node APIAfter clustering Phase 5-6
7. KV cross-node testsRouting, replication, cross-node readsAfter clustering Phase 7
8. HA testsNode failure, rolling restart, scalingAfter Phase 7
9. Network partition testsSplit-brain, reconciliationAfter Phase 8
10. ACME HA testsCert coordination across nodesAfter ACME on KV
11. Response cache testsETag interception, cache hit/missAfter PHP response cache