Building a Production-Grade E-Commerce Platform with Laravel 12 - Part 8: Zero-Downtime Deployments, Monitoring & Incident Response

A symlink swap or a blue/green flip won't save you from a migration that breaks the code still serving traffic. This part shows that failure with real numbers, fixes it with expand/contract migrations, adds a readiness endpoint that knows about migrations, wires up four alerts that matter for a store (each one triggered on purpose), and ends with an incident runbook you can paste into your repo.

Everything here ran in Docker on one Windows machine, and only blue/green was run, so these are lab measurements, not claims about real traffic; the Kubernetes manifest is untested. This is the last part of the series; the earlier ones covered architecture, containerization, scaling and testing. You don't need them to follow along.

Minimal setup (stand-alone)

Versions: PHP 8.4.26, Laravel Framework 12.69.3, MySQL 8.4, nginx 1.27-alpine, Prometheus v3.15.0 (prom/prometheus:v3.15.0). Behaviour was verified on 12.69.3 only. Laravel 13 now exists, so check the release notes before assuming the same retry and health-route behaviour there.

docker run --rm -v "$PWD":/w -w /w composer:2 create-project laravel/laravel:^12.0 shop

The app is a stripped-down store: an orders table, POST /orders (inserts a row and queues a CapturePayment job), GET /orders/{id}, and a queue worker. The job sleeps 200 ms where a real payment-provider call would go; it does not call Stripe. A Dockerfile.base installs dependencies once and a Dockerfile.release copies a per-release overlay of files on top, so shop:v1, shop:v2 and so on build in seconds (this is the image setup from Part 5, minus everything not needed here):

# Dockerfile.base
FROM php:8.4-cli
RUN apt-get update && apt-get install -y --no-install-recommends unzip git curl && rm -rf /var/lib/apt/lists/* \
 && docker-php-ext-install pdo_mysql pcntl
COPY --from=composer:2 /usr/bin/composer /usr/bin/composer
WORKDIR /app
COPY shop/composer.json shop/composer.lock ./
RUN composer install --no-dev --no-scripts --no-autoloader --no-interaction
COPY shop/ ./
RUN composer dump-autoload --optimize --no-dev --no-scripts && php artisan package:discover

# Dockerfile.release  (docker build -f Dockerfile.release --build-arg RELEASE=v2 -t shop:v2 .)
ARG BASE=shop-base
FROM ${BASE}
ARG RELEASE
ENV RELEASE=${RELEASE} PHP_CLI_SERVER_WORKERS=4
COPY common/ /app/
COPY overlays/${RELEASE}/ /app/
RUN composer dump-autoload --optimize --no-dev --no-scripts
CMD ["sh", "-c", "cd public && exec php -S 0.0.0.0:8000 ../vendor/laravel/framework/src/Illuminate/Foundation/resources/server.php"]

Two shortcuts not to copy into production: PHP's built-in server instead of PHP-FPM or FrankenPHP, and a single MySQL container. Compose runs blue and green behind one nginx, and the health check calls /ready (defined below):

x-app: &app
  env_file: app.env
  depends_on: { db: { condition: service_healthy } }
  healthcheck:
    test: ["CMD", "php", "-r", "exit(@file_get_contents('http://127.0.0.1:8000/ready') === false ? 1 : 0);"]
    interval: 2s
    timeout: 2s
    retries: 3
    start_period: 2s
services:
  blue:   { <<: *app, image: "${BLUE_IMAGE}",  profiles: [apps] }
  green:  { <<: *app, image: "${GREEN_IMAGE}", profiles: [apps] }
  worker:
    image: ${WORKER_IMAGE}
    env_file: app.env
    depends_on: { db: { condition: service_healthy } }
    command: ["php", "artisan", "queue:work", "--sleep=1", "--tries=3"]
    profiles: [apps]
  web:
    image: nginx:1.27-alpine
    ports: ["8080:80"]
    volumes: ["./nginx:/etc/nginx/shop:ro", "./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro"]
  prometheus:
    image: prom/prometheus:v3.15.0
    ports: ["9090:9090"]
    volumes: ["./prom/prometheus.yml:/etc/prometheus/prometheus.yml:ro", "./prom/rules.yml:/etc/prometheus/rules.yml:ro"]
  db:
    image: mysql:8.4
    environment: { MYSQL_ROOT_PASSWORD: labsecret, MYSQL_DATABASE: shop }
    healthcheck: { test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-plabsecret"], interval: 3s, retries: 20 }

The compose file starts with name: lab (hence lab-blue-1). .env holds the three image variables the deploy script rewrites; app.env holds APP_ENV=production, DB_HOST=db, root credentials, LOG_CHANNEL=stderr, SESSION_DRIVER=array, CACHE_STORE=array and QUEUE_CONNECTION=database. Lab credentials only.

The rest of the shop is small enough to list: the routes, the orders migration and the job.

// routes/web.php
Route::withoutMiddleware([ValidateCsrfToken::class])->group(function () {
    Route::post('/orders', [OrderController::class, 'store']);
    Route::get('/orders/{id}', [OrderController::class, 'show']);
});
Route::get('/ready', [HealthController::class, 'ready']);     // below
Route::get('/metrics', [HealthController::class, 'metrics']); // below
Route::get('/version', fn () => ['release' => env('RELEASE', 'unknown')]);

// 2026_01_01_000000_create_orders_table.php, up():
$t->id(); $t->string('email'); $t->decimal('total', 10, 2);
$t->string('status')->default('pending'); $t->timestamps();

// app/Jobs/CapturePayment.php
class CapturePayment implements ShouldQueue
{
    use Queueable;
    public function __construct(public int $orderId) {}
    public function handle(): void
    {
        usleep(200_000); // stands in for the payment provider call (not a real Stripe request)
        DB::table('orders')->where('id', $this->orderId)->update(['status' => 'paid']);
    }
}

OrderController for v1 validates email and total, inserts with DB::table('orders')->insertGetId(...), dispatches CapturePayment and returns 201 {id}. Later releases change only the column handling. HealthController is in the readiness and monitoring sections.

nginx and Prometheus config in full. nginx/active.inc holds one line, set $active green;, which the deploy script rewrites:

log_format shop '$time_iso8601 $status $request_time $upstream_addr';
server {
    listen 80;
    resolver 127.0.0.11 valid=1s;          # Docker's DNS, so the upstream name is re-resolved
    access_log /var/log/nginx/access.log shop;
    include /etc/nginx/shop/active.inc;     # set $active blue;
    location / {
        proxy_pass http://$active:8000;     # a variable, so `nginx -s reload` is the whole flip
        proxy_connect_timeout 1s;
        proxy_read_timeout 10s;
    }
}

global: { scrape_interval: 5s, evaluation_interval: 5s }
rule_files: [/etc/prometheus/rules.yml]
scrape_configs:
  - job_name: shop
    metrics_path: /metrics
    static_configs: [{ targets: ["web:80"] }]

Before each experiment the database was rebuilt at the v1 schema and seeded with 200,000 orders using a recursive CTE.

The load generator is what turns "it should be fine" into a number. Four threads each create an order and read it back through nginx, and every response code is counted per second:

# load.py <seconds> <threads>
import json, sys, threading, time, urllib.request, urllib.error, collections, random
BASE = "http://localhost:8080"
DUR = int(sys.argv[1]); WORKERS = int(sys.argv[2]) if len(sys.argv) > 2 else 4
codes = collections.Counter(); per_sec = collections.defaultdict(collections.Counter)
lock = threading.Lock(); t0 = time.time()

def call(method, path, body=None):
    req = urllib.request.Request(BASE + path, method=method,
        data=json.dumps(body).encode() if body else None,
        headers={"Content-Type": "application/json", "Accept": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=5) as r: return r.status, r.read()
    except urllib.error.HTTPError as e: return e.code, b""
    except Exception: return 0, b""

def worker():
    while time.time() - t0 < DUR:
        s, b = call("POST", "/orders", {"email": "[email protected]", "total": round(random.uniform(5, 99), 2)})
        s2 = call("GET", "/orders/%d" % json.loads(b)["id"])[0] if s == 201 else None
        with lock:
            for c in (s, s2):
                if c is not None: codes[c] += 1; per_sec[int(time.time() - t0)][c] += 1
        time.sleep(0.05)

ts = [threading.Thread(target=worker) for _ in range(WORKERS)]
[t.start() for t in ts]; [t.join() for t in ts]
tot = sum(codes.values()); bad = sum(v for k, v in codes.items() if k not in (200, 201))
print("requests=%d  status=%s  non-2xx=%d (%.2f%%)" % (tot, dict(sorted(codes.items())), bad, 100 * bad / tot))
print("seconds containing errors:", [s for s, c in sorted(per_sec.items()) if any(k not in (200, 201) for k in c)])

Laravel zero downtime deployment: why the flip alone isn't enough

The usual advice: build the release in its own directory or container, run migrations, flip a symlink or load balancer, reload workers. The Laravel 12.x deployment docs add php artisan optimize and php artisan reload for long-running processes such as queue workers.

None of that helps if the migration changes a column the old release still uses, because migrations run before the flip. For a stretch of the deploy, old code runs against the new schema. I tested the naive version. Release vbad renames orders.total to orders.total_cents in one migration and ships code that uses the new name. The commands were these (deploy.sh is listed in the next section; T0 makes it print timestamps relative to the load generator's start):

export T0=$(date +%s.%N)
python load.py 45 4 > expA_load.txt &   # 4 threads, 45 seconds
sleep 10
./deploy.sh vbad

The script sleeps 10 s, but the first log line is stamped t+12 (the extra 2 s is Windows process start-up). Output saved as expA_deploy.txt and expA_load.txt:

[t+12s] [deploy] vbad: active=green idle=blue
[t+12s] [migrate] start
[migrate]   2026_02_01_000000_rename_orders_total ......................... 66.39ms DONE
[t+18s] [migrate] end
[t+31s] [deploy] blue health=healthy after ~3s of polling
[t+35s] [nginx] flipped to blue
[t+46s] [deploy] done: traffic -> blue (vbad) in 36.4s

requests=1116  status={200: 328, 201: 331, 500: 457}  non-2xx=457 (40.95%)
seconds containing errors: [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]

41% of requests failed, for 18 consecutive seconds, though the rename itself took 66 ms. The migration committed at about t+17 (the [migrate] start/end lines include roughly 6 seconds of docker run start-up around a 66 ms migration). Errors stop in second 34 and nginx flipped at t+35. So the exposure window is simply migration commit → traffic flip: the old container kept running insert ... total against a table with no such column for as long as the new container took to start and pass its health check. A faster deploy shrinks that window; it doesn't remove it.

Expand / contract: make every step safe for both releases

Split the change so that at every moment the running release works with the current schema, and the previous release does too:

1. Expand (v2): add total_cents as nullable and make total nullable. Ship code that writes both columns and reads total_cents, falling back to total * 100.
2. Backfill: fill total_cents for old rows in small batches, outside the migration.
3. Switch (v3): code reads and writes only total_cents.
4. Contract (v4): drop total, only once nothing running touches it.

// v2 migration: additive only
Schema::table('orders', function (Blueprint $t) {
    $t->unsignedBigInteger('total_cents')->nullable()->after('total');
    $t->decimal('total', 10, 2)->nullable()->change(); // so v3 can stop writing it
});

// php artisan orders:backfill-cents --chunk=1000 : small batches, short pause
do {
    $n = DB::update('update orders set total_cents = round(total * 100)
                     where total_cents is null and total is not null limit '.(int) $this->option('chunk'));
    usleep(50_000);
} while ($n > 0);

I ran all four deploys back to back (expB.sh, with python load.py 230 4 running throughout) on the same 200,000-row table. The lines below are the saved expB_log.txt:

[t+7s] deploy v2 (expand)
[migrate]   2026_02_01_000000_expand_orders_total_cents ........................ 6s DONE
[t+34s] [deploy] blue health=healthy after ~4s of polling
[t+41s] [deploy] done: traffic -> blue (v2) in 32.9s
[t+41s] backfill
backfilled 200538 rows
[t+129s] backfill took 88s
null_cents
0
[t+131s] deploy v3 (read/write new only)
[t+156s] [deploy] green health=healthy after ~4s of polling
[t+163s] [deploy] done: traffic -> green (v3) in 31.8s
[t+164s] deploy v4 (contract)
[migrate]   2026_03_01_000000_contract_drop_orders_total .................. 54.01ms DONE
[t+190s] [deploy] blue health=healthy after ~4s of polling
[t+198s] [deploy] done: traffic -> blue (v4) in 33.9s

requests=8418  status={200: 4209, 201: 4209}  non-2xx=0 (0.00%)
seconds containing errors: []

Zero failed requests across three deploys and an 88-second backfill. (The backfill count is 538 above the 200,000 seed because the load generator kept creating orders.) The same rename that cost 41% errors cost nothing once it was split in four.

The expand migration took 6 seconds on this table, against 66 ms for the rename and 54 ms for the final drop. The ->change() that makes total nullable is a table alter; it caused no errors here, but time it on a copy of your own table before trusting a number from my laptop.

Decision rules

• Merge expand and switch? Only if the old release never writes the changed column and you never need to roll back. For a rename or retype, keep them apart.
• Backfill in a migration? No, use a command. A migration holds the deploy for the whole backfill (88 s here).
• When to contract? When the switch release has outlived your longest rollback window and the query below returns 0. Not after a fixed number of days. Rolling back after contract needs a backup, because the dropped data is gone.

Jobs queued by older releases can still sit in the jobs table, so run this with @switch_flip set to the time the switch release went live:

-- no job or failed job older than the switch release may remain
set @switch_flip = unix_timestamp('2026-03-01 12:00:00');   -- when v3 went live
select count(*) as jobs_older_than_switch, min(from_unixtime(created_at)) as oldest
  from jobs where created_at < @switch_flip;
select count(*) as failed_older_than_switch
  from failed_jobs where unix_timestamp(failed_at) < @switch_flip;

Both counts need to be 0. I ran the query on a throwaway MySQL at the v1 schema (expE_sql.txt, with @switch_flip set to unix_timestamp() - 600), first with an empty queue and then after inserting one job created an hour ago:

--- empty queue, cutoff = 10 minutes ago:
jobs_older_than_switch	oldest
0	NULL
failed_older_than_switch
0
--- after queueing one job created an hour ago:
jobs_older_than_switch	oldest
1	2026-10-07 18:37:52
failed_older_than_switch
0

A non-zero jobs_older_than_switch is the stop sign: those jobs would run old payloads against new code. Retry or delete old failed jobs first. Workers also hold the old code in memory until php artisan reload (or a container restart) replaces them; see Part 6.

The blue/green flip script

The whole deploy is five steps: migrate, start the idle colour on the new image, wait until it's ready, flip nginx, restart the worker. The complete deploy.sh:

#!/usr/bin/env bash
# usage: ./deploy.sh <release>
set -euo pipefail
export MSYS_NO_PATHCONV=1
REL=$1
ACTIVE=$(sed -E 's/.*\$active ([a-z]+);/\1/' nginx/active.inc)
IDLE=$([ "$ACTIVE" = blue ] && echo green || echo blue)
t0=$(date +%s.%N)
T0=${T0:-$t0}   # the caller can pass its own start time so the log lines up with the load generator
el(){ awk "BEGIN{printf \"t+%ds\", $(date +%s.%N)-$T0}"; }
echo "[$(el)] [deploy] $REL: active=$ACTIVE idle=$IDLE"

echo "[$(el)] [migrate] start"
docker run --rm --network lab_default --env-file app.env shop:$REL php artisan migrate --force --no-interaction | sed 's/^/[migrate] /'
echo "[$(el)] [migrate] end"

sed -i "s#^$(echo $IDLE | tr a-z A-Z)_IMAGE=.*#$(echo $IDLE | tr a-z A-Z)_IMAGE=shop:$REL#" .env
docker compose --profile apps up -d --no-deps --force-recreate $IDLE >/dev/null 2>&1
for i in $(seq 1 60); do
  s=$(docker inspect -f '{{.State.Health.Status}}' lab-$IDLE-1)
  [ "$s" = healthy ] && break
  sleep 1
done
echo "[$(el)] [deploy] $IDLE health=$s after ~${i}s of polling"
[ "$s" = healthy ] || { echo "[$(el)] [deploy] ABORT: $IDLE never became ready; traffic stays on $ACTIVE"; exit 1; }

echo "set \$active $IDLE;" > nginx/active.inc
docker compose exec -T web nginx -s reload 2>&1 | sed 's/^/[nginx] /'
echo "[$(el)] [nginx] flipped to $IDLE"
sed -i "s#^WORKER_IMAGE=.*#WORKER_IMAGE=shop:$REL#" .env
docker compose --profile apps up -d --no-deps --force-recreate worker >/dev/null 2>&1
echo "[$(el)] [deploy] done: traffic -> $IDLE ($REL)"

A graceful nginx -s reload is the whole flip. If the new colour never becomes ready, the script aborts and traffic stays put; that abort path is what blue/green buys you over an in-place restart.

Rolling updates on Kubernetes (not run)

On Kubernetes a rolling update replaces pods gradually, so old and new releases serve traffic at the same time for the whole rollout. Expand/contract is then mandatory. This untested sketch shows where /up and /ready from the next section plug in; run migrations as a separate step before applying it:

# UNTESTED: I did not run Kubernetes for this post.
apiVersion: apps/v1
kind: Deployment
metadata: { name: shop }
spec:
  replicas: 3
  strategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  selector: { matchLabels: { app: shop } }
  template:
    metadata: { labels: { app: shop } }
    spec:
      containers:
        - name: app
          image: registry.example.com/shop:v2
          ports: [{ containerPort: 8000 }]
          livenessProbe:  { httpGet: { path: /up,    port: 8000 }, periodSeconds: 10 }
          readinessProbe: { httpGet: { path: /ready, port: 8000 }, periodSeconds: 3, timeoutSeconds: 3, failureThreshold: 2 }

maxUnavailable: 0 keeps every old pod until a new one is ready.

Readiness vs liveness: keep them different

The Kubernetes docs define the difference that matters: a failing liveness probe gets the container killed and restarted; a pod whose readiness probe fails does not receive traffic through Services. Laravel 12 ships a built-in /up route that returns 200 if the app booted, and fires a DiagnosingHealth event you can hook for extra checks (docs).

My rule: use /up for liveness and write your own /ready for readiness. Restarting a container does nothing for a database outage, so liveness shouldn't depend on the database. Readiness should, plus one thing most guides skip: whether the migrations this release ships have been applied.

My first version of /ready set PDO::ATTR_TIMEOUT => 2 and ran its check through Laravel's DB::select(). Pointed at a black-holed address (10.255.255.1), it took 8 seconds, not 2 (expD_ready.txt, run inside the app container):

dbselect         8.02s
rawpdo           2.00s
dbselect         8.02s
rawpdo           2.00s
rawpdo-nooption  21.03s

rawpdo is a bare new PDO(...) with the same 2 s option; rawpdo-nooption has no option at all. The likely cause is framework retry layers. In the installed 12.69.3 source: LostConnectionDetector lists 'SQLSTATE[HY000] [2002] Connection timed out' (line 53); Connectors/Connector::createConnection catches a failed connect and, for a lost-connection error, calls createPdoConnection a second time (tryAgainIfCausedByLostConnection, line 82); and Connection::tryAgainIfCausedByLostConnection (line 994) reconnects and re-runs the query once more. Two connect attempts per connection, on two connections, is four 2 s timeouts, which fits the 8.02 s. I didn't step through it with a debugger, so this is a reading that fits the measurement. The probe therefore opens its own PDO connection and bypasses the retry layer:

public function ready()
{
    $checks = [];
    try {
        // Raw PDO, not DB::select(): Laravel retries "Connection timed out", turning 2s into ~8s.
        $c = config('database.connections.mysql');
        $pdo = new \PDO(
            "mysql:host={$c['host']};port={$c['port']};dbname={$c['database']}",
            $c['username'], $c['password'],
            [\PDO::ATTR_TIMEOUT => 2, \PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION]
        );
        $checks['database'] = true;

        $applied = $pdo->query('select migration from migrations')->fetchAll(\PDO::FETCH_COLUMN);
        $shipped = array_map(fn ($f) => basename($f, '.php'), glob(database_path('migrations/*.php')));
        $missing = array_values(array_diff($shipped, $applied));
        $checks['migrations'] = $missing === [];
        if ($missing) $checks['missing'] = $missing;
    } catch (\Throwable $e) {
        $checks['database'] = $checks['database'] ?? false;
    }
    $ok = ($checks['database'] ?? false) && ($checks['migrations'] ?? false);
    return response()->json(['ready' => $ok, 'checks' => $checks], $ok ? 200 : 503);
}

Run against throwaway containers of the same image (expD_ready.sh, output quoted from expD_ready.txt): v1 code on a v1 schema, v4 code on a v1 schema (migrations pending), and v1 code with the database host set to 10.255.255.1, with three warm repeats of the unreachable case:

--- ready (v1 code, v1 schema):
/ready HTTP 200 in 0.380476s
{"ready":true,"checks":{"database":true,"migrations":true}}
--- pending migrations (v4 code, v1 schema):
/ready HTTP 503 in 0.396736s
{"ready":false,"checks":{"database":true,"migrations":false,"missing":["2026_02_01_000000_expand_orders_total_cents","2026_03_01_000000_contract_drop_orders_total"]}}
/up HTTP 200 in 0.059794s
--- db unreachable (black-holed 10.255.255.1), first request then 3 warm repeats:
/ready HTTP 503 in 2.399744s
{"ready":false,"checks":{"database":false}}
/up HTTP 200 in 0.042889s
/ready HTTP 503 in 2.023684s
{"ready":false,"checks":{"database":false}}
/ready HTTP 503 in 2.032170s
{"ready":false,"checks":{"database":false}}
/ready HTTP 503 in 2.035110s
{"ready":false,"checks":{"database":false}}

So the probe is now bounded at about the 2 s I configured (2.40 s on the first request to a fresh container), down from 8.02 s. Two caveats. PDO::ATTR_TIMEOUT is a connect timeout; a database that accepts the connection and then stalls is a failure I didn't test. And the Kubernetes docs give timeoutSeconds a default of 1 second, so a 2 s dependency check needs it raised on the pod (my Compose check uses timeout: 2s). Otherwise the probe is cut off before your own timeout fires.

The split works as intended: /up stays 200 in both failure cases while /ready returns 503, and the pending-migration body names the migrations, which is what you want to see in a failed rollout.

Laravel zero downtime deployment monitoring: metrics and alerts for a store

Two sources set the approach. The Prometheus alerting practices say to alert on symptoms rather than causes, "aim to have as few alerts as possible", and to test the whole alert path end to end. The Google SRE book defines four golden signals: latency, traffic, errors and saturation. For a store, map them to money:

Page on the first two. Zero orders at a time you normally get them is the cheapest "something is wrong" alarm there is, and it fires on causes you never thought of.

Queue age is the metric that matters most here, and it comes from the queue setup in Part 6 (reuse any queue metrics you export there). The app exposes /metrics in Prometheus text format. Shortcut warning: these are computed with SQL at scrape time, which is fine for a lab and wrong for production. In production, increment real counters in code and keep database queries off your scrape path.

// routes/web.php: Route::get('/metrics', [HealthController::class, 'metrics']);
public function metrics()
{
    $now = (int) DB::selectOne('select unix_timestamp() as t')->t;
    $orders = (int) DB::table('orders')->count();
    $byStatus = DB::table('orders')->selectRaw('status, count(*) c')->groupBy('status')->pluck('c', 'status');
    $pending = (int) DB::table('jobs')->whereNull('reserved_at')->count();
    $oldest = DB::table('jobs')->whereNull('reserved_at')->min('available_at');
    $age = $oldest ? max(0, $now - (int) $oldest) : 0;
    $failed = (int) DB::table('failed_jobs')->count();

    $out = "# TYPE shop_orders_created_total counter\nshop_orders_created_total $orders\n";
    $out .= "# TYPE shop_orders_by_status gauge\n";
    foreach (['pending', 'paid'] as $s) {
        $out .= 'shop_orders_by_status{status="'.$s.'"} '.($byStatus[$s] ?? 0)."\n";
    }
    $out .= "# TYPE shop_jobs_pending gauge\nshop_jobs_pending $pending\n";
    $out .= "# TYPE shop_oldest_job_age_seconds gauge\nshop_oldest_job_age_seconds $age\n";
    $out .= "# TYPE shop_failed_jobs gauge\nshop_failed_jobs $failed\n";

    return response($out, 200, ['Content-Type' => 'text/plain; version=0.0.4']);
}

Prometheus scrapes it through nginx every 5 seconds (scrape_interval: 5s, evaluation_interval: 5s, target web:80). The rules, with deliberately short lab thresholds so the experiment fits in minutes:

groups:
  - name: shop
    rules:
      - alert: ShopDown
        expr: up{job="shop"} == 0
        for: 15s
      - alert: PaymentQueueStalled
        expr: shop_oldest_job_age_seconds > 20
        for: 15s
      - alert: NoOrdersCreated
        expr: increase(shop_orders_created_total[1m]) == 0
        for: 30s
      - alert: FailedJobsGrowing
        expr: increase(shop_failed_jobs[1m]) > 0

For real traffic I'd start nearer 2 to 5 minutes of queue age and 5 to 15 minutes of zero orders (suggestions, not measurements), then tune to your daily curve.

Testing each alert

I drained the queue first (a leftover backlog had made an earlier run useless), ran 5 workers (about 25 jobs/s of capacity against roughly 14 orders/s of load) and waited for alerts: none before starting the clock. One fault at a time:

• t+20: stop all queue workers
• t+170: queue a job whose handler class doesn't exist, so it fails every attempt (failjob.sh inserts one row into jobs)
• t+260: stop the app container behind nginx
• t+340: stop the load generator (no more orders)

[t+0s] alerts: none   (jobs=2 oldest=0s failed=0)
[t+20s] ACTION: FAULT 1  stop all queue workers
[t+52s] alerts: {'PaymentQueueStalled': 'pending'}   (jobs=299 oldest=31s failed=0)
[t+66s] alerts: {'PaymentQueueStalled': 'firing'}   (jobs=485 oldest=45s failed=0)
[t+70s] ACTION: RECOVER 1 start the 5 workers
[t+131s] alerts: none   (jobs=155 oldest=15s failed=0)
[t+170s] ACTION: FAULT 2  queue one job that always throws
[t+181s] alerts: {'FailedJobsGrowing': 'firing'}   (jobs=1 oldest=0s failed=1)
[t+237s] alerts: none   (jobs=1 oldest=0s failed=1)
[t+260s] ACTION: FAULT 3  stop the app container behind nginx
[t+271s] alerts: {'ShopDown': 'pending'}   (metrics unreachable)
[t+287s] alerts: {'ShopDown': 'firing'}   (metrics unreachable)
[t+300s] ACTION: RECOVER 3 start the app container
[t+311s] alerts: none   (jobs=2 oldest=0s failed=1)
[t+340s] ACTION: FAULT 4  stop the load generator (no more orders)
[t+407s] alerts: {'NoOrdersCreated': 'pending'}   (jobs=0 oldest=0s failed=1)
[t+436s] alerts: {'NoOrdersCreated': 'firing'}   (jobs=0 oldest=0s failed=1)
[t+450s] ACTION: RECOVER 4 start the load generator
[t+462s] alerts: none   (jobs=3 oldest=0s failed=1)

All four fire, one at a time, from none, and all four resolve. What the timings say:

• Detection lag is threshold plus for plus scrape. PaymentQueueStalled went pending 32 s after the workers stopped and fired 14 s later. ShopDown took 27 s from stop to firing.
• NoOrdersCreated is the slowest (96 s from stopping the load to firing) because the one-minute increase() window has to empty out before it reads zero.
• Restarting the workers did not clear the queue alert instantly. It resolved 61 s after the restart, at oldest=15s with 155 jobs still queued. The rule watches the age of the oldest job, so it clears when that age drops under the threshold, not when the backlog hits zero. "Workers are back" is not "resolved", and "alert cleared" is not "backlog gone".
• FailedJobsGrowing has no for, so it fired 11 s after the bad job was queued, and it clears on its own after the one-minute window even though failed=1 is still in the table. Treat it as a "look at failed_jobs" ticket, not a status gauge.

I didn't test notification delivery (Alertmanager, Slack, a pager). Whatever tool you pick (Nightwatch, Pulse or this setup), check that it can page on "zero orders" and "oldest job age", not just on exceptions.

An incident runbook template

Keep it in the repo next to deploy.sh. This is a template, not a record of any incident of mine.

# Incident: <one-line symptom>      Severity: SEV1 | SEV2 | SEV3
Started: <time, with timezone>   Commander: <name>   Channel: <link>

## 1. Stabilise (first 10 minutes, before diagnosing)
- [ ] Did a deploy go out in the last hour? `git log -5`, check release version at /version
- [ ] If yes and it is the likely cause: flip back to the previous colour (nginx/active.inc), reload.
- [ ] Is the contract migration already applied? If yes, rollback is NOT safe: see step 3.
- [ ] Post status to support: what's broken, what works, next update time.

## 2. Triage by signal
| Check | Where | Meaning |
|---|---|---|
| /ready of each app instance | curl | 503 + `missing` => migration/release mismatch |
| shop_oldest_job_age_seconds | Prometheus | payments not being captured |
| shop_failed_jobs | Prometheus | jobs failing, read failed_jobs.exception |
| orders per minute | Prometheus | checkout broken or traffic gone |

## 3. Fix
- Roll back code only if the schema supports the old release (expand done, contract not).
- Otherwise roll forward with a hotfix; never edit data by hand without a backup and a second pair of eyes.
- After a worker/queue fix: confirm the oldest-job age is dropping, not just that the worker is running.

## 4. Close
- [ ] Alerts resolved (not just silenced); support informed
- [ ] Affected orders identified by time window and refunded/recaptured
- [ ] Post-incident note within 2 working days: timeline, cause, what detected it
- [ ] Each follow-up has an owner and a date

What to do next

1. Write your next rename, retype or drop as expand, backfill, switch, contract, and run the jobs query before the last deploy.
2. Add a /ready that checks the database and unapplied migrations, keep /up for liveness, and test the timeout against a black-holed host.
3. Add the two money alerts first, orders per minute and oldest-job age, and make your deploy script abort when the new release isn't ready.
4. Copy the runbook into your repo and run one fire drill with the worker stopped.