Laravel Queue Batches: Recovering Failed Jobs Without Re-running Everything

A Laravel batch of ten jobs lost one job to an upstream 500. You want that job back without re-running the other nine. This is the laravel queue batch failed jobs problem, and I ran it to see what queue:retry-batch really does. With the SkipIfBatchCancelled middleware on the jobs, retry-batch re-queues the failed job, the worker prints DONE, then() fires, progress reads 100%, and the work is still not done. Without that middleware, the same retry does finish the job.

The one-line fix, if your chunks are independent: add ->allowFailures() to the batch, and retry-batch works. If the work is all-or-nothing, skip retry-batch and recover from a ledger table. The rest of this post shows both, with output, then covers the settings that decide whether a slow job runs twice.

Environment, so you can check my numbers

Everything ran in throwaway composer:latest Docker containers: PHP 8.5.11, Laravel 13.35.0, no pcntl. The small tests used the database queue driver on SQLite. The 10,000-job runs and one retry_after run used Redis 7.4 (a second container) through predis/predis 3.7, with the ledger still on SQLite. Outputs come from two script runs on 2026-10-09 (the first at 04:44 to 04:53 UTC, the second ending at 05:23 UTC).

$ php artisan --version
Laravel Framework 13.35.0
$ grep -n "'retry_after'" config/queue.php | head -1
43:            'retry_after' => (int) env('DB_QUEUE_RETRY_AFTER', 90),
$ php -m | grep -i pcntl || echo "no pcntl"
no pcntl

Doc quotes come from the 13.x queue docs, read from the docs repo's queues.md on 2026-10-09.

The test rig

Each job is an ImportChunk using Batchable and SkipIfBatchCancelled. Chunks listed in EXP_FAIL (default 4) throw until a cache flag flips, which simulates an outage that later recovers. Every successful chunk writes a row to imported_rows and a row to chunk_ledger in one transaction. Skipped jobs write nothing, so they can't hide behind a green worker line.

<?php
// app/Jobs/ImportChunk.php
namespace App\Jobs;

use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Middleware\SkipIfBatchCancelled;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\DB;

class ImportChunk implements ShouldQueue
{
    use Batchable, Queueable;

    public $tries = 1;

    public function __construct(public int $chunk, public string $import = 'demo') {}

    public function middleware(): array
    {
        // EXP_NO_SKIP=1 is the no-middleware control run
        return env('EXP_NO_SKIP') ? [] : [new SkipIfBatchCancelled];
    }

    public function handle(): void
    {
        $done = DB::table('chunk_ledger')->where(['import' => $this->import, 'chunk' => $this->chunk])->exists();
        if ($done) {
            return; // already committed by an earlier run
        }

        $failing = array_map('intval', explode(',', env('EXP_FAIL', '4')));
        if (in_array($this->chunk, $failing, true) && ! Cache::get('upstream_fixed')) {
            throw new \RuntimeException("chunk {$this->chunk}: upstream returned 500");
        }
        // the work and its ledger row commit together or not at all
        DB::transaction(function () {
            DB::table('imported_rows')->insert(['import' => $this->import, 'chunk' => $this->chunk]);
            DB::table('chunk_ledger')->insertOrIgnore(['import' => $this->import, 'chunk' => $this->chunk]);
        });
        if (! env('EXP_QUIET')) { // off for the 10k runs, to keep the log small
            file_put_contents(storage_path('logs/done.log'), "chunk {$this->chunk} done\n", FILE_APPEND);
        }
    }
}

The dispatcher, run as php artisan exp:batch default|allow --n=10:

<?php
// app/Console/Commands/Exp.php (imports: ImportChunk, Batch, Command, Bus, Cache)
class Exp extends Command
{
    protected $signature = 'exp:batch {mode} {--n=10}';

    public function handle()
    {
        Cache::forget('upstream_fixed');
        $jobs = collect(range(1, (int) $this->option('n')))->map(fn ($i) => new ImportChunk($i))->all();
        $b = Bus::batch($jobs)->name($this->argument('mode'))
            ->then(fn (Batch $b) => file_put_contents(storage_path('logs/done.log'), "THEN fired\n", FILE_APPEND))
            ->catch(fn (Batch $b, \Throwable $e) => file_put_contents(storage_path('logs/done.log'), "CATCH fired: {$e->getMessage()}\n", FILE_APPEND))
            ->finally(fn (Batch $b) => file_put_contents(storage_path('logs/done.log'), "FINALLY fired\n", FILE_APPEND));
        if ($this->argument('mode') === 'allow') {
            $b->allowFailures();
        }
        $this->info($b->dispatch()->id);
    }
}

php artisan exp:show <id> is a second command that prints Bus::findBatch($id) as JSON: totalJobs, pendingJobs, failedJobs, processedJobs(), progress(), finished(), cancelled(). Workers were plain php artisan queue:work --stop-when-empty.

Laravel queue batch failed jobs: how to recover

What a failed job does to a default batch

The docs say it plainly: "When a job within a batch fails, Laravel will automatically mark the batch as 'cancelled'" (Batch Failures). Ten jobs, chunk 4 failing:

2026-10-09 04:45:27 App\Jobs\ImportChunk ..................... 179.45ms FAIL
2026-10-09 04:45:28 App\Jobs\ImportChunk ...................... 66.63ms DONE
2026-10-09 04:45:28 App\Jobs\ImportChunk ...................... 91.58ms DONE
... chunks 7-10 also print DONE ...
--- done.log
chunk 1 done
chunk 2 done
chunk 3 done
CATCH fired: chunk 4: upstream returned 500
FINALLY fired
{"total":10,"pending":1,"failed":1,"processed":9,"progress":90,"finished":true,"cancelled":true}
3 rows, 3 ledger rows, 3 distinct chunks | ledger: 1,2,3

The worker printed DONE for chunks 5-10, but none of them wrote anything. SkipIfBatchCancelled returned early and the worker counts that as success. The batch row says processed: 9, which includes the skipped jobs. Watch only worker output and you'd think nine of ten chunks worked. Three did.

queue:retry-batch on a cancelled batch depends on the middleware

The docs introduce the middleware like this: "for convenience, you may assign the SkipIfBatchCancelled middleware to the job" (Cancelling Batches). That is an offer, not a recommendation, but plenty of batch jobs add it. With it on the job, I set the flag so the "upstream" was healthy again, then ran the documented recovery:

$ php artisan queue:retry-batch a2f019fa-4e84-49bb-96e0-1eac8987921f
   INFO  Pushing failed queue jobs of the batch [a2f019fa-...] back onto the queue.
  955a0664-530c-42f9-a0b2-28881d878d01 ......................... 372.59ms DONE
$ php artisan queue:work --stop-when-empty
THEN fired
{"total":10,"pending":0,"failed":1,"processed":10,"progress":100,"finished":true,"cancelled":true}
3 rows, 3 ledger rows, 3 distinct chunks | ledger: 1,2,3

The failed job was re-queued and the worker printed DONE. But the batch was still cancelled: true, so SkipIfBatchCancelled skipped it again. Nothing was written (ledger still 1,2,3), yet then() fired and progress hit 100%.

The control is the same rig run with EXP_NO_SKIP=1. Without the skip, the cancelled batch keeps executing jobs 5-10, and retry-batch then does work:

chunk 1 done;chunk 2 done;chunk 3 done;CATCH fired: chunk 4: upstream returned 500;chunk 5 done;...;chunk 10 done;FINALLY fired;
{"total":10,"pending":1,"failed":1,"processed":9,"progress":90,"finished":true,"cancelled":true}
9 rows, 9 ledger rows, 9 distinct chunks | ledger: 1,2,3,5,6,7,8,9,10
... after retry-batch ...
chunk 4 done;THEN fired;
{"total":10,"pending":0,"failed":1,"processed":10,"progress":100,"finished":true,"cancelled":true}
10 rows, 10 ledger rows, 10 distinct chunks | ledger: 1,2,3,4,5,6,7,8,9,10

So the trap is the combination of a cancelled batch and SkipIfBatchCancelled. The control has its own cost: a cancelled batch without the middleware keeps running the remaining jobs after the first failure, which is rarely what "cancel" was meant to do.

My rule: pick one behaviour on purpose. Independent chunks get allowFailures() (below). All-or-nothing work keeps the middleware and treats a cancelled batch as dead, recovered from a ledger.

Recover a cancelled batch from a per-chunk ledger

job_batches can't tell you what is missing: it stores counts and failed job IDs, and processed includes skipped jobs. The reliable record is a table you write yourself, one row per chunk, inserted in the same transaction as the work (see handle() above, including the guard at the top that makes a repeated chunk a no-op):

// database/migrations/2026_10_09_000000_create_chunk_ledger.php
Schema::create('chunk_ledger', function (Blueprint $t) {
    $t->string('import');
    $t->unsignedInteger('chunk');
    $t->timestamp('done_at')->useCurrent();
    $t->primary(['import', 'chunk']);
});

The todo list is "everything in the plan that has no ledger row", and recovery dispatches only that as a fresh batch:

<?php
// routes/console.php
Artisan::command('exp:recover {--n=10}', function () {
    Cache::put('upstream_fixed', true);
    $done = DB::table('chunk_ledger')->where('import', 'demo')->pluck('chunk')->all();
    $todo = array_values(array_diff(range(1, (int) $this->option('n')), $done));
    $this->line('ledger has: '.implode(',', $done).' | todo: '.implode(',', $todo));
    $new = Bus::batch(array_map(fn ($i) => new ImportChunk($i), $todo))
        ->then(fn (Batch $b) => file_put_contents(storage_path('logs/done.log'), "THEN fired\n", FILE_APPEND))
        ->dispatch();
    $this->line("recovery batch: {$new->id}");
});

Run against a fresh cancelled batch of ten:

(batch row identical to the first run: 9 processed, 1 failed, cancelled)
ledger has: 1,2,3 | todo: 4,5,6,7,8,9,10
recovery batch: a2f01b5d-7ee0-49ae-aa97-75ecbd299ac3
chunk 1 done;...;CATCH fired: ...;FINALLY fired;chunk 4 done;chunk 5 done;...;chunk 10 done;THEN fired;
{"total":7,"pending":0,"failed":0,"processed":7,"progress":100,"finished":true,"cancelled":false}
--- original
(same cancelled batch row as above)
10 rows, 10 ledger rows, 10 distinct chunks | ledger: 1,2,3,4,5,6,7,8,9,10

All seven missing chunks ran, chunks 1-3 were not repeated, and the new batch reports failed: 0, so alerting on that number stays meaningful. The old batch row still says cancelled at 90%, so store the recovery batch ID next to the original. The ledger guard is meant for overlapping recoveries (the second finds the row and returns); I did not test that.

Use allowFailures() when jobs are independent

For chunked imports, one bad chunk shouldn't block the rest. Same rig, exp:batch allow:

chunk 1 done;chunk 2 done;chunk 3 done;CATCH fired: chunk 4: upstream returned 500;chunk 5 done;...;chunk 10 done;FINALLY fired;
{"total":10,"pending":1,"failed":1,"processed":9,"progress":90,"finished":false,"cancelled":false}

Chunks 5-10 ran. After fixing the upstream, queue:retry-batch on this batch works:

chunk 4 done;THEN fired;
{"total":10,"pending":0,"failed":1,"processed":10,"progress":100,"finished":true,"cancelled":false}

Three details from the runs:

• catch fires once per batch. The docs say "This callback is only invoked for the first job that fails within the batch." To check it, I ran allowFailures() with two failing chunks (EXP_FAIL=4,7): grep -c "CATCH fired" printed 1, and the batch read "pending":2,"failed":2. Use $batch->failedJobs to count failures. The docs also say allowFailures accepts a closure that "will be executed on each job failure"; I did not run that form.
• finally fired while finished was still false. Put "mark import complete" in then, not finally.
• failedJobs stayed at 1 after a successful retry. Alert on pendingJobs and finished(), not on that number alone.

The same thing with 10,000 chunks

Ten jobs prove behaviour, not cost. So I repeated both patterns with 10,000 chunks, chunk 4 failing, on a Redis queue with one worker (QUEUE_CONNECTION=redis, exp:batch ... --n=10000, EXP_QUIET=1):

##### allowFailures, 10000 chunks
dispatch 10000 jobs: 15.2s
worker: 333.9s
{"total":10000,"pending":1,"failed":1,"processed":9999,"progress":100,"finished":false,"cancelled":false}
9999 rows, 9999 ledger rows, 9999 distinct chunks
retry-batch + worker: 22.9s
{"total":10000,"pending":0,"failed":1,"processed":10000,"progress":100,"finished":true,"cancelled":false}
10000 rows, 10000 ledger rows, 10000 distinct chunks

##### default batch, 10000 chunks, then ledger recovery
worker: 239.5s
{"total":10000,"pending":1,"failed":1,"processed":9999,"progress":100,"finished":true,"cancelled":true}
3 rows, 3 ledger rows, 3 distinct chunks
todo query: 9997 todo, 539.5 ms
recover + worker: 390.6s
{"total":9997,"pending":0,"failed":0,"processed":9997,"progress":100,"finished":true,"cancelled":false}
10000 rows, 10000 ledger rows, 10000 distinct chunks

What this shows:

• progress reads 100 with one job still pending. progress() is a rounded percentage, so at this size it hides a missing job. Check pendingJobs and finished, never progress.
• Retrying one failed job cost 22.9 s end to end (retry-batch plus a worker pass); I did not profile where that time goes. Re-dispatching the 9,997 missing chunks after the cancel took 390.6 s, versus 22.9 s when allowFailures() had let the other chunks finish. At this size the choice between the two patterns is the choice between 23 seconds and over six minutes of repeated work.
• Even after the cancel, the worker spent 239.5 s walking through 9,996 skipped jobs.
• Throughput was about 30 jobs/s for one worker, with each job doing a SQLite transaction. That number is a property of my rig, not of Redis; I did not separate the two.
• The todo query took 539.5 ms, but the ledger held only 3 rows at that moment. I did not time it against a nearly full ledger.

Laravel queue retry settings: tries, backoff and maxExceptions

This is the Flaky job. It always throws:

class Flaky implements ShouldQueue
{
    use Queueable;

    public $tries = 4;
    public $backoff = [1, 3];
    public $maxExceptions = 2;

    public function handle(): void
    {
        file_put_contents(storage_path('logs/flaky.log'), date('H:i:s')." attempt {$this->attempts()}\n", FILE_APPEND);
        throw new \RuntimeException('payment gateway timeout');
    }

    public function failed(\Throwable $e): void
    {
        file_put_contents(storage_path('logs/flaky.log'), date('H:i:s')." failed() hook: {$e->getMessage()}\n", FILE_APPEND);
    }
}

Dispatched with App\Jobs\Flaky::dispatch() from a bootstrapped php -r, then php artisan queue:work --stop-when-empty --sleep=1 (database driver):

05:19:04 attempt 1
05:19:05 attempt 2
05:19:05 failed() hook: payment gateway timeout
RuntimeException: payment gateway timeout in /app/app/Jobs/Flaky.php:1
1 failed_jobs row

(Exception line cut at 70 characters.) The job stopped after two attempts, not four, and the second came one second after the first, matching the first backoff entry. maxExceptions won over tries, and failed() ran once, at the end. So set $maxExceptions below $tries when a job that throws every time should fail fast, and give $backoff an array so retries spread out. The right numbers depend on how long your upstream stays down.

Laravel queue retry_after: why a slow job runs twice

The docs warn that a job's timeout must be less than retry_after, "Otherwise, the job may be re-attempted before it has actually finished executing or timed out" (Max Job Attempts and Timeout). I tested the violation at the default retry_after of 90, on both drivers. SlowJob has $tries = 3 and sleeps SLOW_SECONDS=100:

class SlowJob implements ShouldQueue
{
    use Queueable;

    public $tries = 3;

    public function handle(): void
    {
        $log = storage_path('logs/'.env('SLOW_LOG', 'slow.log'));
        file_put_contents($log, date('H:i:s')." start attempt={$this->attempts()} pid=".getmypid()."\n", FILE_APPEND);
        sleep((int) env('SLOW_SECONDS', 6)); // "charge the card"
        file_put_contents($log, date('H:i:s')." end   attempt={$this->attempts()} pid=".getmypid()."\n", FILE_APPEND);
    }
}

Two workers, the second started one second after the first, with no --stop-when-empty:

php artisan queue:work --sleep=1   # worker 1
php artisan queue:work --sleep=1   # worker 2, started 1s later

--- database driver
05:19:28 start attempt=1 pid=26
05:20:58 start attempt=2 pid=30
05:21:08 end   attempt=1 pid=26
05:22:38 end   attempt=2 pid=30
--- redis driver
05:19:46 start attempt=1 pid=41
05:21:15 start attempt=2 pid=43
05:21:26 end   attempt=1 pid=41
05:22:45 start attempt=3 pid=41
05:22:55 end   attempt=2 pid=43

The same job ran twice at once on both drivers, overlapping for about 10 seconds, with no failure anywhere. The second start came 90 seconds after the first (91 on Redis), which is retry_after. On Redis the job even started a third time, once a worker was free. I did not investigate why the database run showed no third start. For a job that charges a card or sends an email, that's a duplicate.

The fix has two parts. Keep $timeout below retry_after, and have PCNTL installed, because the docs say it "must be installed in order to specify job timeouts". My container has none (no pcntl above), so a timeout would not have saved the job here. A worker that exits at its first empty poll, with --stop-when-empty, never picks up the second attempt; my first try used that flag on worker 2 and showed no double run.

ShouldBeUnique won't save you inside a batch: "Unique job constraints do not apply to jobs within batches" (Unique Jobs). Make the work idempotent, as the ledger guard above does.

Housekeeping: worker limits and pruning

queue:work --help on this install shows the worker limits:

--max-jobs[=MAX-JOBS]   The number of jobs to process before stopping [default: "0"]
--max-time[=MAX-TIME]   The maximum number of seconds the worker should run [default: "0"]
--timeout[=TIMEOUT]     The number of seconds a child process can run [default: "60"]

Both recycle limits default to 0, so workers never restart unless you set them (and then Supervisor or systemd must bring them back). Batch rows pile up too. The docs say: "Without pruning, the job_batches table can accumulate records very quickly" and show queue:prune-batches --hours=48, plus --unfinished=72 and --cancelled=72 for batches that never completed or were cancelled (Pruning Batches). I did not run the command myself. Schedule it, because Laravel doesn't. For graceful restarts on deploy, see our guide on graceful shutdown for Laravel queue workers.

I did not run Horizon or third-party retry packages such as kirschbaum-development/laravel-queue-batch-retry; everything above is core Laravel behaviour.

What to do next

1. Search for Bus::batch(. Independent chunks get allowFailures(); all-or-nothing work gets a ledger table and a fresh recovery batch.
2. If a job uses SkipIfBatchCancelled, never expect retry-batch to fix a cancelled batch.
3. Move "mark complete" logic out of finally into then, and stop reading progress() as done.
4. Find slow jobs and compare with retry_after: grep -rn "timeout" app/Jobs and grep -n "retry_after" config/queue.php. Every $timeout must be lower than the retry_after of the connection the job runs on. Confirm PCNTL with php -m | grep pcntl on the machine that runs workers.
5. After any retry, check the output (rows written), not the worker log.