What reliable background jobs taught me about system design

The job is not the work

A background job is usually where a system tries to hide complexity: emails, exports, payment follow-ups, and anything that should happen after the user has moved on. The useful mental model is simpler: the job is a record of work that still needs to be made true.

Make retries boring

A retry should be safe enough that nobody has to guess whether it will charge a card twice, send another message, or overwrite a result. That means choosing an idempotency key before choosing a queue library.

Leave a trail

The most valuable part of a job system is often its visibility. A clear status, a useful error, and a link back to the request that created it make production support much less mysterious.

Start with the failure mode

Before adding a worker, write down what happens if it runs twice, runs late, or never runs. The answer is usually a better design than the first implementation.