What you'll learn

  1. Quick answer
  2. Scheduled or queued?
  3. Cron syntax
  4. The overlap problem
  5. Making jobs reliable
  6. Beyond plain cron
  7. FAQ

Quick Answer

Cron runs something on a schedule. A job queue runs something triggered by an event, as soon as a worker is free. Use cron for periodic maintenance and a queue for work generated by user actions.

Scheduled or queued?

The distinction people get wrong.

Cron is time-driven. "Every night at 2am, delete expired sessions." Nothing triggers it; the clock does. Good for cleanup, reports, backups, and pulling from an external system on a schedule.

A queue is event-driven. "When someone signs up, send a welcome email." A user action creates work, and a worker picks it up immediately — see message queues explained.

The common mistake is using cron for event-driven work: writing rows to a pending_emails table and running a cron job every minute to send them. That works, and it adds up to a minute of latency and re-invents a queue badly. If the work is caused by an event, use a queue.

Conversely, using a queue for periodic maintenance means something has to enqueue it — which is cron again, indirectly.

Cron syntax

Five fields: minute, hour, day of month, month, day of week.

# ┌ minute (0-59)
# │ ┌ hour (0-23)
# │ │ ┌ day of month (1-31)
# │ │ │ ┌ month (1-12)
# │ │ │ │ ┌ day of week (0-6, Sunday = 0)
# * * * * *  command

0 2 * * *      /usr/bin/python3 /app/cleanup.py     # 2am daily
*/15 * * * *   /app/sync.sh                          # every 15 minutes
0 9 * * 1      /app/weekly_report.sh                 # 9am Mondays
0 0 1 * *      /app/monthly_invoice.sh               # midnight, 1st of month

*/15 means every 15 units. Edit with crontab -e, list with crontab -l.

Three things that break real cron jobs:

  • Use absolute paths. Cron does not run in your shell and has a minimal PATH. python3 script.py frequently fails where /usr/bin/python3 /app/script.py works.
  • Your environment variables are not there. Anything sourced from .bashrc is absent. Set them explicitly or source a file inside the script.
  • Redirect the output. Without >> /var/log/job.log 2>&1, output goes to local mail nobody reads, and a failing job is invisible.

The overlap problem

A job scheduled every five minutes that occasionally takes seven minutes will start again while the previous run is still going. Two copies now process the same rows.

The consequences range from duplicated emails to corrupted data, and it only happens under load — so it passes every test and fails in production.

Prevent it with a lock. The simplest reliable approach is a lock file:

*/5 * * * * /usr/bin/flock -n /tmp/sync.lock /app/sync.sh

flock -n exits immediately if the lock is held, so the second run simply does not start.

For jobs running on several servers, a file lock is not enough — each machine has its own filesystem. Use a shared lock in Redis or a database row with a unique constraint, so only one server proceeds.

Making jobs reliable

  • Make them idempotent. Running twice should be harmless, because eventually one will run twice — after a retry, a manual re-run, or an overlap you did not prevent.
  • Process in batches with a cursor. A job handling everything since the beginning of time gets slower forever. Track what was processed and continue from there.
  • Alert on failure and on silence. Failure alerts catch crashes. They do not catch a job that stopped being scheduled at all — for that you need a heartbeat that alerts when a job has not run.
  • Log start and finish with a duration. A job creeping from 30 seconds to 4 minutes is a warning you can act on before it collides with its own schedule.
  • Think about time zones. Servers usually run UTC. "Midnight" in cron is server midnight, which is not your users' midnight, and daylight saving makes local-time schedules genuinely ambiguous twice a year.

Beyond plain cron

Cron is fine on a single server. It struggles once you have several, because it has no idea another machine exists — the same job runs everywhere unless you prevent it.

Options as things grow: systemd timers, which add logging and dependency handling on Linux; application-level schedulers such as Celery beat or a Node scheduler, which keep the schedule in your codebase under version control; managed cloud schedulers, which handle the multi-server problem by invoking one endpoint; and workflow orchestrators for pipelines with dependencies between steps.

Keeping the schedule in your repository rather than in a server's crontab is a real improvement — a crontab edited by hand on one machine is invisible to code review and lost when the machine is rebuilt.

For a student project, plain cron plus flock plus a log file covers everything, and understanding those three is enough to discuss the topic sensibly.

Frequently Asked Questions

What is the difference between cron and a job queue? Cron runs work on a time schedule. A queue runs work triggered by an event, as soon as a worker is available. Use cron for periodic maintenance and a queue for work created by user actions.
Why does my cron job work manually but not on schedule? Almost always the environment. Cron runs with a minimal PATH and none of your shell configuration, so use absolute paths and set required variables explicitly inside the script.
How do I stop two runs of the same job overlapping? Wrap the command in flock -n so a second run exits immediately while the first holds the lock. Across multiple servers, use a shared lock in Redis or a database instead.
How do I know if a cron job stopped running? Failure alerts do not catch a job that never started. Use a heartbeat check that alerts when a job has not reported success within its expected window.
Should the schedule live in the crontab or in my code? In your code where possible, so it is version controlled and reviewed. A hand-edited crontab on one server is invisible to your team and lost when the machine is replaced.