The Problem: A 7-Hour Git Repack Job

At a company (referred to as $WORK), background jobs pack "reference repos" — aggressively compressed git repositories stored in object storage. Two repacking strategies exist:

  • Wholesale repacking: Recomputes everything from scratch, taking 7 hours. Produces a repo 50-60% smaller.
  • Incremental repacking: Reuses baseline packing, taking 2 hours. Produces a larger repo but is more up-to-date.

Downstream consumers are more active on weekdays, so the ideal plan: run incremental repacks on weekdays (fast, current) and wholesale repacks on weekends (thorough, smaller).

The Naive Approach: Set Interval to 3 Hours

The job queue exposes two knobs: scheduling interval and concurrency limit. Initially, interval=9 hours (headroom for 7h job) and concurrency=1.

The natural idea: set interval to 3 hours. On weekdays, the 2-hour job finishes before the next trigger, so no overlap. On weekends, the 7-hour job will still be running when the next trigger fires, but concurrency=1 prevents a second job from starting — so it should be fine, right?

Wrong. The behavior depends on what the queue does when a new job is triggered while one is already running. There are four possible semantics:

  1. Parallel Spawn – Start the new job if concurrency limit allows.
  2. Prefer New – Cancel the old job and start the new one.
  3. Wait – Queue the new job to start when the old one finishes.
  4. Prefer Old – Cancel the new job and let the old one continue.

Why the Naive Approach Fails

With Prefer New (common in many job queues), the 7-hour weekend job gets canceled every 3 hours. A new 7-hour job starts, only to be canceled again. Over 48 hours, that's 16 wasted jobs — none finish, wasting 48+ hours of compute.

With Wait, the queue would accumulate pending jobs. With interval=3h and job duration=7h, the queue grows unboundedly. Even with a limit, you'd need to decide what happens when the limit is hit (drop oldest? newest?).

With Prefer Old, the weekend job runs to completion, but the 3-hour interval means new jobs are constantly canceled — wasting resources on scheduling overhead.

The Lenses: Queues, Limits, Fault Models

The author applies three lenses:

  1. Queues are tricky: They tend to be nearly full or nearly empty. Latency-focused queue theory often ignores throughput, which matters for batch jobs.

  2. Explicit limits: Inspired by TigerBeetle's Tiger Style guide. The Wait semantics require a bounded per-config queue. What's the limit? What happens when it's hit? These questions are often overlooked.

  3. Fault models: Each semantic makes different assumptions:

    • Prefer New: Assumes the long job hit a transient issue (e.g., rate limit). Optimistic that the new job will succeed faster.
    • Wait: Assumes dropping jobs is unacceptable. Requires queue management.
    • Prefer Old: Assumes the running job is not hung and will finish. Pessimistic about new jobs also being slow.

Real-World Example: Git Repack Config

Suppose the queue offers a scheduling_interval of 3 hours and concurrency_limit of 1. Here's how each semantic plays out over a weekend (48 hours):

  • Prefer New: 16 failed jobs, 0 completions.
  • Wait: Queue grows to ~16 pending jobs (assuming no limit). With a limit of 2, the 3rd job onward gets dropped or rejected.
  • Prefer Old: 1 successful job (7h), 15 canceled scheduling attempts.

The Right Approach: Separate Configs or Custom Scheduling

Since typical job queues lack control over these semantics, the author suggests workarounds:

  • Run two separate job configurations: one for weekdays (interval=3h, incremental) and one for weekends (interval=8h, wholesale).
  • Use a custom scheduler that understands the desired behavior (e.g., cron with conditional logic).
  • If the queue supports it, set the scheduling interval to match the longest expected runtime (8h) and accept less frequent updates on weekdays.

Key Takeaways

  • Job queue semantics matter more than most developers assume.
  • Always check what happens when a job overlaps with the next scheduled trigger.
  • Explicitly model fault scenarios: Is the job hung? Transiently slow? Always slow?
  • Consider using separate configurations for different job profiles rather than overloading one.

Further Reading

  • Marc Brooker's blog on queue behavior
  • TigerBeetle's Tiger Style guide on limits
  • Alex Miller's writing on fault models