The dashboard was green. The jobs weren't running.

The menu availability was stale. Not broken — stale. A store would flip closed and customers could still add its items for a few minutes. The cache that drives availability was supposed to refresh every minute, and Bull Board — the BullMQ dashboard — showed the refresh job completing, on schedule, green across the board. The jobs were running. They just weren’t doing anything.

One queue, seven owners

We run background work on BullMQ through NestJS (@nestjs/bullmq). Jobs are grouped into queues; we had a CLEANUP queue for the housekeeping — expired-token sweeps, webhook-log pruning, API-key usage rollups, and the important one: the catalog availability cache refresh.

Each kind of work had its own @Processor class, and each guarded the jobs it cared about:

@Processor(QueueType.CLEANUP)
export class MenuCacheProcessor extends WorkerHost {
  async process(job: Job) {
    if (job.name !== 'REFRESH_MENU_STATUS_CACHE') return; // not mine — skip
    return this.menuCache.refresh();
  }
}

Clean, obvious, the way you’d reasonably structure it. Seven classes, one CLEANUP queue. That was the bug.

What "competing consumers" actually means

Here’s the part the decorator hides. @nestjs/bullmq creates one BullMQ Worker per @Processor class — I confirmed it in node_modules/@nestjs/bullmq/dist/bull.explorer.js: it loops over processors and calls new Worker(queueName, processor, …) once each, with no job-name filtering at the worker level.

So with seven @Processor(CLEANUP) classes, the queue had seven workers all competing for the same jobs. When a job is enqueued:

1. BullMQ hands it to one worker — whichever idle worker wins the atomic moveToActive race. Effectively random.
2. That worker runs its process(). If the job isn’t the one it owns, the if (job.name !== …) return guard fires and returns undefined.
3. BullMQ sees a resolved promise and marks the job completed successfully — no error, no retry, no redelivery to another worker.

A job for a queue with N competing workers runs only about 1/N of the time. With seven workers, the menu-cache refresh landed on the right one roughly one time in seven.

Why nothing alarmed

A completed job and a skipped job are the same shape. Your dashboard can’t tell them apart — only the work can.

There was no error to catch, no failed job to retry, no dead-letter entry. process() returned undefined and BullMQ did exactly what you tell it to: marked it done. Bull Board was green because the jobs were running — they were just immediately deciding the work wasn’t theirs.

The only honest signal was downstream and indirect: availability refreshed about seven times slower than configured — a once-a-minute job behaving like once-every-seven-minutes. (It also quietly co-starred in a read-replica CPU incident, but that’s another post.)

The audit

First move: find the blast radius — count @Processor classes per queue.

grep -rhoP "@Processor\(\s*QueueType\.\K[A-Z_]+" src | sort | uniq -c

Fourteen other queues already did it right — one worker, multiple job names routed internally. Only CLEANUP had grown extra @Processor classes over time. The invariant we’d violated, stated plainly:

The fix: one dispatcher, a registry of handlers

One worker per queue that routes by name. To avoid a god-module that imports every feature, handlers self-register into a small runtime registry on init:

@Injectable()
export class CleanupRegistry {
  private readonly map = new Map<string, CleanupHandler>();
  register(h: CleanupHandler) {
    for (const name of h.jobNames) {
      if (this.map.has(name)) throw new Error(`Duplicate cleanup handler for ${name}`);
      this.map.set(name, h);
    }
  }
  get(name: string) { return this.map.get(name); }
}

@Processor(QueueType.CLEANUP) // the ONLY one
export class CleanupProcessor extends WorkerHost {
  constructor(private readonly registry: CleanupRegistry) { super(); }
  async process(job: Job) {
    const handler = this.registry.get(job.name);
    if (!handler) {
      Logger.warn({ message: 'No cleanup handler for job', jobName: job.name });
      return;
    }
    return handler.run(job);
  }
}

Each old processor becomes a handler: drop @Processor and the job.name guard, move the body into run(), and register(this) on init. The schedulers don’t change — they still enqueue by job name.

Two properties I care about: there’s now exactly one consumer, so there’s no race to lose; and the registry throws on a duplicate job name at startup — the old failure mode (a silent skip) becomes a loud crash on boot.

Making it stay fixed

A fix that one stray @Processor can silently undo isn’t a fix. So the invariant is now a CI guard — the same grep, failing the build if any queue has more than one processor:

# fails CI if any QueueType has >1 @Processor
grep -rhoP "@Processor\(\s*QueueType\.\K[A-Z_]+" src | sort | uniq -d

What I took from it

Competing consumers is a textbook messaging pattern — I just met it through a decorator that turned one reasonable refactor into six invisible no-ops. The textbook version has the workers racing to do the same job. The cruel version has them silently agreeing not to.