⚙️ Background Job Queue Designer

Queue Architecture: B2B SaaS Multi-Job-Type System

Queue Infrastructure Choice

BullMQ on dedicated Redis (Upstash + Fly.io workers). Reasons:

• Team has BullMQ experience; switching costs are real
• Volume (~20K/day, ~100K target) is well within BullMQ capability
• Redis already in stack; no new infrastructure
• Move workers off Vercel onto Fly.io / Railway / dedicated long-running infra

Alternatives considered:

• AWS SQS: would work, but you don't have AWS-native infra; adds vendor lock-in
• Inngest: great for workflow orchestration but overkill for your job types (mostly single-step, not multi-step workflows)
• Temporal: great for complex workflows, but learning curve + complexity not justified for your job mix

Reconsider migration to Inngest or Temporal IF you start needing complex multi-step workflows (saga pattern, long-running approvals, complex state machines). Right now: BullMQ.

Queue Inventory

Queue                  Priority    Latency Target   Avg Throughput
─────────────────────────────────────────────────────────────────
user-fast              HIGH        <10s             ~3K/day
  (image thumbnails, password reset emails)

user-medium            MEDIUM      <60s             ~6K/day
  (transactional emails, Stripe webhook processing)

outbound-webhooks      MEDIUM      <60s             ~10K/day
  (delivery to customer webhook URLs — separate due to potentially-slow customer endpoints)

batch-fast             LOW         <15min           ~100/day
  (analytics sync, small reports)

batch-slow             LOW         <24h             ~500-2K/day spike
  (monthly customer reports, end-of-month bulk)

DLQ                    n/a         n/a              accumulator

Why these splits:

• user-fast keeps user-waiting jobs out of the same lane as long-running reports
• outbound-webhooks separated because customer endpoints can be slow (10s timeout) — don't block other jobs
• batch-slow handles month-end spikes without affecting daily ops

Per-Job-Type Specification

Transactional emails

• Queue: user-fast for password resets (user is waiting); user-medium for receipts (less urgent)
• Idempotency key: email:${userId}:${templateId}:${eventId} — same key = same email, deduped
• Retry policy: 3 attempts, 1min/5min/15min backoff
• Failure: DLQ; daily review by support team

Image thumbnails

• Queue: user-fast
• Idempotency key: thumb:${imageId}:${size} — keyed by image + size variant
• Retry policy: 2 attempts, immediate then 30s backoff
• Failure: DLQ; user can retrigger by re-uploading
• Note: CPU-bound — limit concurrency per worker (1-2 per CPU)

PDF reports (monthly bulk)

• Queue: batch-slow
• Idempotency key: report:${customerId}:${period} — once per customer-period
• Retry policy: 5 attempts, 5min/15min/30min/1h/2h backoff
• Failure: DLQ; alert support
• Note: memory-heavy; size workers accordingly (~512MB each)

Webhook delivery (to customer URLs)

• Queue: outbound-webhooks
• Idempotency key: webhook:${eventId} — enforced by us, not by customer
• Retry policy: 6 attempts over 24h: 1m, 5m, 15m, 1h, 4h, 24h backoff
• Failure: DLQ + customer notification email + UI shows 'webhook failed'
• Timeout: 10s per delivery attempt (customer endpoints may be slow)
• Note: customer-facing — clear error reporting matters

Analytics sync

• Queue: batch-fast
• Idempotency key: analytics-sync:${batchTimestamp} — one per 15-min window
• Retry policy: 3 attempts
• Failure: DLQ + alert; reconcile with next batch
• Schedule: cron at :00, :15, :30, :45 of each hour

Stripe webhook processing

• Queue: user-medium
• Idempotency key: stripe:${eventId} — Stripe's event.id
• Retry policy: 5 attempts, exponential 5s/25s/125s/625s/52min
• Failure: DLQ + alert (financial data — high priority)
• Note: event handlers idempotent at DB level (UPSERT, unique constraint)

Priority Strategy

BullMQ supports priority within a queue + multiple queues. Use both:

Across queues: workers process queues in declared order. user-fast workers checked first, then user-medium, etc.

Within queue: for user-fast and user-medium, use BullMQ's priority option — password reset (priority 1) before receipt (priority 5).

await emailQueue.add('send-email', { ... }, { priority: 1 }); // password reset
await emailQueue.add('send-email', { ... }, { priority: 5 }); // receipt

Lower number = higher priority.

Worker Pool Sizing

Per queue:

• user-fast: 2 workers, concurrency 10 each (IO + light CPU). Total: 20 jobs concurrent.
• user-medium: 2 workers, concurrency 20 each (IO-heavy). Total: 40 concurrent.
• outbound-webhooks: 1 worker, concurrency 50 (highly IO-bound). Total: 50 concurrent.
• batch-fast: 1 worker, concurrency 5 (IO + some CPU). Total: 5 concurrent.
• batch-slow: 2 workers, concurrency 1 each (CPU + memory heavy). Total: 2 concurrent.

Reasoning:

• IO-bound jobs benefit from high concurrency (Node async)
• CPU-bound jobs limit to ~1 per CPU core to avoid contention
• Memory-heavy jobs: low concurrency, dedicated workers

Total worker resources:

• 5 small workers (1 vCPU, 512MB) for fast queues
• 2 medium workers (2 vCPU, 1GB) for batch-slow
• ~$50-80/month on Fly.io / Railway

Retry & DLQ

Backoff strategy (exponential + jitter):

await queue.add('job-name', data, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 5000, // 5s base
  },
  // BullMQ adds jitter automatically with `removeDelay` randomization
});

DLQ pattern:

• BullMQ's failed job state is your DLQ
• Don't auto-delete failed jobs
• Build admin tool to inspect + replay

Admin DLQ tool:

// /admin/dlq
GET /admin/dlq?queue=user-fast&since=2026-04-25
// Returns: list of failed jobs with id, name, lastError, failedAt

POST /admin/dlq/replay
  body: { jobId: 'abc123' }
// Re-enqueues to original queue

POST /admin/dlq/replay-bulk
  body: { queue: 'user-fast', failedSince: '2026-04-28', errorPattern: 'TimeoutError' }
// Bulk replay matching pattern

DELETE /admin/dlq/:jobId
// Permanently delete

Alerts:

• DLQ size >0 for 'must-not-lose' queues (emails, reports, webhooks, Stripe) → page within 30 min
• DLQ size >50 for any queue → Slack alert
• DLQ growth rate >10/hour → Slack alert (something systemic)
• Oldest-failed-job-age >24h → daily digest

Idempotency Pattern

Key strategy by job type listed above. Critical patterns:

// Use BullMQ's jobId for queue-level dedup
await queue.add('job-name', data, {
  jobId: idempotencyKey,  // BullMQ rejects duplicate jobIds
});

// Defense in depth: handler also idempotent at DB layer
async function processEmail(jobData) {
  const dedupKey = `email:${jobData.userId}:${jobData.templateId}:${jobData.eventId}`;
  const alreadySent = await db.email_sent.findUnique({ where: { dedupKey } });
  if (alreadySent) return { status: 'duplicate' };
  
  // Send email
  const result = await sendgrid.send(...);
  
  // Record success (transaction with the actual side effect)
  await db.email_sent.create({ dedupKey, sendgridId: result.id });
  
  return { status: 'sent' };
}

Observability

Per queue, track:

• queue.depth (gauge: pending jobs)
• queue.age.oldest (gauge: how stale is the oldest pending job)
• queue.processed.rate (counter: completed/sec)
• queue.failed.rate (counter: failed/sec)
• queue.duration (histogram: job processing time)
• queue.wait.duration (histogram: time from enqueue to start)

Alert rules:

• queue.age.oldest for user-fast >30s → page (user-waiting backlog)
• queue.depth for user-medium >500 sustained 10min → Slack (capacity issue)
• queue.failed.rate >5% → Slack (something's wrong)
• queue.depth for batch-slow >2000 during month-end → expected, no alert
• Worker process not consuming for >5 min → page (worker died)

Dashboard:

• Per-queue depth + age over last 24h
• Top 5 failing job types
• DLQ size trends
• Worker resource utilization

Implementation Skeleton

/services/queue/
  src/
    queues.ts             (queue configurations + connections)
    types.ts              (job data type definitions)
  package.json

/services/worker/
  src/
    workers/
      user-fast-worker.ts
      user-medium-worker.ts
      outbound-webhooks-worker.ts
      batch-fast-worker.ts
      batch-slow-worker.ts
    handlers/
      send-email.ts
      generate-thumbnail.ts
      generate-pdf-report.ts
      deliver-webhook.ts
      sync-analytics.ts
      process-stripe-event.ts
    metrics.ts
    server.ts             (worker process bootstrap)
  Dockerfile
  fly.toml

/services/api/  (your existing)
  uses queue.ts to ENQUEUE jobs
  doesn't process them

Worker bootstrap pattern:

// /services/worker/src/server.ts
import { Worker } from 'bullmq';
import { redis } from './redis';
import { handleEmail } from './handlers/send-email';
import { handleThumbnail } from './handlers/generate-thumbnail';
// ... other handlers

const userFastWorker = new Worker(
  'user-fast',
  async (job) => {
    switch (job.name) {
      case 'send-email': return handleEmail(job.data);
      case 'generate-thumbnail': return handleThumbnail(job.data);
      default: throw new Error(`Unknown job ${job.name}`);
    }
  },
  {
    connection: redis,
    concurrency: 10,
  }
);

// ... similar for other queues

userFastWorker.on('failed', (job, err) => {
  logger.error({ jobId: job?.id, error: err }, 'Job failed');
  metrics.increment('queue.failed', { queue: 'user-fast', name: job?.name });
});

userFastWorker.on('completed', (job, result) => {
  metrics.increment('queue.completed', { queue: 'user-fast', name: job.name });
});

Scaling Boundaries

This architecture handles:

• Up to ~500K jobs/day comfortably
• Up to ~2K/sec sustained
• 5x your current peak month-end volume

Migration triggers (>3y from now):

• Total volume >1M jobs/day → consider sharding queues across Redis Cluster
• Need workflow orchestration (multi-step jobs with state) → migrate to Inngest or Temporal
• Need exactly-once semantics (BullMQ is at-least-once) → migrate to Temporal
• Cross-region replication needs → managed services (AWS SQS, Google Pub/Sub)

What This Architecture Won't Solve

• Won't handle exactly-once delivery. BullMQ is at-least-once; idempotency at handler is your defense.
• Won't enforce ordering across queues. Within a queue, FIFO. Across queues, no ordering.
• Won't replace workflow engines. Long-running multi-step (with checkpoints, compensations) needs Inngest/Temporal.
• Won't compensate for poor handler design. If handlers aren't idempotent, retries cause data corruption — queue can't fix.
• Won't auto-scale workers. You need to monitor + manually scale (or set up auto-scaling separately).

Migration from Existing

Week 1: Infrastructure

• Set up dedicated worker service on Fly.io / Railway
• BullMQ queues defined
• Move existing image thumbnail BullMQ to new structure

Week 2: Email migration

• Replace synchronous Sendgrid with email queue + worker
• Idempotency keys at enqueue
• Test in staging — verify no email duplicates

Week 3: Stripe webhook migration

• Build outbound-webhooks queue
• Move Stripe webhook processing to user-medium queue
• Receiver does sig verify + enqueue (Receiver pattern from webhook-handler-architect)

Week 4: Reports + analytics

• Move PDF report generation to batch-slow
• Move analytics sync to batch-fast cron schedule

Week 5: Webhook delivery to customers

• Build outbound-webhooks queue with retry
• Migrate from no-retry direct send
• DLQ + customer notification on permanent failure

Week 6: Observability + DLQ tools

• Datadog dashboard
• Admin DLQ replay tool
• Alert configuration

Maintenance Cadence

Weekly:

• DLQ review per queue. Patterns?
• Queue depth + age trends — anything degrading?

Monthly:

• Worker resource utilization. Time to scale up/down?
• Job duration trends — anything getting slower?
• Cost review (Redis + worker hosts)

Quarterly:

• Architecture audit. New job types added? Queue partitioning still right?
• Scale test: replay 3 days of historical jobs to validate capacity

Key Takeaways

• 5 queues by job class (priority + latency), not by domain. user-fast / user-medium / outbound-webhooks / batch-fast / batch-slow.
• BullMQ for your scale + team. Inngest/Temporal only when complex workflows justify.
• Move workers off Vercel to dedicated infra. Serverless is wrong for long-running queue workers.
• Idempotency keys at job creation + DB-level dedup at handler. Defense in depth.
• DLQ inspection tool from day one. Build admin replay UI; don't wait until you need it at 3am.
• Migrate in 6 weeks. Email first (highest current pain), then Stripe webhooks, then reports + analytics + outbound webhooks.