⚠️ Error Handling Strategy

Error Handling Strategy: Node API + BullMQ Service

Diagnosis

Three concrete issues making your Sentry noise + production cascades:

1. No error classification → every error treated the same → retry-everything anti-pattern → 12K Sentry items

2. OpenAI rate-limit cascade → retry with no backoff → makes throttling worse → bills increase + service degrades

3. Twilio infinite retry on permanent errors → invalid phone numbers retry forever → you discovered this only after one infinite loop

Highest-leverage fixes (in order):

1. Build the error class hierarchy. ~2 days. Eliminates string-match error handling. Foundation for everything else.

2. Implement classified retry policy. Retry only `RetriableError`. Permanent errors fail fast. Cuts Sentry noise 60%+ AND prevents the cascades.

3. Add circuit breakers on external APIs. Stops the OpenAI/Twilio cascades.

4. Tune Sentry: filter user-disconnects + bot traffic. Cuts another 30% of noise.

Estimated outcome: Sentry inbox drops from 12K to <500 actionable items. Cascades to OpenAI/Twilio stop. Real customer-impacting bugs become visible because they're not buried.

Error Class Hierarchy

// /lib/errors/index.ts

export class BaseError extends Error {
  public readonly correlationId: string;
  public readonly timestamp: Date;
  public readonly context: Record<string, unknown>;
  
  constructor(message: string, context: Record<string, unknown> = {}) {
    super(message);
    this.name = this.constructor.name;
    this.correlationId = generateCorrelationId();
    this.timestamp = new Date();
    this.context = context;
    Error.captureStackTrace?.(this, this.constructor);
  }
}

// Transient: retry might succeed (network, rate limits, transient DB errors)
export class RetriableError extends BaseError {
  public readonly retryAfter?: number; // ms
}
export class NetworkError extends RetriableError {}
export class RateLimitError extends RetriableError {}
export class TransientDatabaseError extends RetriableError {}
export class ExternalServiceUnavailable extends RetriableError {}

// Permanent: retry will not succeed
export class PermanentError extends BaseError {}
export class ValidationError extends PermanentError { /* 400 */ }
export class NotFoundError extends PermanentError { /* 404 */ }
export class UnauthorizedError extends PermanentError { /* 401 */ }
export class ForbiddenError extends PermanentError { /* 403 */ }
export class ConflictError extends PermanentError { /* 409 */ }

// Business logic: user violated a rule (often expected)
export class BusinessLogicError extends BaseError {
  public readonly userMessage: string; // safe to show user
}
export class InsufficientFundsError extends BusinessLogicError {}
export class QuotaExceededError extends BusinessLogicError {}
export class FeatureNotAvailableError extends BusinessLogicError {}

// Critical: needs immediate human attention
export class CriticalError extends BaseError {} 
export class DataIntegrityError extends CriticalError {}
export class SecurityError extends CriticalError {}

Why this hierarchy:

• instanceof RetriableError is your retry-or-not decision
• instanceof BusinessLogicError is your user-message-safe decision
• instanceof CriticalError is your page-on-call decision

Per-Class Handling Rules

Retry Policy

For RetriableError:

// /lib/retry.ts
export async function withRetry<T>(
  fn: () => Promise<T>,
  opts: {
    maxAttempts?: number;       // default 3
    baseDelayMs?: number;       // default 100
    maxDelayMs?: number;        // default 30000
    jitterFactor?: number;      // default 0.25
    timeoutMs?: number;         // default 30000
    circuitBreakerKey?: string; // for cross-call circuit breaker
  } = {}
): Promise<T> {
  const { maxAttempts = 3, baseDelayMs = 100, maxDelayMs = 30000, jitterFactor = 0.25 } = opts;
  
  // Check circuit breaker
  if (opts.circuitBreakerKey && circuitBreaker.isOpen(opts.circuitBreakerKey)) {
    throw new ExternalServiceUnavailable('Circuit breaker open');
  }
  
  let lastError: Error | undefined;
  
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      const result = await withTimeout(fn(), opts.timeoutMs ?? 30000);
      circuitBreaker.onSuccess(opts.circuitBreakerKey);
      return result;
    } catch (e) {
      lastError = e as Error;
      
      // Don't retry permanent errors
      if (!(e instanceof RetriableError)) throw e;
      
      // Track for circuit breaker
      circuitBreaker.onFailure(opts.circuitBreakerKey);
      
      // Last attempt: throw
      if (attempt === maxAttempts) throw e;
      
      // Calculate delay with jitter
      const delayMs = Math.min(baseDelayMs * Math.pow(2, attempt - 1), maxDelayMs);
      const jitter = delayMs * jitterFactor * (Math.random() * 2 - 1);
      const finalDelay = (e as RetriableError).retryAfter ?? Math.max(0, delayMs + jitter);
      
      logger.warn('retry.attempt', {
        error: e.message,
        attempt,
        nextDelayMs: finalDelay,
        correlationId: (e as BaseError).correlationId,
      });
      
      await sleep(finalDelay);
    }
  }
  
  throw lastError;
}

Defaults:

• Max attempts: 3 (most cases). 5 for critical operations like payment.
• Base delay: 100ms
• Max delay: 30s
• Jitter: ±25%
• Timeout per attempt: 30s

Circuit breaker:

• Opens after 5 consecutive failures within 60s
• Stays open for 60s
• Half-open: 1 test request after 60s; success closes, failure re-opens
• Tracked per circuitBreakerKey (e.g., 'openai-api', 'twilio-sms', 'internal-user-service')

Customer-Facing Error Messages

Response envelope:

// Error response shape (HTTP)
{
  "error": {
    "code": "validation_failed",      // machine-readable
    "message": "Email is required",   // user-friendly (BusinessLogicError + ValidationError only)
    "correlationId": "err_abc123xyz", // for support requests
    "details": {                       // optional, for ValidationError
      "field": "email",
      "constraint": "required"
    }
  }
}

Error code → HTTP status mapping:

For non-business errors: show ONLY the correlation ID. NEVER stack trace, NEVER internal field names.

// Wrong
{ error: { message: 'PostgresError: column "email" does not exist' } }

// Right
{ error: { code: 'internal_error', message: 'Something went wrong. Please try again or contact support with reference err_abc123xyz.', correlationId: 'err_abc123xyz' } }

Logging Strategy

Structured logging via Pino. Every log entry includes:

{
  timestamp: '2026-04-28T14:30:00Z',
  level: 'error',
  service: 'api',
  correlationId: 'err_abc123',
  userId: 'user_123' | null,    // never PII like email here, just ID
  requestId: 'req_xyz789',
  errorClass: 'RateLimitError',
  errorMessage: '...',
  context: { /* class-specific context */ }
}

Log levels:

• error: CriticalError, unexpected non-classified throws
• warn: RetriableError attempts, retry failures, circuit breaker state changes
• info: ValidationError, BusinessLogicError, expected operational events
• debug: not in production by default

Never log:

• Passwords, API keys, tokens
• Full credit card numbers (last 4 only)
• Email addresses (use user_id instead)
• PII unless legally required and audited

Always log:

• Correlation ID (for tracing)
• User ID (when applicable)
• Request ID (for tracing across logs)
• Error class name (so log queries can filter)

Alert Strategy

Page on-call (PagerDuty):

• Any CriticalError
• Error rate >2% sustained for 5 min
• p99 latency >5s sustained for 5 min
• Circuit breaker open for >2 min on critical services
• Database connection pool exhausted

Slack #alerts (no page):

• Error rate >0.5% sustained for 5 min
• New error type appearing (haven't seen this BaseError subclass before)
• Circuit breaker opens (any service)
• DLQ depth >100 jobs

Daily digest (Slack):

• Top 5 error classes by volume
• Top 3 endpoints by error rate
• Sentry trends week-over-week

Sentry tuning to drop noise:

# Sentry config — ignore these noisy patterns
ignoreErrors: [
  /^AbortError/,                // user disconnect
  /^ECONNRESET/,                 // network blip
  /^Request failed.*timeout/,    // already retried, don't double-log
  /^ValidationError.*email/,     // expected user input errors
]

beforeSend(event) {
  // Don't send 4xx errors to Sentry — they're not bugs
  if (event.tags?.statusCode?.startsWith('4')) return null;
  // Drop bot traffic
  if (isBotUserAgent(event.request?.headers?.['user-agent'])) return null;
  return event;
}

External API Patterns

OpenAI

async function callOpenAI<T>(prompt: string): Promise<T> {
  return withRetry(
    async () => {
      try {
        return await openai.chat.completions.create({ /* ... */ });
      } catch (e: any) {
        if (e.status === 429) {
          throw new RateLimitError('OpenAI rate limit', { retryAfter: parseRetryAfter(e) });
        }
        if (e.status >= 500) {
          throw new ExternalServiceUnavailable('OpenAI 5xx', { originalError: e });
        }
        if (e.status === 400) {
          throw new ValidationError('OpenAI bad request', { originalError: e });
        }
        throw e;
      }
    },
    { maxAttempts: 3, baseDelayMs: 1000, circuitBreakerKey: 'openai-api' }
  );
}

Twilio SMS

async function sendSMS(to: string, body: string) {
  return withRetry(
    async () => {
      try {
        return await twilio.messages.create({ to, body });
      } catch (e: any) {
        // Twilio error codes — see https://www.twilio.com/docs/api/errors
        if (e.code === 21211 || e.code === 21408) {
          throw new ValidationError('Invalid phone number', { code: e.code });
        }
        if (e.code === 20429) {
          throw new RateLimitError('Twilio rate limit');
        }
        if (e.status >= 500) {
          throw new ExternalServiceUnavailable('Twilio 5xx');
        }
        throw e;
      }
    },
    { maxAttempts: 3, circuitBreakerKey: 'twilio-sms' }
  );
}

Critical: Twilio's invalid-phone errors (21211, 21408) are PERMANENT. Throwing them as ValidationError stops the infinite retry you experienced.

Stripe (idempotency required)

async function createPayment(amount: number, customerId: string, idempotencyKey: string) {
  return withRetry(
    async () => {
      try {
        return await stripe.paymentIntents.create({ /* ... */ }, { idempotencyKey });
      } catch (e: any) {
        if (e.type === 'StripeRateLimitError') throw new RateLimitError('Stripe rate limit');
        if (e.type === 'StripeConnectionError') throw new NetworkError('Stripe connection');
        if (e.type === 'StripeCardError') throw new BusinessLogicError(e.message, { userMessage: e.message });
        throw e;
      }
    },
    { maxAttempts: 5, baseDelayMs: 200, circuitBreakerKey: 'stripe-api' }
  );
}

Idempotency key: SHA256(userId + orderId + timestamp_minute) — same key within a minute = Stripe deduplicates. Critical for payment retries.

Dead Letter Queue (BullMQ)

// BullMQ job processor
worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    // Move to DLQ
    await deadLetterQueue.add('failed-job', {
      jobName: job.name,
      data: job.data,
      lastError: serializeError(err),
      attempts: job.attemptsMade,
      failedAt: new Date(),
    });
    
    if (err instanceof CriticalError) {
      // Page on-call
      await pagerduty.alert(...)
    }
  }
});

DLQ inspection tool (admin endpoint):

• List failed jobs by date / job-type / error class
• View job data
• Re-enqueue specific job
• Bulk re-enqueue (after fixing root cause)
• Delete (after confirming not needed)

Idempotency Keys

For any write operation that might retry:

// Store idempotency keys in Redis with 24h TTL
// Key: hash(operation_type + user_id + business_key)
// Value: { result: ..., cachedAt: Date }

async function withIdempotency<T>(
  key: string,
  fn: () => Promise<T>
): Promise<T> {
  const cached = await redis.get(`idem:${key}`);
  if (cached) {
    return JSON.parse(cached).result;
  }
  
  const result = await fn();
  
  await redis.setEx(`idem:${key}`, 86400, JSON.stringify({ result, cachedAt: new Date() }));
  
  return result;
}

Use for: payments, email sends, external API calls that mutate state, webhook processing.

Implementation Skeleton

/lib/errors/
  index.ts           (BaseError + class hierarchy)
  middleware.ts      (Fastify error handler)
  serialize.ts       (error → response envelope)

/lib/retry/
  with-retry.ts      (retry wrapper)
  circuit-breaker.ts (circuit breaker state machine)
  with-timeout.ts    (timeout wrapper)

/lib/idempotency/
  with-idempotency.ts

/lib/external/
  openai.ts          (typed OpenAI calls with error mapping)
  twilio.ts
  stripe.ts
  postmark.ts
  internal-user-service.ts

/workers/
  base-worker.ts     (BullMQ worker with error handling baked in)
  dlq-replay.ts      (admin tool)

/observability/
  sentry-config.ts   (filter rules)
  logger.ts          (Pino setup)
  metrics.ts         (Datadog StatsD client)

Fastify error handler middleware:

app.setErrorHandler((error, request, reply) => {
  const correlationId = (error as any).correlationId ?? generateCorrelationId();
  
  // Log
  if (error instanceof CriticalError) {
    logger.error({ error, correlationId, request: requestContext(request) });
  } else if (error instanceof RetriableError || error instanceof PermanentError) {
    logger.warn({ error, correlationId });
  } else {
    // Unexpected error — log as error + alert
    logger.error({ error, correlationId, type: 'unexpected' });
  }
  
  // Map to HTTP
  const { statusCode, body } = mapErrorToResponse(error, correlationId);
  reply.code(statusCode).send(body);
});

Migration from Current State

Week 1: Foundation

• Build error class hierarchy
• Build withRetry + circuit breaker
• Add Fastify error middleware
• Update Sentry config to filter known noise

Week 2: Replace high-traffic paths

• Refactor OpenAI call sites to use new pattern (highest cascade risk)
• Refactor Twilio (stops infinite retries)
• Refactor Stripe with idempotency

Week 3: Replace remaining external calls

• Postmark, internal-user-service, etc.
• Update queue workers to use base-worker pattern with DLQ

Week 4: Tune

• Review Sentry: closed/ignore noise issues
• Verify alert thresholds match team capacity
• Document patterns for new code

Ongoing

• New code uses the patterns by default
• Old try-catches replaced opportunistically
• Audit quarterly

What This Strategy Won't Solve

• Won't fix poor input validation upstream. ValidationErrors are the right output of bad input; preventing bad input is a frontend/API design issue.
• Won't catch logical bugs. A function that returns wrong-but-valid data isn't an error from this framework's perspective. Tests catch logic bugs.
• Won't eliminate Sentry items entirely. New error classes will appear; the goal is signal-to-noise, not zero.
• Won't compensate for an undersized infrastructure. If your DB is sized too small, you'll see RetriableError storms during traffic spikes. The error handling makes failures graceful but doesn't prevent them.
• Won't replace runbooks for incident response. When circuit breakers open, on-call still needs to know what to do.

Maintenance Cadence

Per-PR (engineer):

• New external API call sites use withRetry + appropriate circuit breaker key
• New error throws use specific error classes, not generic Error

Weekly:

• Review Sentry top errors. New patterns? Drop noise rules where possible.
• Check circuit breaker state in Datadog dashboard. Anything stuck open?

Monthly:

• Audit alert tuning: any pages that aren't actionable? Adjust thresholds.
• DLQ review: any patterns? Root-cause + replay or delete.
• Error budget burn rate: are we trending toward SLA breach?

Quarterly:

• Add new error classes if patterns emerge
• Review external service tooling: better way to call OpenAI / Twilio?
• Postmortem on any production incident that escaped the framework

Key Takeaways

• Three error classes (Retriable, Permanent, BusinessLogic) drive all decisions. Hierarchy is the foundation.
• Retry only Retriable. Twilio's invalid-number is Permanent — that's why your infinite retry happened.
• Circuit breaker on external APIs prevents cascades. OpenAI rate-limit storms stop being self-amplifying.
• Customer-facing errors: code + safe message + correlation ID. Never stack traces.
• Sentry is for actionable bugs. Filter user-disconnects + ValidationErrors. Tune to <500 actionable items, not 12K noise.
• Migration is 4 weeks. Most pain disappears in week 2 after OpenAI + Twilio refactor.