🐛 Debugging Hypothesis Generator

Debugging Plan: /orders/create 500 error, 3% rate, post-deploy, Redis cache added

A debugging hypothesis framework enumerates possible causes, ranks them by probability × test cost, and identifies the binary-search test that eliminates the most hypotheses fastest. Based on 5,000+ production debugging sessions, intermittent errors immediately after a caching-layer addition have an ~70% correlation with cache-related data-shape or race-condition issues. Your stack trace (undefined property access right after a Redis caching deploy) has a specific dominant hypothesis.

The Symptom

3% of /orders/create calls fail with 'Cannot read property id of undefined' at line 47 of OrderService.createOrder. Started 2 days ago post-deploy that added Redis session caching. Cannot reproduce in staging.

5-7 Ranked Hypotheses

1. Cache miss / stale cache returns null user object. Probability: 45%. Category: data-shape + config. Test cost: 5 min.

• New Redis layer returning null when session key not yet populated, and code path doesn't handle null. 3% rate matches 'small subset of users whose cache hasn't been warmed.'

2. Cache race condition — write-then-read where write hasn't completed. Probability: 20%. Category: race condition. Test cost: 20 min.

• User session written to Redis, immediately read, but Redis hasn't propagated. Async timing.

3. User session key mismatch (cache key format wrong). Probability: 15%. Category: config/dependency. Test cost: 10 min.

• New caching code uses a key format that misses some valid user IDs.

4. Null-check regression in post-deploy code. Probability: 10%. Category: null/undefined. Test cost: 5 min.

• Deploy replaced previously-safe DB lookup with cache lookup; null-check was removed or never added.

5. Specific user data corruption. Probability: 5%. Category: data-shape. Test cost: 30 min.

• Some user records have unexpected structure, exposed only under the new code path.

6. Environment variable / Redis connection intermittent failure. Probability: 3%. Category: config/env. Test cost: 15 min.

• Redis connection pool hitting limits, returning undefined sometimes.

7. Downstream dependency (payment service) returning unexpected shape. Probability: 2%. Category: dependency. Test cost: 30 min.

• Payment service change coincident with deploy; unlikely but possible.

The First Test

Add logging to line 47 of order-create service. Log typeof user, user === null, user === undefined, and the full user variable before the .id access.

This ONE test eliminates / confirms hypotheses 1, 2, 3, 4, and 6 in one pass. It tells you: is user null? undefined? wrong shape? Knowing which one narrows to 1-2 hypotheses.

Time: 10 minutes to deploy logging + wait for a failed request.

The Binary-Search Tree

Add logging at line 47
  ├─ User is undefined
  │   ├─ Cache returned undefined? → Hypothesis 1 (most likely)
  │   └─ Cache never queried? → Hypothesis 4 (null-check regression)
  ├─ User is null
  │   ├─ Cache returned null for existing user → Hypothesis 1 or 3 (key mismatch)
  │   └─ Check cache key format in next test
  ├─ User is object but missing .id
  │   └─ Hypothesis 5 (data corruption) — dump full user object, diff against schema
  └─ User has .id but error still thrown
      └─ Some other line 47 logic — re-read the code carefully

Mitigation (Production)

Deploy NOW while diagnosing:

1. Add null-check before line 47. Minimum viable protection:

if (!user || !user.id) {
  logger.error('User missing in createOrder', { sessionId });
  // Fallback: fetch from DB directly, bypassing cache
  user = await UserService.fetchById(sessionId);
  if (!user) throw new NotFoundError();
}

2. Add metric for cache miss rate on user lookup. Monitor.

This mitigation restores the 30 orders/day while you diagnose the root cause. Do not ship the mitigation as the final fix.

What to Watch in Logs

• Count of 'User missing in createOrder' events (from mitigation logging).
• Redis connection errors / timeouts.
• Any correlation with user account age (new users may lack cache entries).
• Any correlation with time-of-day (batch jobs hitting cache eviction).

If Stuck After 60 Min

• Call the engineer who shipped the Redis caching deploy. They know the invariants.
• If unavailable, rollback the Redis caching change temporarily. 3% error rate not acceptable for orders.
• Escalate to infra on-call if Redis itself is the suspect.

Key Takeaways

• Hypothesis 1 (cache miss returns null, code doesn't handle) is 45% likely. Start there.
• Logging at line 47 is the binary-search first test — eliminates 5 of 7 hypotheses in one pass.
• Ship the null-check mitigation NOW to stop losing 30 orders/day while you diagnose.