🔍 Flaky Test Diagnoser

Flaky Test Diagnosis: 'processes order and updates inventory'

Pattern Identification

This is shared-state flakiness with a parallel-execution amplifier.

Not the other 6 patterns:

• Not timing/race within a single test (the test itself is sequential awaited)
• Not real-network (using local DB)
• Not async race within the order processor (would manifest differently)
• Not environment-dependence (same code path)
• Not time-dependence (no Date.now usage)
• Not order-dependence within the file (this test runs in isolation in its own describe)

The signature: test expects inventory.available to be 98 (assumes 100 initial - 2 from this test) but receives 96 (100 - 2 - 2 from another concurrent test running in parallel that's also processing a WIDGET-1 order).

Why now (3 weeks ago): you doubled the parallel worker count from 4 → 8. With 4 workers, this test was unlikely to overlap with the other order-related tests touching WIDGET-1. With 8 workers, overlap probability rose. The test was always shared-state-vulnerable; you just stayed under the failure threshold before.

Reproduction Steps

# Reproduce reliably:
VITEST_MIN_WORKERS=8 npx vitest run tests/integration/ --pool=threads

# Or force the issue with focused runs:
for i in {1..50}; do
  npx vitest run tests/integration/order-processing.test.ts &
  npx vitest run tests/integration/order-bulk.test.ts &
  wait
  echo "Iteration $i complete"
done

This intentionally runs the order-processing test concurrently with other order tests that may share inventory state.

Root Cause

Multiple test files use setupTestData() which seeds the database with WIDGET-1 at quantity 100. They then run various orders that decrement WIDGET-1 inventory.

Because setupTestData() is in beforeAll (not beforeEach), the DB state is shared across all tests in the file. Worse: all parallel test files share the SAME database instance — meaning they share the same WIDGET-1 inventory record.

When tests run in parallel:

1. Test A reads inventory (100), creates order for 2, expects to see 98 after.

2. Concurrent Test B reads same inventory (100), creates order for 2, expects to see 98 after.

3. Both succeed — final inventory is 96.

4. Test A's assertion fails: expected 98, received 96.

This is a database-level shared state issue, amplified by test parallelism.

The Specific Fix

Choose ONE of these approaches based on team preference:

Option 1 (preferred): Per-test ephemeral database

Each test gets its own isolated DB instance. No sharing.

// /tests/helpers/setup.ts
import { newDb } from 'pg-mem'; // or test-containers for Postgres

export async function createTestDb() {
  const db = await newDb();
  await migrateAll(db);
  await seedDb(db);
  return db;
}

// Each test gets its own:
describe('Order Processing', () => {
  let db;
  beforeEach(async () => {
    db = await createTestDb();
  });
  
  it('processes order and updates inventory', async () => {
    // ... uses db (this test's instance)
  });
});

Pro: total isolation. No flakiness possible from shared state.

Con: slower (each test creates a DB instance). Use pg-mem for speed.

Option 2: Transaction-rollback pattern

Each test runs in a transaction; rolled back at the end. State changes don't persist.

import { beforeEach, afterEach } from 'vitest';

let tx;

beforeEach(async () => {
  tx = await db.beginTransaction();
});

afterEach(async () => {
  await tx.rollback();
});

it('processes order and updates inventory', async () => {
  const order = await tx.orders.create(...);
  await processOrder(order.id, { db: tx });
  // assertions...
  // After test: tx.rollback() resets everything
});

Pro: fast (no full DB recreation).

Con: requires processOrder to accept a DB connection (refactor effort). Doesn't work for tests that span the transaction (e.g., commit-required tests).

Option 3: Per-test schema/database

Each test gets a unique Postgres schema or database name; tests don't see each other.

beforeEach(async () => {
  const schemaName = `test_${randomString()}`;
  await db.query(`CREATE SCHEMA ${schemaName}`);
  await db.query(`SET search_path TO ${schemaName}`);
  await migrateAll(db);
});

afterEach(async () => {
  await db.query(`DROP SCHEMA ${schemaName} CASCADE`);
});

Pro: real Postgres. Full isolation.

Con: slowest. Setup overhead per test.

Recommended for your case: Option 1 (pg-mem) for unit-style tests, Option 2 (transaction-rollback) for tests that need real Postgres features.

What NOT to Do

1. Don't add `it.retry(3)` to make the test pass. It trains the team to ignore failures + hides the underlying issue.

2. Don't drop Vitest worker count back to 4. That's papering over the problem; it'll re-emerge as your test count grows.

3. Don't use `beforeAll` more aggressively for setup. That's the root cause; doing more of it makes flakiness worse.

4. Don't add `mutex` / `lock` to test execution. Killing parallelism kills your CI speed; isolation is the right answer.

5. Don't make the assertion non-strict (`toBeLessThan(99)` instead of `toBe(98)`). That hides the bug; you'd ship code that occasionally over-decrements inventory.

Verification Procedure

After implementing the fix:

# Run the test 100 times locally
for i in {1..100}; do
  npx vitest run tests/integration/order-processing.test.ts || break
done
echo "Completed without failure"

# Or use Vitest's built-in --repeat-each
npx vitest run tests/integration/order-processing.test.ts --repeat-each=100

# Pass criteria: 100/100 pass
# If any fail: fix is incomplete; investigate

Run with full parallelism:

npx vitest run --pool=threads --max-workers=16 --repeat-each=20

Forcing higher concurrency than CI normally uses surfaces edge cases.

Prevention Pattern

For the team going forward:

1. Default to per-test database isolation. Make this the easy path. Helper function `createTestDb()` returns isolated instance.

2. Forbid `beforeAll` for state-mutating setup. ESLint rule:

// .eslintrc.js
rules: {
  'vitest/no-shared-state-in-beforeall': 'error',
}

3. Run tests with `--randomize` to surface order-dependence. Add to CI.

4. Track flaky tests in a registry. When a test flakes, log it. Weekly review: any tests flaking? Fix or skip with `it.skip` (NOT silent retry).

5. Code-review checklist: any test that touches DB without explicit isolation gets a review comment.

If Other Tests Have This Pattern

Quick audit:

# Find tests using beforeAll with DB operations
grep -rn "beforeAll.*db\|beforeAll.*setupTestData" tests/

# Find tests modifying shared records
grep -rn "WIDGET-1\|inventory\|shared seed" tests/

Likely sister issues:

• Other order-test files touching same WIDGET-1 record (you mentioned 2 other tests started flaking — likely same pattern)
• User tests sharing same seed user records
• Subscription tests sharing seed plans

Migrate them with the same fix in priority order: anything customer-facing (payments, orders) first.

Key Takeaways

• Your bug is shared-state flakiness amplified by parallel workers. Doubled workers 3 weeks ago = doubled overlap probability.
• Fix at the database layer with per-test isolation. pg-mem for fast tests; transaction-rollback for tests needing real Postgres.
• Verify with 100x runs at high parallelism. Anything less than 100/100 stable means fix is incomplete.
• Ban beforeAll for state-mutating setup. ESLint rule prevents regression.
• Audit sister tests now. Same pattern almost certainly exists elsewhere; preempt before they flake too.