🧪 Test Strategy Architect

Test Strategy: Next.js + tRPC + Stripe SaaS

Diagnosis

Your test pain isn't 'low coverage' — it's 'wrong tests.' You have ~35% coverage on utility functions you wrote 18 months ago, and 0% coverage on the Stripe webhook handler that's been your top production bug source.

Critical fixes (in order):

1. Build a contract test layer for Stripe webhooks. Replay real Stripe events through your handler. Catches 80% of the bugs that currently hit production.

2. Delete the broken Playwright e2e test + the flaky integration tests. They're costing CI time + trust without finding bugs.

3. Invert the pyramid: more unit + contract tests; fewer integration tests. CI drops from 12min → 4-5min.

4. Establish flaky-test policy: flakies are bugs to fix, not retries to add.

Estimated outcome: CI drops to ~4 min. Stripe webhook bugs to production drop ~80%. Engineers regain trust in CI signal.

The Pyramid for This Codebase

For your scale + stack, target:

        ┌─────┐
        │ E2E │   5-10 tests   (5%)
        ├─────┤
        │     │
        │ INT │   30-50 tests  (15%)
        │     │
        ├─────┤
        │ CON │   20-30 tests  (15%)  ← CONTRACT TESTS, the missing layer
        ├─────┤
        │     │
        │     │
        │ UNIT│   200+ tests   (65%)
        │     │
        │     │
        └─────┘

Why contract tests as a layer: Your top bug source (Stripe webhooks) lives at a service boundary. Contract tests are perfect for this — fast, focused, deterministic.

What to Unit-Test

Unit-test (Vitest, fast, no I/O):

• Pure functions in /lib/: pricing calculations, date helpers, validators, formatters.
• Business logic that doesn't touch DB/HTTP: subscription tier logic, feature-flag evaluation, user permission checks.
• Complex tRPC procedure logic that you can extract from the procedure into a pure function — then unit-test that function.
• React component logic isolated from API calls (use vi.mock for trpc client).
• Drizzle query builders that return SQL — test the SQL is correct, not the result of running it.

Example: subscription pricing. If you compute tiered pricing, this is the highest-leverage place to test. Cover all tier boundaries, prorations, edge cases.

Don't unit-test:

• Trivial getters/setters
• Direct DB ORM calls (those need integration tests)
• HTTP client wrappers (mock the HTTP, test the wrapper logic only)
• React components that just render UI without logic (visual regression covers these if needed)

What to Integration-Test

Integration tests (slower, real DB or close to it):

• tRPC procedures end-to-end that involve DB writes + reads (use a test DB or transaction rollback).
• Auth flows: login, token refresh, permission enforcement.
• Critical multi-step flows: create order → process payment → mark shipped.
• DB migration tests: verify migration up + down work without data loss on representative data.

For your codebase specifically (~30-50 integration tests):

• 15 for tRPC procedures (top user-facing flows: subscribe, cancel, upgrade, billing portal)
• 8-10 for auth (login flows, MFA if applicable, role checks)
• 5-8 for critical billing flows (create-subscription, change-tier, refund)
• 4-6 for queue workers (job runs end-to-end through DB)

Don't integration-test:

• Every CRUD endpoint (most are trivial; cover via unit tests on the validation layer)
• Frontend → backend → DB full chain (that's e2e territory; expensive)
• Third-party API calls live (use contract tests)

What to E2E-Test

E2E (Playwright, slowest, brittle):

Only the 5-10 critical user flows that MUST work:

1. Sign up → email confirm → first login

2. Subscribe to paid plan (Stripe Checkout end-to-end)

3. Cancel subscription via portal

4. Upgrade tier mid-cycle

5. Reset password flow

Don't e2e-test:

• Every page renders
• Every form submits
• Every error state
• Every UI interaction

E2E is for 'this critical path must not break in production.' Not for coverage.

What NOT to Test

1. Trivial code with no logic. Getters, setters, factory functions returning literals, type definitions.

2. Implementation details. 'Did this function call X then Y' tests lock in implementation. Test outputs, not call sequences.

3. Generated code. tRPC client types, Drizzle schema-generated types, OpenAPI clients.

4. Framework code. Don't test that Next.js routes work; test YOUR route handlers.

5. Third-party libraries. Don't test that Stripe SDK works; test that YOUR handler does the right thing when Stripe events arrive.

6. The README, the docs, the config files (occasionally a config validation test is useful — usually overkill).

Tests to Delete from Your Current Suite

Without seeing your code, I'd guess these patterns:

1. Snapshot tests of components. Brittle, lock implementation, train team to thoughtlessly update. Replace with explicit assertions on visible behavior.

2. Tests that mock everything. If a unit test has 8 vi.mock() calls, you're testing your mocks, not your code. Either extract logic to a pure function (and test that) or move to integration test.

3. The broken Playwright test (the one not running for 3 months). Delete it. If it was important, you'd have fixed it. The fact it's been broken means it wasn't catching anything.

4. Tests that have been retried/flagged as 'allow flaky.' If the team retries, the test is signaling something. Either fix or delete.

5. Tests that exercise the same code path 10 times with trivially different inputs. Pick 1-2 representative cases + edge cases.

Run a 'tests-not-failing-when-code-is-wrong' audit. Take 10 tests at random. Comment out the assertion. Does the test still pass? If yes, the test isn't testing anything. Delete or fix.

Contract Test Layer (the missing piece)

This is your highest-leverage addition.

For Stripe webhooks

/tests/contracts/stripe/
  fixtures/
    invoice.payment_succeeded.json    (real Stripe event payload)
    customer.subscription.created.json
    customer.subscription.updated.json
    customer.subscription.deleted.json
    invoice.payment_failed.json
    charge.dispute.created.json
    [12-15 fixtures total of events you handle]
  webhook-handler.test.ts             (tests handler against each fixture)

Each test:

it('handles invoice.payment_succeeded by marking customer paid', async () => {
  const event = loadFixture('invoice.payment_succeeded.json');
  const result = await handleStripeWebhook(event);
  expect(result.status).toBe('processed');
  expect(await db.customer(event.data.object.customer)).toMatchObject({
    payment_status: 'paid',
    last_payment_at: expect.any(Date),
  });
});

How to capture fixtures: Stripe CLI can listen to test events + dump them. Or use Stripe Dashboard's event log to copy real production events (anonymize sensitive data).

Contract test for unknown events:

it('logs + ignores unknown event types', async () => {
  const event = { type: 'fictional.event.type', data: { object: {} } };
  const result = await handleStripeWebhook(event);
  expect(result.status).toBe('ignored');
  expect(logger.warn).toHaveBeenCalledWith('Unknown Stripe event type', expect.anything());
});

This catches the 'event type we didn't handle' bug class.

For other external services

Also add contract tests for:

• Auth provider webhooks (if using Auth0/Clerk/etc.)
• Email service (if using SendGrid/Postmark/etc.)
• Anything that posts events to your system

Flaky Test Policy

Rule: a flaky test is broken until proven otherwise.

When a test flakes:

1. Mark `.skip()` (don't delete yet)

2. Open a ticket: 'Fix flaky test X'

3. Investigate root cause within 1 sprint

4. Common causes: race conditions, time-dependent code (use clock-mocking), order-dependent tests, shared state, real network calls

5. Fix or delete. Never 'add retry to make it pass.'

Why this matters: Retried-flaky tests train engineers to ignore CI signal. Once that habit forms, real failures get ignored too.

CI Execution Strategy

On save (local, instant):
  → Vitest watching changed unit-test files (~50ms feedback)

On commit (pre-push hook):
  → Lint + type-check + relevant unit tests (~30s)

On PR open:
  → Full unit suite (parallel, ~1.5 min)
  → Integration tests (parallel, ~2 min)
  → Contract tests (~30s)
  → Type-check (~30s)
  → Lint (~20s)
  Total: ~3-4 min, parallel where possible

On merge to main:
  → Above PR pipeline
  → E2E tests against staging (~3-5 min)
  → Visual regression on key pages (~1 min)
  Total: ~6-8 min before deploy

Drop CI from current 12 min → ~4 min on PR. Engineers will care about CI again.

Parallelization:

• Vitest natively parallels by file
• Integration tests parallelize per-test-DB-instance (use Docker for ephemeral DBs)
• E2E with Playwright parallelizes by worker

AI-Generated Test Quality Gate

If engineers use Cursor/Claude Code/Copilot to write tests:

Mandatory verification: for any AI-generated test, run the 'mutation' check:

1. Comment out the production code's logic (e.g., return early)

2. Run the test

3. If the test still passes → the test isn't actually testing the logic. Reject.

Code review red flags:

• Test mocks the function under test (the test mocks itself effectively)
• Test asserts on internal implementation details (expect(spy).toHaveBeenCalledWith(...))
• Test asserts trivial things (expect(result).toBeDefined())
• Test has 6+ mock setups (you're testing the mock contract, not the code)

Test Data Strategy

For your scale, use:

Factories (test-data builders):

// /tests/factories/user.ts
export const buildUser = (overrides = {}) => ({
  id: 'user_test_' + Math.random(),
  email: 'test@example.com',
  plan_tier: 'starter',
  ...overrides,
});

Use for unit + integration tests. Avoid hardcoded JSON fixtures except for contract tests.

Fixtures (real-world payloads):

Reserved for contract tests. Real Stripe events, real auth provider events.

Avoid snapshots. Especially for components. Use explicit assertions.

Test DB strategy:

• For integration tests: ephemeral Postgres via Docker (postgres:16 image, fresh per test suite)
• Use Drizzle's transaction rollback for per-test isolation
• Or: per-test-database (slower but bulletproof)

Implementation Order

Week 1: Audit + delete

• Run the 'mutation' check on 30 random existing tests. Delete failures.
• Delete the broken Playwright test.
• Delete snapshot tests where present.
• Goal: existing test count drops 30-40%; CI time drops some.

Week 2: Contract tests for Stripe

• Capture 12-15 Stripe webhook event fixtures
• Write contract tests against your webhook handler
• This alone should catch your monthly Stripe bug

Week 3: Unit-test the billing logic

• Extract pricing/billing functions from tRPC procedures into pure functions
• Unit-test the pure functions (high coverage on the actual money paths)
• Aim: 30+ tests on billing logic, fast

Week 4: Critical integration tests

• Subscribe / Cancel / Upgrade tRPC procedures with real DB
• Auth flows
• Queue worker happy paths
• Aim: 10-15 strong integration tests

Week 5: E2E rebuild

• 5-7 critical user flows in Playwright (subscribe, cancel, upgrade, password reset, login)
• Run on merge to main only
• Aim: stable, fast, valued

Week 6: CI optimization

• Parallelize aggressively
• Move slow tests to merge-only
• Establish flaky-test policy
• Goal: PR CI under 4 min

Week 7+: Iterate

• Re-run mutation check quarterly
• Add tests for production bugs as they happen (regression prevention)
• Monitor flaky-test count (target: 0)

Coverage Target

Real target: 60-70% line coverage, with 90%+ on critical paths (billing, auth, webhooks).

Not 100%. Not 35%. The shape matters more than the number.

Track these instead of overall coverage:

• % coverage on /lib/billing/ → target 90%
• % coverage on Stripe webhook handler → target 100% (yes, here)
• % coverage on auth code → target 90%
• % coverage on UI components → don't track (waste of metric)

What This Strategy Won't Solve

• Won't replace product QA. Tests catch regressions; they don't catch 'this feature is wrong.'
• Won't prevent all production bugs. Test strategy reduces certain bug classes; bugs still happen. Plan for fast incident response too.
• Won't help if engineers don't run tests locally. Culture matters. CI catches eventually, but inner-loop testing is where bugs die fastest.
• Won't fix architectural bugs. If your billing logic is fundamentally wrong, tests pass + bugs ship. Architecture review > tests.
• Won't compensate for skipping types. TypeScript types catch a class of bugs tests can't easily reach. Use both.

Maintenance Cadence

Per-PR (engineer):

• Run unit tests locally before pushing
• New code includes tests at the right layer

Per-sprint (tech lead):

• Review flaky-test count (target: 0)
• Review CI time (target: <5 min for PR)
• Investigate any test that takes >2s

Quarterly (eng team):

• Mutation check on 50 random tests. Delete failures.
• Coverage audit on critical-path code
• Test strategy review: are we still hitting the right bug classes?
• Flaky-test root cause analysis

Annually:

• Full test architecture review
• Tooling evaluation (still Vitest? Time to consider alternatives?)
• Postmortem on production bugs that escaped tests

Key Takeaways

• Your problem is wrong tests, not low coverage. 35% coverage on the wrong code is worse than 60% on the right code.
• Contract tests for Stripe webhooks are your highest-leverage addition. Replay 15 real events; catch 80% of your monthly bugs.
• Delete the broken Playwright test + flaky integration tests. CI trust matters more than coverage numbers.
• Invert the pyramid: more unit + contract; fewer integration. CI drops from 12min to 4min.
• Flaky tests are bugs in the test, not retries to add. Fix root causes; don't normalize ignoring CI.
• AI-generated tests need the mutation check. Comment out the logic; if test passes, the test isn't testing anything. Reject.