🪜 Multi-Step Agent Workflow Designer

Workflow Design: Podcast pitch triage with multi-agent + human-in-the-loop for high-stakes decisions

Architecture Recommendation

Multi-agent (3 agents) + deterministic orchestration via n8n, with mandatory human-in-the-loop for the 'book' decision.

Your current single-agent design fails because one OpenAI call is doing FOUR distinct cognitive tasks: (1) extract guest claims, (2) compare against your topic preferences, (3) compare against recent guests for redundancy, (4) synthesize a final fit score. Single LLM calls do task #4 well; task #2 and #3 require deterministic comparison against your stored data, which the LLM does badly because it's hallucinating against your actual preferences.

The wrong-decision rate of 35% is consistent with this anti-pattern. After redesign you should see <10% wrong decisions.

The Workflow Graph

[1] Pitch Ingest (deterministic, n8n)
      ↓ {guest_email, pitch_text, sender_metadata}
[2] Extractor Agent (LLM, ~$0.01)
      ↓ {guest_name, claimed_expertise[], topics_pitched[], proof_links[], pitch_quality_signals{}}
[3] Topic Match Scorer (deterministic + small LLM, ~$0.02)
      ↓ {topic_overlap_score, redundancy_score, freshness_score}
[4] Decision Synthesizer Agent (LLM, ~$0.03)
      ↓ {recommendation: 'book' | 'decline' | 'human-review', reasoning, draft_reply, confidence}
[5] Routing Switch (deterministic)
      ↓ if 'book' OR confidence < 0.85 → human-review queue (Slack DM to you)
      ↓ if 'decline' AND confidence ≥ 0.85 → auto-decline path
[6a] Auto-Decline Path (deterministic, ~$0.005)
      ↓ Send draft_reply, tag email, log decision
[6b] Human-Review Path (waits for your Slack reaction)
      ↓ ✅ → book path | ❌ → decline path | 🤔 → manual takeover
[7] Book Path (deterministic + LLM for invite text)
      ↓ Calendar event, send thread reply with proposed times
[8] Audit Logger (deterministic)
      ↓ Append decision to weekly review sheet for retrospective

Hand-Off Contracts

[1]→[2] Pitch Ingest → Extractor Agent

• Input: {guest_email: string, pitch_text: string, sender_domain: string, sender_first_seen: ISO-date}
• Output: {guest_name: string, claimed_expertise: string[], topics_pitched: string[], proof_links: string[], pitch_quality_signals: {has_specific_topic: bool, has_credentials: bool, has_links: bool, length_signal: 'too-short'|'right'|'wall-of-text', personalization_signal: bool}}
• Failure mode: pitch is non-English or too short to extract → return {insufficient_pitch: true} and skip to [6a] auto-decline path with template reply.

[2]→[3] Extractor Agent → Topic Match Scorer

• Input: extractor output
• Output: {topic_overlap_score: 0-1, redundancy_score: 0-1, freshness_score: 0-1, matched_preferences: string[], conflicting_recent_guests: string[]}
• topic_overlap_score: cosine similarity between topics_pitched embeddings and your preferences markdown embeddings. Deterministic.
• redundancy_score: 1 - max(similarity to last 30 guests). Deterministic.
• freshness_score: small LLM call ("is this topic something the audience has heard 5x in the last year?"). ~$0.005.
• Failure mode: embedding service down → fall back to LLM-only scoring + log degraded mode.

[3]→[4] Topic Match Scorer → Decision Synthesizer

• Input: extractor output + match scorer output
• Output: {recommendation: 'book'|'decline'|'human-review', reasoning: string, draft_reply: string, confidence: 0-1}
• Decision rules baked into the prompt: if redundancy_score > 0.7 → decline regardless. If topic_overlap_score < 0.3 → decline regardless. Else use LLM judgment.

[4]→[5]→[6a/6b] Routing

• Deterministic switch. NOT an LLM. The LLM proposed; the router executes policy.

Failure Recovery Policy

Checkpoint Strategy

Persist after each step in a single Postgres table pitch_workflow_state keyed by email message ID:

{
  message_id, ingested_at, extractor_output, scorer_output,
  decision, routing_decision, human_action, final_action,
  audit_trail[]
}

If step N fails, the workflow can resume from step N-1's persisted output. No LLM step ever re-runs unless explicitly requested.

Cost saved: re-running a failed step 7 currently costs $0 (vs $0.06 if we had to re-run from step 1).

Observability Plan

Metrics:

• Pitches/day ingested
• Auto-decline rate, auto-book rate, human-review rate
• Mean confidence score
• Mean latency from ingest to final action
• Cost per pitch
• Wrong-decision rate (computed weekly from your audit)

Logs at each hand-off: the input + output of each step, plus timing.

Alerts:

• Workflow crash (PagerDuty / Slack DM)
• Auto-decline rate >70% for 3 days (drift detector — your taste may be miscalibrated in the prompt)
• Confidence distribution shifts (mean confidence drops 20% week-over-week)
• Human-review queue >5 items for >2 hours

Platform-Specific Implementation (n8n + Postgres + small Python service)

n8n workflow nodes (high level):

1. Webhook trigger (Gmail push)

2. HTTP Request → Python service `/extract` (Extractor Agent)

3. HTTP Request → Python service `/score` (Topic Match Scorer)

4. HTTP Request → Python service `/decide` (Decision Synthesizer)

5. Switch node (routing)

6a. Gmail send + label (auto-decline)

6b. Slack DM with action buttons → wait for response

7. Calendar create + Gmail send (book)

8. Postgres insert (audit log)

Python service structure (FastAPI, ~150 lines):

/extract  → extractor_agent.run(pitch) → ExtractorOutput
/score    → scorer.run(extractor_output, preferences_md, recent_guests) → ScorerOutput
/decide   → synthesizer_agent.run(extractor + scorer + decline_examples) → Decision

Each agent is a SEPARATE prompt with single responsibility. Prompts live in version-controlled files; preferences live in preferences.md (LLM reads this); recent guests in recent_guests.json.

Anti-Patterns to Avoid

1. Putting your topic preferences in the prompt. They will drift. Keep them in `preferences.md` and inject. When you change preferences you change a file, not a prompt.

2. Single mega-prompt that does extract+score+decide. This is what you have now. It is why you're at 35% wrong decisions. Decompose.

3. No human-in-the-loop for 'book' decisions. Wrong-decline costs you a guest; wrong-book costs you a guest AND credibility. Always have human approval on book.

4. Embedding the recent-guests list in the prompt. Same issue as preferences — drifts and is expensive. Embed once, store, lookup at runtime.

5. Using LLM for the routing switch. Routing is policy, not judgment. Determine it deterministically from confidence + recommendation.

Test & Validation Plan

Step-isolation tests:

• Extractor: 20 pitches you've manually labeled. Validate output structure and field accuracy.
• Scorer: same 20 pitches. Validate scores match your intuition (topic_overlap correlation > 0.7 with your manual scoring).
• Synthesizer: same 20 pitches with extractor+scorer output. Validate recommendation accuracy. Target >85% match with your decisions.

End-to-end smoke test: 5 historical pitches replayed (3 you accepted, 2 you declined). System should produce same final action.

Shadow mode for 1 week: run new system in parallel with old, compare decisions, do not act on new system's output. Migrate when shadow agreement > 90%.

Migration Path

1. Deploy new system in shadow mode (read-only, decision logged but not executed).

2. Run for 7 days. Compare with current n8n decisions and your overrides.

3. If shadow agreement with your overrides > 90%: cut over.

4. Keep old workflow disabled-but-deployable for 14 days as rollback.

5. After 14 days clean shutdown, archive old workflow.

Key Takeaways

• Your current 35% wrong-decision rate is an architecture issue, not a model issue. One LLM call doing four cognitive tasks is the bug. Decomposing fixes most of it.
• The decision synthesizer needs human-in-the-loop on 'book' decisions. The asymmetric cost of wrong-book vs wrong-decline mandates this.
• Topic preferences belong in a file, not in the prompt. Drift will kill accuracy otherwise.
• **Use deterministic scoring where possible (embeddings + cosine similarity for topic overlap, vector lookups for guest redundancy). Reserve LLM for genuine judgment.
• Persist state at every hand-off. A $0.06 LLM call rerun because your calendar API hiccupped is preventable design.
• Set up the drift detector from day one. Auto-decline rate trending up over weeks is the early-warning signal that your preferences markdown is out of date.