🛟 Customer Support Agent Architect

Support Agent Architecture: SaaS Project-Management Support

Resolution Tier Map

Honest target: 55-65% true resolution rate. Your stated 50% target is achievable. Going past 70% would require expanding the agent's scope into territory where hallucination risk grows.

Architecture Overview

Single agent (Claude Sonnet) with 8 tools, plus Claude-Opus-as-judge for an offline trajectory eval pipeline.

Multi-agent isn't justified at 200 tickets/day; the orchestration overhead exceeds the benefit. Single agent with strong prompt + good tool design is the right pattern at this scale.

Why Sonnet for production: latency target <1.5s for chat feels fluid; Opus is overkill and slower. Reserve Opus for offline eval judging where latency doesn't matter.

The Conversation Flow (Chat)

[GREET]
  'Hi! I'm the AI support agent for [product]. I can help with logins, how-to questions, billing, account changes, and integration setup. For complex issues I'll loop in a human. What do you need help with? (You can also type "human" anytime to talk to our team.)'
  ↓
[INTENT_DETECT]
  Classify into 1 of 10 categories. Branch.
  - If user just typed 'human': straight to ESCALATE
  ↓
[CATEGORY-SPECIFIC FLOW]
  Each category has its own sub-flow (see Per-Category Handling)
  ↓
[CONFIRM_RESOLVED]
  After resolution attempt: 'Did that solve it?'
  - 'yes' → CLOSE (with thank)
  - 'no' or unclear → ESCALATE with full context
  ↓
[ESCALATE / HANDOFF]
  - In hours: live transfer to human via Intercom routing rule with context
  - Out of hours: 'Our team is offline (Mon-Fri 9-6 [timezone]). I've created a ticket; we'll respond by [next business morning].'
  ↓
[CLOSE]
  'Glad I could help. We send a 1-question CSAT to know how I did.'

Per-Category Handling

Password/login (Tier 3 — full resolve)

Flow: 'I can help reset that. What email do you use to log in?'
  → tool: lookup_user_by_email
  → if found: trigger Auth0 reset email via tool
    → 'I sent a reset link to [masked email]. Check your inbox + spam. Did it arrive?'
  → if not found: 'I don't see that email. Could you double-check, or are you on a teammate's account?'

How-to questions (Tier 2/3 hybrid)

Flow: 
  → tool: kb_semantic_search(query)
  → if KB hit with confidence >0.7: 'Here's how: [extract key steps from KB article + link]'
  → ask: 'Did that work?'
  → if no: ESCALATE with KB article reference + customer's question
  → if low confidence: 'Let me get a teammate on this.' → ESCALATE

Billing (Tier 2)

Flow:
  → ask intent: 'What's the billing question?'
  → tool: lookup_account_billing(workspace_id)
  → branches:
    - invoice question: render invoice info from Stripe + link to invoice PDF
    - plan info: render current plan + link to upgrade/downgrade
    - 'how do I cancel': honest answer + link to settings + 'I can also help with that here, want to?'
  → if customer says 'I want to cancel': ask reason, log it, then ESCALATE to retention (do NOT auto-execute cancellation)
  → for refund requests: ESCALATE always (humans handle)

Bug reports (Tier 1+4)

Flow:
  → 'Thanks for the report. Could you share: what you were doing, what happened, what you expected, and your browser/OS?'
  → capture answers
  → tool: create_engineering_ticket(captured_data, customer_workspace_id)
  → 'Got it. Logged as [ticket ID]. Engineering will review. We'll email if we need more info.'
  → ESCALATE to support team for follow-up tracking

Feature requests (Tier 1)

Flow:
  → 'Thanks for the suggestion. What problem would this solve for your team?'
  → capture
  → tool: log_feature_request(suggestion, problem, customer_workspace_id)
  → 'Logged. Roadmap is at [link]. Anything else?'

Account changes/cancel (Tier 2)

Flow:
  → 'What change?'
  → simple changes (update billing email, change workspace name): execute via admin API
  → cancellation: explain process + 'A teammate will help close it out — they'll also note any feedback.' → ESCALATE
  → seat downgrades: explain implications + escalate if confused

Integration help (Tier 2/3)

Flow:
  → 'Which integration?'
  → for top 5 (Slack, Linear, GitHub, Notion, Zapier): tool: kb_integration_setup(name) + walk-through
  → for other integrations: escalate with the customer's specific question

Escalation Policy

Always-escalate triggers:

• Customer types 'human', 'agent', 'real person', 'representative'
• Refund requests
• Cancellation requests (after capturing reason)
• Bug reports (after intake)
• Performance complaints
• 2 consecutive turns where the agent's confidence in its answer is <0.6
• Any mention of legal, lawyer, GDPR data request, ToS dispute
• Any tone signal of significant frustration ('this is ridiculous', 'I'm so frustrated', 'I want to cancel because of this')

Context to pass on escalation:

• Full conversation transcript
• Customer workspace ID + plan tier + tenure
• The category the agent classified into
• The specific reason for escalation (what the agent couldn't handle)
• Any actions already taken (e.g., 'I sent a password reset email; customer says it didn't arrive')

Out-of-hours:

• 'Our team is offline. Mon-Fri 9am-6pm [timezone].'
• Create ticket in Intercom inbox with same context
• Promise response by '[next business morning]' — be specific about the time window
• For high-severity (paid customer, performance complaint, bug affecting their workflow): page on-call

Tone & Language Guide

Voice: Direct, friendly, plain language. Like a competent colleague, not a corporate apology bot.

Use:

• 'Got it.' / 'Sure.' / 'Quick check —'
• 'I can fix that.' / 'Let me look that up.'
• 'Did that work?' / 'Anything else?'
• 'A teammate will help — let me get them.' (for handoffs)

Avoid:

• 'I sincerely apologize for the inconvenience.' → Just 'Sorry about that.'
• 'I understand this is frustrating.' → Often empty; either help or escalate
• 'Please bear with me.'
• 'I'm afraid I'm unable to assist with that at this time.' → 'A teammate handles that — let me get them.'
• Excessive 'thank you for your patience'
• Pretending to be human; using human-sounding names

Never:

• Promise specific dates without checking ('Engineering will fix this by Friday')
• Quote pricing or plan details from memory; always look up
• Generate KB content the agent isn't sure exists

Anti-Bot-Hatred Patterns

1. Always show 'human' is an option — every greeting and escalation. Don't hide it 3 menus deep.

2. Don't pretend to be human. 'I'm the AI support agent' upfront.

3. Don't loop on FAQ links. If customer says 'I read that, didn't help' — escalate, don't return more links.

4. Don't hold customers hostage. No 'first describe your problem in detail before I can transfer you.' Transfer on request.

5. No corporate-apology spam. Customers find it insincere.

6. No fake empathy. 'That sounds frustrating' from a bot is universally hated. Just help.

7. Don't gate the human option behind questions. 'Type human anytime' should always work, no questionnaire required.

Implementation Skeleton

/support_agent
  /prompts
    system_prompt.md          (tone, principles, scope)
    intent_classifier.md      (10-category classifier)
    flow_definitions.md       (per-category sub-flows)
  /tools
    lookup_user_by_email.py
    trigger_password_reset.py
    kb_semantic_search.py
    lookup_account_billing.py
    create_engineering_ticket.py
    log_feature_request.py
    update_account.py
    escalate_to_human.py
  /knowledge_base
    kb_index.json             (vector index over your Notion KB)
    refresh_kb.py             (cron daily to re-index)
  /eval
    golden_cases.jsonl        (200 historical tickets with ground truth)
    judge_prompt.md           (Claude Opus judge for trajectory eval)
    run_evals.py              (CI hook)
  /runtime
    intercom_webhook.py       (entry point)
    agent_loop.py             (main orchestration)
    metrics.py                (logging + dashboard)
  rollout_state.json          (shadow / partial / full state)

Metrics & Improvement Loop

Track per-conversation:

• Resolution status: resolved / escalated / abandoned
• Resolution time
• Tool-call count + cost
• Customer satisfaction (1-question CSAT after close)
• Escalation reason (categorized)

Weekly dashboard:

• True resolution rate (resolved without escalation, customer confirmed)
• Escalation rate by category — find the categories where you're under-performing
• Top 5 escalation reasons — these are your improvement backlog
• CSAT distribution — track separately for resolved-by-bot vs resolved-by-human
• Cost per resolved ticket

Monthly improvement loop:

• Pick top 2 escalation reasons from the week. Either: (a) add them to the agent's scope with new tooling, or (b) accept they're out of scope and tune escalation copy to feel less frustrating.
• Refresh KB index if Notion KB has changed.
• Re-run eval suite on golden cases. Address regressions.

Rollout Plan

Week 1-2: Shadow mode

• Agent runs on every chat in parallel with human routing.
• Agent's responses are logged but not shown to customers.
• Compare: would the agent's response have been correct? Use Opus judge + spot-check 50 conversations/day.
• Goal: agent agreement with human responses >75%.

Week 3-4: Tier 1 + Tier 2 only (safer categories)

• Agent handles greetings, FAQ, password resets, simple billing lookups.
• Hard escalation on everything else.
• Goal: 30% of tickets fully resolved by agent, no customer complaints about being stuck with the bot.

Week 5-6: Add Tier 3 categories (how-to, integration help)

• Expand resolution scope.
• Goal: 45% resolution rate.

Week 7+: Full operation, tune from escalation data.

• Goal: 55-60% sustained resolution rate.

Rollback criteria at any stage:

• CSAT for bot-resolved tickets <3.5/5
• Escalation rate >2× baseline expectation for a category
• Any single hallucination incident affecting customer trust (e.g., wrong account info given)

Known Limitations

Will not handle well:

• Novel bugs not in KB (escalates by design — fine)
• Complex multi-step billing situations (refunds, prorations, plan changes mid-cycle) — escalate
• Anything requiring policy judgment (refund requests, special pricing) — escalate
• Customers who write in non-English (depending on your customer base, may need to add multilingual support; current scope is English-only)
• Customers in active churn frustration — escalate with priority flag, don't try to retain
• Integrations not in your top 5 — KB likely doesn't have setup steps; escalate

Set explicit expectations: in your help docs, mention that AI handles common questions with a human team for the rest. Customers who know the system works WITH humans (not instead of) trust it more.

Key Takeaways

• 55-65% honest resolution rate is achievable; 80% is not without hallucination risk. Set expectations accordingly.
• The architecture is single-agent + 8 tools, NOT multi-agent. At 200 tickets/day, multi-agent is overkill.
• Be honest the agent is AI from the first message. Customers prefer this once they know your other support is good.
• Always show 'type human anytime' visibly. Hidden escalation is the universal anti-pattern of hated bots.
• Track escalation REASONS, not just count. The reasons are your roadmap for what to expand into next.
• Roll out shadow → tier 1+2 → add tier 3 → full. 6-7 weeks to full operation. Don't shortcut to full from day one.