🐛 Agent Prompt Debugger — Diagnose Why Your Claude Agent Is Drifting

Agent Debug: Support agent creating tickets instead of answering

An agent debugging protocol runs 8 structured diagnostic checks (system prompt clarity / tool description match / examples / context / stop conditions / termination / state / reward-hacking shape) to identify which of the 5 drift patterns (goal-loop / tool-confusion / context-degradation / instruction-dilution / reward-hack) your agent is exhibiting. Based on 100+ production agent debugs, the most common pattern — by a wide margin — is tool-confusion caused by ambiguous tool descriptions. Your reproducer (40% ticket-creation rate on simple doc-answerable questions) is a textbook case of this pattern: create_ticket has no 'when NOT to use me' clause, while search_docs has no 'use me FIRST for any question' clause. This debug prescribes specific tool-description rewrites + 1 system-prompt structural fix that should drop ticket-creation rate to <5% on doc-answerable questions.

Reproducer Status

Good: you have a reproducer. 'how do I change my email address' → creates ticket 40% of time.

Before fixing anything, nail down the reproducer more precisely:

1. Run the prompt 20 times with `temperature=0`. Count ticket-creation rate.

2. Run 20 times with `temperature=0.7`. Count again.

3. Add 5 variants of the question ('how to change email', 'email change', 'update my email address') and count.

This gives you baseline metrics. After the fix, re-run — expect >90% drop in ticket-creation rate.

Drift Pattern Diagnosis

Pattern: TOOL-CONFUSION

Evidence:

• Agent consistently picks create_ticket when it should pick search_docs
• The wrong tool does a LOT of work (ticket creation has side effects, costs, user UX impact)
• Tool descriptions don't disambiguate WHEN to use each
• Agent behavior suggests it's treating 'create_ticket' as 'handle this request' rather than 'escalate unresolvable request'

Secondary pattern: Minor instruction-dilution (system prompt is vague about sequence of operations).

Not these patterns:

• Not goal-loop (agent doesn't loop, it fails on turn 1)
• Not context-degradation (short-conversation failure)
• Not reward-hacking (agent isn't 'gaming' — it's confused about tools)

The 8 Diagnostic Checks

Check 1: System Prompt Clarity — FAIL

Current: 'You are a helpful customer support agent for ProjectHub. Answer questions accurately. If you cannot answer, create a support ticket.'

Problems:

• 'If you cannot answer' — undefined condition. Claude may interpret 'I don't have immediate knowledge' as 'cannot answer' rather than 'should search docs first.'
• No explicit sequence ('first search, then answer, ticket only as last resort')
• 'Use the available tools when appropriate' — too vague. Doesn't say which order.

Check 2: Tool Description Match — FAIL (Root Cause)

search_docs(query) current description is presumably empty or minimal.

create_ticket(subject, description, priority) current description is presumably empty or minimal.

The tool descriptions are NOT disambiguating. Claude picks create_ticket because it's listed as available and the system prompt doesn't force search-first.

Check 3: Examples Distribution — N/A

No few-shot examples provided in system prompt. This is actually OK for Opus/Sonnet with clear tool descriptions — but your tool descriptions aren't clear.

Check 4: Context Window Usage — PASS

Short conversations. No context degradation issue.

Check 5: Stop Conditions — PASS

Agent terminates correctly after tool calls. Not a termination bug.

Check 6: Termination Logic — PASS

Single-turn failure — not a termination issue.

Check 7: State Management — FAIL (Secondary)

No tracking of 'have I already created a ticket for this topic' = why you're seeing DUPLICATE tickets in same conversation. Agent has no memory that it just created ticket #4892 on the previous turn.

Check 8: Reward-Hacking Shape — N/A

No reward-hacking here. Agent is following what it thinks are instructions.

Root Cause Analysis

Root cause: Tool descriptions don't enforce sequence.

Claude, when faced with a user question, evaluates which tool best matches. With vague descriptions, create_ticket looks like a reasonable tool for 'handling a user request.' The system prompt's 'if you cannot answer' is ambiguous enough that Claude sometimes interprets 'I should search docs first, but ticket is also valid' as 'ticket is fine.'

Secondary cause: No duplicate-prevention state = same-conversation duplicates.

The Fix

Fix 1: Rewrite system prompt with explicit sequence

You are a customer support agent for ProjectHub (a project management SaaS).

Your job has this strict sequence:
1. FIRST: search the documentation via `search_docs` to find an answer.
2. IF docs have the answer: answer the user directly, citing the relevant doc section.
3. IF docs don't have the answer AND the question requires account-specific info: use `get_user_account` to fetch it.
4. IF docs don't have the answer AND you can't resolve it yourself: create a support ticket via `create_ticket`.
5. IF the issue is urgent or the user is frustrated: escalate to a human via `escalate_to_human`.

You MUST call `search_docs` before creating any ticket. If the question is 'how do I do X' or 'where is Y' — docs almost certainly have the answer.

If you have already created a ticket in this conversation, do NOT create another one. Refer the user to the existing ticket instead.

Be friendly but concise. Don't apologize unnecessarily. Answer the question.

Fix 2: Rewrite tool descriptions

{
  name: 'search_docs',
  description: 'Search the ProjectHub documentation for an answer to the user\'s question. ALWAYS call this FIRST for any question that starts with "how do I", "where is", "can I", "does ProjectHub support", or similar doc-answerable patterns. Returns relevant doc sections with URLs to cite.',
  // ...
},
{
  name: 'get_user_account',
  description: 'Fetch the authenticated user\'s account information (plan, billing, settings). Only call this AFTER `search_docs` if the question requires account-specific info (e.g., "what plan am I on?", "when did I sign up?").',
  // ...
},
{
  name: 'create_ticket',
  description: 'Create a support ticket for a human agent to handle. ONLY call this as a LAST RESORT after `search_docs` failed to find an answer AND you cannot resolve the issue yourself. DO NOT create a ticket for questions that have doc-answerable patterns like "how do I" or "where is". DO NOT create multiple tickets in the same conversation — if a ticket exists, refer to it.',
  // ...
},
{
  name: 'escalate_to_human',
  description: 'Hand off to a live human support agent immediately. Use when: user explicitly asks for a human, user is clearly frustrated/angry, or the issue is urgent/security-related. Different from create_ticket — this is real-time handoff, not async ticket creation.',
  // ...
}

Fix 3: Add conversation state for duplicate prevention

Pass a conversation_state field through tool calls that tracks ticket_created_this_conversation: boolean. Update after first create_ticket call. System prompt already addresses this, but belt + suspenders.

Before/After Examples

Example 1: User asks 'how do I change my email?'

Before:

• 40% of time: agent calls create_ticket (wrong)
• 60% of time: agent calls search_docs then answers (correct)

After:

• Agent reads system prompt: 'MUST call search_docs before creating ticket'
• Agent reads create_ticket description: 'DO NOT create ticket for "how do I" questions'
• Agent calls search_docs('how to change email address')
• Docs return: 'Settings → Account → Email → Edit → Save'
• Agent answers user directly, citing doc
• Ticket-creation rate on this question: expected <5%

Example 2: User asks 'my account is locked, help'

Before:

• Agent might create ticket directly

After:

• Agent calls search_docs('account locked')
• If docs have self-service unlock: agent provides it
• If docs require human action: agent calls escalate_to_human (real-time) OR create_ticket (async, if non-urgent)
• Correct path selected based on issue urgency

Regression Test Case

Add these tests to your agent test suite:

describe('Agent: tool selection', () => {
  test('Simple doc-answerable question uses search_docs, not create_ticket', async () => {
    const result = await runAgent('how do I change my email address?');
    expect(result.toolCalls[0].name).toBe('search_docs');
    expect(result.toolCalls.find(t => t.name === 'create_ticket')).toBeUndefined();
  });
  
  test('Account-specific question uses get_user_account after search_docs', async () => {
    const result = await runAgent('what plan am I on?');
    const toolSequence = result.toolCalls.map(t => t.name);
    expect(toolSequence).toEqual(expect.arrayContaining(['search_docs', 'get_user_account']));
  });
  
  test('No duplicate tickets in same conversation', async () => {
    const conversation = await startConversation();
    await conversation.send('I have a bug to report'); // might create ticket
    await conversation.send('another issue'); // must NOT create second ticket
    const allTicketCalls = conversation.toolCalls.filter(t => t.name === 'create_ticket');
    expect(allTicketCalls.length).toBeLessThanOrEqual(1);
  });
  
  test('Urgent issue triggers escalate_to_human not create_ticket', async () => {
    const result = await runAgent('urgent: my data is deleted, I need help NOW');
    expect(result.toolCalls.find(t => t.name === 'escalate_to_human')).toBeDefined();
  });
});

Success criteria: All 4 tests pass reliably (>95% of runs) after fix.

Monitoring Plan

In production after deploying fix:

1. Metric: `support_agent.ticket_created_per_conversation`

- Before: ~0.6 (high — many conversations create tickets unnecessarily)

- Target: ~0.15 (only for actually-unresolvable issues)

2. Metric: `support_agent.doc_searched_before_ticket_ratio`

- Should be ~100%. If not, tool sequence fix failed.

3. Metric: `support_agent.duplicate_tickets_per_conversation`

- Target: 0 or near-zero.

4. Weekly review: sample 10 conversations with ticket creation. Manually verify: was ticket creation correct (docs didn't have the answer)?

Alert: if ticket-creation rate jumps back above 0.3/conversation, investigate immediately (usually regression from a prompt change elsewhere).

Common Misfires

Things that LOOK like fixes but won't actually help:

1. Adding more 'be helpful' instructions. Doesn't address the sequence issue. The agent IS trying to be helpful — creating tickets is (to it) a helpful action.

2. Switching model (Sonnet → Opus). You tried this. Marginally helps but doesn't fix root cause. Opus is better at following ambiguous instructions; it's not magical.

3. Temperature=0. Makes wrong behavior MORE consistent, not less. Don't use this as a debug tool.

4. Adding a global rule 'always search docs first.' Helps somewhat but doesn't compound with specific tool descriptions. Both are needed.

5. Removing `create_ticket` from tool list entirely. Over-correction. Tickets ARE needed sometimes. Keep the tool, fix the description.

Key Takeaways

• Drift pattern: tool-confusion. Claude picks create_ticket because tool descriptions don't disambiguate + system prompt doesn't enforce sequence.
• Fix is 3 parts: (1) system prompt with explicit 5-step sequence, (2) tool descriptions with 'when to use me' and 'when NOT to use me', (3) conversation state to prevent duplicate tickets.
• Expected improvement: ticket-creation rate drops from 40% to <5% on doc-answerable questions. Duplicate tickets eliminated.
• Regression test suite: 4 test cases in Jest covering tool selection. Run on every prompt change to catch drift.
• Monitor: ticket-created-per-conversation metric weekly. Alert if it jumps — usually signals prompt regression from unrelated change.