📊 AI Agent Observability Stack — Logs, Traces, Evals, Cost For Production Agents

Observability Architecture: Marketing AI Assistant

An AI agent observability architecture is a 6-layer system (invocation trace / tool-call spans / prompt-response logs / cost attribution / eval pipeline / alerting) that transforms opaque agents into debuggable systems. Based on patterns from teams running 1M+ agent invocations/month, the typical failure pattern is 'observability bolted on after scale pain' — retrofitting costs 5x more than building it in from the start. Your situation (8K→40K invocations/day projected, SOC2 achieved, GDPR concerns, need per-user cost attribution, $2-5K/mo budget) is ideal for the mid-tier paid stack: Langfuse (hosted) + OpenTelemetry + Datadog (existing) + custom eval pipeline. This architecture produces full observability in 4 weeks of implementation, ~$1,800/mo run-rate, and positions you for the 5x growth coming.

Current State + Gap Analysis

Current state: CloudWatch logs only. Effectively zero LLM-specific observability.

Gaps (critical):

• No distributed tracing — can't reproduce failures
• No prompt/response logging — can't debug quality complaints
• No cost attribution — can't identify expensive users
• No eval pipeline — prompt changes ship without validation
• No LLM-aware alerting — only infrastructure alerts
• No audit trail for compliance

Gaps (important but not critical):

• No prompt versioning — hard to correlate behavior changes with prompt changes
• No hallucination detection — quality issues discovered by users

Verdict: Current state is below the bar for a B2B SaaS processing customer data at this scale. Urgency: start this within 2 weeks.

Layer 1: Invocation Trace

Tool: OpenTelemetry + Langfuse

Instrument every agent invocation with a root span:

• agent.invocation (root span)

- user_id

- organization_id

- feature (which agent — analyze / draft / query)

- input_type (text / voice / structured)

- total_latency_ms

- total_cost_usd

- success/failure

- output_length

Python SDK pattern:

from opentelemetry import trace
from langfuse.decorators import observe

@observe(name='agent.invocation')
async def run_agent(user_id: str, query: str):
    span = trace.get_current_span()
    span.set_attribute('user_id', user_id)
    span.set_attribute('feature', detect_feature(query))
    # ... agent logic
    return response

Langfuse integrates natively with OTEL and Anthropic SDK. Traces from Anthropic SDK (prompt, response, tokens) flow automatically into Langfuse.

Layer 2: Tool-Call Spans

Tool: OpenTelemetry child spans within invocation trace

Every tool call gets a child span:

• tool.hubspot_query (child of agent.invocation)

- tool_name

- input_params

- latency_ms

- status (success/error)

- error_message (if applicable)

- output_size_bytes

This lets you reconstruct the full graph of a failed invocation:

agent.invocation (10.2s, failed)
├── llm.claude.call (2.1s, ok)
├── tool.hubspot_query (6.8s, ok)
├── tool.ga4_query (0.9s, ok)
└── llm.claude.call (0.4s, failed — context window exceeded)

Immediately visible: context window exceeded after tool calls returned too much data.

Layer 3: Prompt-Response Logs

Tool: Langfuse (hosted) for structured LLM logs + S3 for long-term archive

Every LLM call logs:

• prompt (full text, PII-redacted)
• response (full text, PII-redacted)
• model (claude-sonnet-4-5-20250929 or similar)
• prompt_version (if versioned)
• input_tokens
• output_tokens
• cost_usd (computed from tokens × rates)
• latency_ms
• tool_calls_requested (names + params)
• tool_calls_executed (names + params + results)
• user_id (linked to invocation trace)

PII redaction:

• Pre-write middleware scrubs emails, phone numbers, credit cards via regex
• For customer data (marketing contacts), hash PII into opaque IDs before logging
• Keep original data in Postgres (encrypted) if debugging requires it

Retention: 90 days in Langfuse (hot, queryable), 2 years in S3 (cold, compliance).

Layer 4: Cost Attribution

Your critical need. Per-user cost attribution for pricing tier decisions.

Implementation:

• Every LLM call includes user_id in Langfuse metadata
• Every tool call has cost attribution (HubSpot API = free, GA4 API = free, but compute = something)
• Aggregate hourly into a user_costs Postgres table

Schema:

CREATE TABLE user_costs (
  user_id TEXT,
  organization_id TEXT,
  date DATE,
  hour INT,
  invocation_count INT,
  total_input_tokens INT,
  total_output_tokens INT,
  total_cost_usd DECIMAL(10,4),
  by_model JSONB,  -- {claude-sonnet: $0.45, claude-opus: $1.20}
  by_feature JSONB  -- {analyze: $0.80, draft: $0.85}
);

Dashboards:

• Top 10% spenders (daily/weekly/monthly)
• Cost-per-user distribution histogram
• Unit economics: cost-per-user vs. plan-revenue-per-user
• Feature cost breakdown (which features are expensive?)

Weekly cost-attribution report:

• Emailed to product + exec
• Identifies: top spenders, cost anomalies, features driving growth in spend

Layer 5: Eval Pipeline

Tool: Custom pipeline using Langfuse datasets + Claude as judge

Regression eval (runs on every prompt change):

• Test set: 50 curated real user queries with known-good outputs
• Run each through agent with new prompt
• Use Claude Opus as judge: rates output vs. expected on 5-point scale
• Flag regressions: any query scoring <3 when expected ≥4
• Block deploy if >2 regressions

Drift eval (runs daily):

• Sample 100 real production invocations
• Have Claude Opus score them on 5-point scale for quality
• Track rolling 7-day average score
• Alert if score drops >0.3 below baseline

Eval implementation:

async def run_eval(dataset_id: str, agent_version: str):
    dataset = langfuse.get_dataset(dataset_id)
    results = []
    for item in dataset.items:
        response = await run_agent(item.input, agent_version=agent_version)
        score = await judge_with_claude(item.input, response, item.expected_output)
        results.append({'item_id': item.id, 'score': score, 'response': response})
        langfuse.log_eval_result(dataset_id, item.id, score)
    return summarize(results)

CI integration: GitHub Actions runs eval on every PR that touches prompts.

Layer 6: Alerting

Tool: Datadog (existing) + PagerDuty (existing)

Alert categories:

1. Availability alerts (P0):

- Agent invocation error rate > 5% over 10 min → PagerDuty

- Agent invocation p99 latency > 30s → PagerDuty

- Full service down (zero invocations in 5 min during business hours) → PagerDuty

2. Cost alerts (P1):

- Hourly cost > $500 (≈ 10x normal) → Slack #ai-ops

- Daily cost > $3,000 (≈ 5x normal) → Slack + PagerDuty

- Single user cost > $50/day → Slack (investigate possible abuse)

- Weekly cost spike detection (3σ above 7-day rolling mean) → Slack

3. Quality alerts (P2):

- Daily eval score drops >0.3 → Slack #ai-eng

- User-reported quality complaints spike (>3/hour) → Slack

- Hallucination detection (user reports 'wrong' feedback) > 5% over 1h → Slack

4. Tool alerts (P2):

- Tool call error rate (HubSpot/GA4 integration) > 10% over 15 min → Slack

- Tool call latency p99 > 10s → Slack

Stack Map

Your specific stack:

Total new spend: ~$550/month. Well within $2-5K budget.

Dashboards

Dashboard 1: Agent Health (primary on-call view)

• Invocations/minute
• Error rate
• Latency P50 / P95 / P99
• Tool call success rate (by tool)
• Active user count

Dashboard 2: Cost Attribution

• Cost today / this week / this month
• Top 10 spenders today
• Cost distribution histogram
• Cost-per-user vs revenue-per-user (unit economics)
• Feature cost breakdown

Dashboard 3: Quality

• Daily eval score (rolling 7-day avg)
• User thumbs-up/down rate
• Hallucination report rate
• Regression eval pass rate (CI)

Dashboard 4: Tool Performance

• Tool latency (HubSpot / GA4 / CMS)
• Tool error rate (per tool)
• Tool call volume

Runbook Integration

When an alert fires at 3am:

1. On-call goes to Agent Health dashboard first — confirms scope (single-user vs systemic)

2. If systemic: check Dashboard 4 (Tool Performance) — often issue is upstream

3. If quality alert: check Langfuse traces for sample failures — reproduce + investigate

4. If cost spike: check Cost Attribution dashboard — identify spiking user, rate-limit if abuse

5. If eval regression: roll back last prompt change, investigate in AM

Runbook links in every alert to direct on-call to right dashboard.

Implementation Roadmap

Week 1: Foundation

• Deploy OpenTelemetry in Python backend
• Integrate Langfuse with Anthropic SDK
• Structured logging with LLM-specific fields
• PII redaction middleware

Week 2: Tracing + Cost

• Tool-call spans
• user_costs Postgres table + hourly aggregation job
• Dashboard 1 (Agent Health) live in Datadog
• Dashboard 2 (Cost Attribution) live in Datadog

Week 3: Evals

• Build test dataset (50 curated queries, known-good outputs)
• Implement regression eval
• CI integration (GitHub Actions)
• Start collecting user feedback (thumbs up/down)

Week 4: Alerts + Polish

• All alerts wired in Datadog + PagerDuty
• Runbook documentation
• Team training session
• 2-year S3 archive for compliance

Post-week 4: drift eval + hallucination detection + prompt versioning (roadmap items).

Key Takeaways

• 6 observability layers: invocation trace, tool-call spans, prompt-response logs, cost attribution, eval pipeline, alerting. All essential for production.
• Your stack: OpenTelemetry + Langfuse + Datadog (existing) + S3 (compliance archive). ~$550/mo new spend.
• Per-user cost attribution is critical for your B2B model. Top 10% users likely drive 80% of cost. Dashboard surfaces this for pricing decisions.
• Eval pipeline on every prompt change prevents silent regressions. 50-query test set + Claude Opus as judge + CI integration blocks bad deploys.
• Implementation: 4 weeks. Start within 2 weeks — retrofitting observability at 40K invocations/day (your 6-month projection) is 5x harder than building now at 8K/day.