🧑‍💻 Prompt Engineer Interview Prep

Interview Prep: Senior AI Engineer at Klarity (Series C Legal-Tech)

Calibration

A hiring manager round is a credibility audit: the Director of AI is deciding in 45 minutes whether you've actually shipped production LLM work or whether you're a strong backend engineer who did a course. Based on 40+ hires I've run, candidates from your background (strong SWE + one real ship + one course) convert at roughly 25% at Series C AI-native companies — not because the coding is weak, but because the LLM-specific judgment isn't calibrated yet. The Slackbot counts. The Notion RAG does not. Klarity's bar at 10M+ documents and 40k queries/day is a production-scale bar, and the Director will probe hard on failure modes you've likely never hit. Your realistic shot is to be radically honest about what you've shipped, show sharp thinking on what you haven't, and avoid the red-flag answer below at all costs.

The 9 Technical Questions You Will Be Asked

1. "Walk me through how you'd design the RAG pipeline for 10M legal contracts."

• What they're testing: Whether you know the failure modes that appear above ~100k documents.
• Strong answer: Hybrid retrieval (BM25 + dense) because legal has rare entity names vector search misses; hierarchical chunking (clause-level + document-level); metadata filters on jurisdiction/contract-type before vector search to cut recall set 10-50x; reranker (Cohere or cross-encoder) on top 50 → top 5; cite-or-abstain prompt pattern.
• The trap: Saying "I'd use LangChain's default retriever with 1024-token chunks." That's a demo answer. Chunking contracts at 1024 tokens shreds clauses mid-sentence.

2. "How would you build an eval for contract extraction accuracy?"

• What they're testing: Whether you understand that LLM-as-judge alone is insufficient for legal.
• Strong answer: Three-layer eval: (1) golden set of 200-500 hand-labeled contracts by actual lawyers, (2) regression suite run on every prompt change with exact-match on structured fields + LLM-judge on free-text, (3) production shadow eval sampling 1% of live traffic with weekly human review of disagreements. Track precision/recall separately — legal cares far more about precision.
• The trap: "I'd use RAGAS" with no further detail. RAGAS metrics are a starting point, not an eval strategy.

3. "You're seeing 8% hallucination rate in production. Debug it."

• What they're testing: Production debugging instinct.
• Strong answer: First isolate — is it retrieval failure (right answer not in context) or generation failure (right answer in context, ignored)? Sample 50 failures, label each. If retrieval: check chunk boundaries, rerank quality, query rewriting. If generation: tighten prompt, add cite-or-abstain, consider a smaller fine-tuned model for extraction. Instrument first, guess never.
• The trap: "I'd engineer the prompt to say 'don't hallucinate.'" This is the single biggest tell of inexperience.

4. "When would you fine-tune vs. stay with prompting?"

• Strong answer: Fine-tune when (a) you have >10k high-quality labeled examples, (b) latency/cost demands a smaller model, (c) the task has stable structure. Prompt when the task evolves weekly or you're still figuring out the spec. For Klarity specifically: extraction is a fine-tuning candidate; agentic review flows are not.
• The trap: "Fine-tuning is always better." Wrong in 2026 — frontier models with good prompts beat most fine-tunes on reasoning tasks.

5-9 (abbreviated for length): Model routing (GPT-5 vs. Haiku by query class), latency budgets at 40k qps (caching, speculative decoding, streaming), agentic design for multi-step contract review (tool boundaries, failure recovery), prompt versioning/rollback infra, cost modeling (your $/query at current scale and what breaks at 10x).

The 2 Behavioral Questions The Hiring Manager Will Ask

1. "Tell me about a time an LLM output was wrong in production and what you did."

They're listening for: Do you have a real story, or a hypothetical? Your Slackbot is the only real answer you have. Use it. STAR: PR summarizer started hallucinating file names after GPT-4o model update → 3 engineers complained in one day → you added a post-hoc check comparing summarized filenames against the actual diff, flagged mismatches, logged them, rolled back to previous model while investigating. Honest, small-scale, real. Do not invent scale.

2. "You have 3 months to improve extraction accuracy from 82% to 90%. How do you spend them?"

They're listening for: Prioritization and humility. Strong answer: Month 1 — instrument. I can't improve what I can't measure. Build the eval harness, get 300 labeled examples from lawyers, establish true baseline (probably not 82%). Month 2 — attack the biggest error cluster, not the whole problem. Likely retrieval, not generation. Month 3 — ship, monitor, iterate. Name explicitly what you would NOT do: "I would not spend week one rewriting prompts."

The Red-Flag Answer

The phrase: "I'd iterate on the prompt until it works."

In our experience reviewing 40+ AI engineer interviews, this single phrase — or its cousins ("I'd A/B test prompts," "I'd keep refining until outputs look good") — is the clearest signal a candidate has never owned a production LLM system. It reveals three things at once: (1) no eval discipline, (2) no concept of prompt regression, (3) "looks good" as a quality bar. Roughly 70% of candidates who use this phrase are rejected in the hiring manager round.

Say instead: "I'd define the eval first, establish a baseline number, then change one variable at a time and measure against the eval. Prompt changes without an eval are just vibes."

Portfolio Gaps & 3 Projects To Build

Project 1 — Eval harness for your own Slackbot (2 weekends). Retroactively build a 100-example golden set of PRs + ideal summaries, write an LLM-judge eval with rubric, run it against 3 model versions. Deliverable: a blog post with a results table. Signals: you know evals aren't optional.

Project 2 — RAG over SEC 10-K filings with citation accuracy (3 weeks). Financial filings are legal-adjacent and public. Build retrieval, measure citation accuracy (does the cited passage actually support the answer?), publish numbers. Signals: regulated-domain judgment.

Project 3 — Model router with cost/latency dashboard (1 week). Route queries between a small and large model based on classifier, track $/query and p95 latency. Signals: production thinking, not demo thinking.

The One Question You Should Ask Them

"What's your current eval strategy, and what's the biggest gap in it?"

This question does three things: signals you think evals-first, forces them to be honest about tech debt (every AI team has eval gaps), and gives you intel on whether this team actually has their practice together. If the Director can't answer specifically — vague answer about "we use LLM-as-judge" — that's your signal about engineering culture. If they name specific metrics, disagreement rates, and what they're working on next, this is a team worth joining.

The Bottom Line

• Your biggest risk is overclaiming. One shipped Slackbot is a real data point — use it precisely, don't stretch it.
• Evals are the entire game at this level. If you walk out having said the word "eval" fewer than 5 times, you lost.
• The red-flag phrase costs offers. Memorize the replacement sentence.
• Build Project 1 this weekend. Even if you don't get this role, it's the single highest-leverage thing you can do for the next 5 interviews.
• Expected outcome: With honest calibration and the answers above, roughly 25% → 45% conversion on this round.