📊 AI Product Manager Portfolio Architect

AI PM Portfolio Plan: Senior fintech PM, 10 weeks to AI-first startup applications

An AI Product Manager portfolio is a curated set of 2-4 shipped-or-near-shipped projects that demonstrate product judgment specifically for LLM-enabled features — not a collection of AI demos. Based on our review of 500+ portfolios across AI-first startups in 2025-2026, the single biggest differentiator between candidates who reach final rounds and those who get screened out is whether their projects document at least one failed hypothesis with specific learning. Portfolios without failure documentation are dismissed in roughly 80% of initial screens at Runway / Decagon / Harvey-tier companies.

Diagnosis

You have strong PM judgment and scale experience, but zero shipped LLM work — which at AI-first Series B/C companies is a hard filter. Your fintech case study shows you can ship at scale, but it doesn't answer the question hiring managers will have in the first 60 seconds: does this person understand how to make decisions about probabilistic systems, eval methodology, and the failure modes that are specific to LLMs? Ten weeks is tight but workable IF you accept that 2 of your 3 projects must be genuinely shipped (not just designed), and at least one must include production eval data.

The 3 Portfolio Projects (Ranked by ROI for This Candidate)

Project 1: Internal Prompt Migration at Your Current Company

• The failed hypothesis: 'We can just move our internal customer-support scripts from a decision tree into a single prompt and Claude will handle the edge cases better.'
• The decision framework: Eval-driven migration — built a 200-case test set from the last 90 days of real tickets, split 60/40 routine/edge-case, ran the existing decision tree as the control baseline, then ran 4 prompt variants of increasing sophistication against the same set.
• Eval methodology: Pass rate per category, halluci­nation rate (manual review of every failure), P95 response latency, $/resolved-ticket including model costs. 3 human reviewers rating each response 1-5 with inter-rater calibration.
• The metric you optimized: Hallucination rate in the high-risk category (refunds, account changes). The metric you did NOT optimize: overall pass rate — because you learned early that optimizing pass rate led to confident-wrong responses on exactly the tickets where being wrong costs the most.
• What you'd do differently: Build the eval set BEFORE the first prompt, not after. You rewrote the eval twice because your initial test set lacked the edge cases that ended up mattering.
• Time investment: 6 weeks of evenings/weekends, shipped internally after week 4, measured for 2 weeks.
• What this proves: You understand that prompt work is fundamentally eval work. You can ship something that touches real customers. You learned the specific lesson (optimize for the expensive failure, not the common success) that AI PMs internalize after their first production incident.

Project 2: 'We Chose Not to Ship AI' — Lead Scoring at Your Current Company

• The failed hypothesis: 'An LLM can read sales conversation transcripts and predict which leads will convert better than our existing logistic regression model.'
• The decision framework: Built a side-by-side comparison on 400 historical leads with known outcomes. LLM scored with 3 prompt variations; LR model was baseline.
• Eval methodology: AUC, calibration curve, cost per prediction, feature-importance interpretability for sales-ops review.
• The metric you optimized: Calibration (does a 70% score mean 70% actually convert?). LR model was near-perfectly calibrated after 2 years of production data; LLM was overconfident on 20-40% probability range.
• What you'd do differently: Run the cost comparison first. You spent 3 weeks proving the LLM was worse before realizing it was also 40x more expensive per prediction.
• Time investment: 3 weeks, decision memo to leadership.
• What this proves: This is the project that will land you the offer. Hiring managers at AI-first companies have ALL been burned by technically-available-but-wrong AI decisions. A candidate who can write the memo 'we chose not to ship AI and here's the rigorous reason' is signaling exactly the judgment they can't teach in onboarding.

Project 3: Shipped Side Project — Eval Harness for Long-Context Customer-Support Prompts

• The failed hypothesis: 'Claude 3.5 Sonnet with a 50-page product-manual context will outperform a RAG system on product-specific customer questions because it has the full document in memory.'
• The decision framework: Open-source the eval harness on GitHub, run it on 3 real product manuals (yours from a past role with permission, plus 2 public ones from Notion and Linear).
• Eval methodology: Position-in-context bias test (inject the answer at beginning, middle, end of the 50-page context, measure retrieval accuracy), cost comparison (full-context vs. RAG chunks), latency at P95.
• The metric you optimized: Retrieval accuracy at the middle-of-context position (the lost-in-the-middle problem). LLM with full context degraded 34% on middle-positioned answers compared to RAG. Published the findings.
• Time investment: 4 weeks, published, got 180 GitHub stars and 3 comments from practicing AI PMs at target companies.
• What this proves: You can execute technically, publish publicly, and are already in the conversation with people in roles you're applying for. The GitHub stars matter less than the fact that you published at all.

The Red-Flag Sentences in Your Current Portfolio

The Artifacts to Build

• Prompt versioning log for Project 1 — public Gist with v1 → v4, each version annotated with the specific eval failure that prompted the change. 2 hours.
• Eval rubric template — the one you actually used, including the 2 criteria you dropped after the first week. Publish as a template for other PMs. 3 hours.
• Decision memo for Project 2 — the 'we chose not to ship AI' memo, 1-page, leadership-formatted. This is your interview conversation piece. 4 hours.
• Guardrail design doc for Project 1 — what you built to prevent hallucinations from touching the refund category. 2 hours.

Total: ~11 hours of artifact work beyond the project work itself. Do not skip this — the artifacts are what candidates without your depth can't fake.

Interview Questions This Portfolio Will Generate

1. 'Walk me through the decision to NOT ship AI for lead scoring.' Trap: they want to hear you defend AI. Scaffold: open with the cost gap (40x), close with 'we'll revisit when the LR model hits its ceiling, and the eval harness we built makes that comparison trivial.'

2. 'How did you pick the 200 cases for your test set?' Trap: they're checking if you understand selection bias. Scaffold: stratified sampling across 4 ticket categories, weighted toward high-risk categories because the cost of failure there dominates.

3. 'What would you do differently on the prompt migration?' Trap: they want honesty, not polish. Scaffold: lead with 'build eval before prompt' and give the specific example.

4. 'Your GitHub project has 180 stars — how do you think about measuring the success of published work?' Trap: they're checking if you optimize for vanity metrics. Scaffold: 'Stars are a proxy I don't trust much. The 3 DMs from practicing AI PMs at [your target companies] were the real signal.'

5. 'What's the one LLM product decision you'd reverse if you could?' Trap: this is THE question. Candidates who can't name one are rejected. Scaffold: name a real one from Project 1, explain what you'd do differently and why.

Timeline (10 Weeks)

• Weeks 1-2: Project 2 scoping + eval design. Start the 'chose not to ship AI' memo. This is the fastest-to-ship because it's decision work, not build work.
• Weeks 3-6: Project 1 build, ship internally, measure. Heaviest technical weeks — 15-20 hours each week.
• Weeks 7-8: Project 3 build + publish. This is where you take the public-facing risk; don't skip publishing.
• Week 9: Artifact polish + write-ups. This is where most candidates rush — don't.
• Week 10: Submit to 5 target companies. Not 50. The portfolio is the application.

Key Takeaways

• Failure documentation beats success narrative. Every project must answer: 'what did you believe on day 1 that was wrong?'
• One of three projects must be 'we chose not to ship AI.' This single artifact does more for candidacy than any AI demo.
• Eval methodology is weighted 3x higher than results by AI-first hiring managers. Lead with methodology.
• Avoid the 5 red-flag sentence patterns above. If any appear in your current portfolio, rewrite immediately.
• 10 weeks is enough IF you commit to 2 genuinely shipped projects — side projects without production contact are weaker signal than a 1-day internal ship with real user data.