Plan AI features as a product manager
I'm in the final stretch of preparing an AI project for release, and one thing has been quietly haunting me this whole time: the planning.
We designed the UX months ago. Then I spent a few more months just iterating on quality - trying to get the AI output to a place where different stakeholders are confident to start a project - tech needed to be convinced it’s a top priority, and business needed to know it would move a meaningful needle.
Then, over the past four weeks, we finally started development and everything came together fast. Almost too fast. Suddenly I was cramming the launch plan, stakeholder coordination, and reporting metrics- all the things I thought I'd had months to prepare into a matter of weeks.
I keep turning the same question over: how did I have so much time to plan, and still end up this overwhelmed in such a short window? In a time of AI, how does a product manager actually plan a project properly?
How AI project is different?
A typical feature roadmap goes something like: find a pain point, explore solutions, pick one, write requirements, get an engineering estimate, hand off to design, hand off to dev, build, launch, collect feedback. It's roughly linear. Once you commit to an idea, the rest is mostly about execution.
AI projects break that shape in one specific way: you can't control the quality going in. You don't actually know, at the start, whether the thing you're building will be good enough to ship. If we move linearly through design and dev, but the quality doesn’t meet threshold, we’d need to restart thinking how to improve it again. I’ve witnessed a different team moved onto Beta testing too fast and stuck in the iteration cycle for month which eventually led to the project being killed.
The real difference between an AI project and a traditional software is - we can’t predict the timeline spent on quality iteration - that’s prompt, model selection, pipeline optimization etc. So the best way to do it - meet your quality threshold first. But not all project could be evaluated standalone - talking to an AI agent or calling an API don’t always give you the full picture. We iterated a transcription pipeline several times by exploring different models, different diarization settings, differnet audio processing…you can’t do them easily with a simple input and output. So you need a proof of concept.
So the real AI project planning should be: POC → evaluation → development → evaluation → Alpha/Beta release → evaluation → release → evaluation. I can go on and on. But the core message is - we need to be even more agile and have the right expecation of fast iteration cycle.
What I'd Actually Do Differently
Here's the revised gates, folded into the same passage:
What I actually needed wasn't more discipline about iterating - it was a clear bar for when to stop iterating and move forward. Three of them, really, one for each phase the loop feeds into:
POC & evaluation start: a minimal proof of concept exists, and the evaluation loop is actually running against it.
Threshold met: quality has cleared the bar where UX and tech can both start real discovery work with confidence, instead of waiting on an open-ended "still improving it."
Release: ensure a controlled alpha, beta, and full rollout sequence.
The mistake I made wasn't spending months in the loop. It was not building a plan with timeline, stakeholder updates, milestones around the loop as its own real phase of work. I was mentally treating those months as "not started yet," when they were actually the most important part of the project.
For My Next Project…
Here's how I'd think about this differently next time, using a hypothetical: say the next thing I want to build is an AI concierge tool for customers.
Start with the problem, before touching AI at all. What's the actual pain today - say, customers dropping off a rigid decision tree because they can't articulate what they need in the categories it offers. What metric moves if this works? Conversion! And before going further: could a non-AI solution like decision tree get most of the value for a fraction of the cost? If yes, this isn't an AI project yet. If the honest answer is no, that's what makes conversion a real North Star for an AI solution specifically.
Then scope the AI component narrowly. “Concierge” isn't one model, it's three distinct jobs: identify intent, make a recommendation, support the purchase. Each carries different risk. Recommendation and purchase support are probably close to solved with an off-the-shelf model. Intent extraction from open-ended language is where the real risk lives. so that's the one thing worth isolating and testing standalone, not the whole pipeline at once.
Define "good enough" before building anything. Write the rubric with actual stakeholders before there are outputs to bias the conversation: what's a correct intent match, what's a partial miss, what's a hard fail. Agree on a real number, for example 90% correct match, before results exist, not after, when everyone's negotiating whether 85% is close enough.
Build the smallest possible POC and run the loop. No UI, no pipeline polish, just enough to feed inputs and inspect outputs against the rubric. Test in small batches of 10-20 first, not the full 100 at once; early batches tell you if you're in the right neighborhood before you sink real time into a large-scale eval. Track why things fail, not just the pass rate - that's what actually drives the next iteration.
Gate: does intent extraction alone clear threshold? If not, stay in discovery. Don't let anyone start scoping UX or dev estimates yet. That's the exact trap I fell into. If yes, only then scope the full pipeline together and re-run evaluation at that level, since each piece working alone doesn't guarantee the combination does.
Once the pipeline clears threshold, unlock UX and dev together. This is the real Threshold-met gate. UX can design around known model behavior instead of guessing, and dev can scope real architecture instead of a speculative one. The evaluation framework becomes a real deliverable here too, built to run in production, not bolted on right before launch.
Then release in stages, not all at once. 5% of customers, a month of monitoring, explicit go/no-go criteria set before the rollout starts. What failure rate would mean pulling it back. Have a rollback plan ready. AI features are the ones most likely to need a fast reverse if quality drifts once it's live.
The one discipline that matters most across all of this: don't let calendar pressure move you to the next phase before the eval gate is actually cleared.
Of course I don't think I've fully cracked this yet. As AI tools speed up development itself, I suspect even this framework will need to change, and I'm mostly figuring it out one project at a time.
