Shipping AI product to Production: my learnings on Evaluation
The past few months I've been working on an AI tooling project - not just prototyping, but taking it all the way to production. The surprising-but-not-surprising part has been evaluation. The surprising part is that it's not part of any traditional product development framework so nobody's prepared me for it. The not surprising part - this is simply the new reality of building with AI as a product manager.
So, What Is "Eval"?
Eval - or Evaluation is the guardrail that ensures your AI product performs as expected.
Unlike traditional software, where results are deterministic - a feature either works or it doesn't, a UI either renders correctly or it doesn't - LLMs don't follow a strict, repeatable path. Run the same AI pipeline ten times, and you'll likely get ten slightly different outputs. If you've worked with an AI agent, you already know this.
That unpredictability creates real challenges when you're building something high-stakes, where one miss can cause real harm. Healthcare is full of exactly that kind of case. Which is why evaluation isn't optional but a whole ballgame. We need to constantly evaluate the outputs to derisk and ensure the AI tool is providing values.
Learning by Making Mistakes
I listened to podcasts, read a bunch of newsletters and watched YouTube videos on the Eval - it’s one of the buzz words that’s hard to miss nowadays. But I'm not great instruction follower in general, so I ended up learning by making enough mistakes along the way.
The project I've been working on is an AI notes-generation tool for veterinarians during consults. The concept is simple, and if you've used Zoom AI or a similar tool, you can probably picture how it works. However, it's easy to build a proof of concept - I could probably vibe-code a demo in one hour, but taking it to production is actually way more challenging than I expected.
The question I’ve been asked a thousand times is: how do you get confidence that AI will consistently produce high-quality results that meet expectations? And the hard part I ask myself: even if I’m confident, how can I convert the quality into a number that number everyone can understand? What does a "90% quality bar" actually mean?
If I were still building this for my own startup, I'd ship a tool with a clean UI, hand it to my clinical staff, and ask them to rate it after every consult. Fifty consults later, I'd have ratings, and I'd sit down weekly to talk through what to improve and tune the prompt from there.
But I'm not in a startup anymore. I'm at a mid-size corporate with real expectations around resourcing and alignment, especially when our roadmap is a battlefield, and clinicians’ time is top priority to protect.
That reality breaks the problem into three distinct stages during my AI notes-generation tool development:
Pre-project evaluation: proving there's a real case to invest resources at all
Development-stage evaluation: proving the tool is ready to ship
Post-production evaluation: proving it stays good once it's live
Evaluation Before the Project Exists
We have an internal process called committee intake, where you bring a project idea to cross-functional stakeholders and leadership to get approval to work on it. When I brought the AI notes-generation tool to intake, I was competing for resources against initiatives tied directly to core business growth.
So the real question I had to answer wasn't "is this a good idea" but whether this AI tool would actually beat the alternative of putting the same resources toward something else. I could calculate the time and cost savings if we hit a quality threshold, but first I had to show we could realistically reach that threshold at all.
Training or fine-tuning a model from scratch is expensive, so the burden of proof was on showing an off-the-shelf model could get us there. I used Amazon Transcribe and a Bedrock off-the-shelf model: take sample audio, generate a transcript, then run a customized prompt through the model to produce notes. Once I had the outputs, I compared the generated transcript against the original recording word-for-word and flagged every delta, then compared the generated notes against both the transcript and the doctor's actual notes. The goal it was understanding the gap and whether we had a path to bridge it. The key information I was specifically looking for are the details that can impose potential risk: medication names, medical terminology, anatomical terms, diagnoses.
The whole exercise was manual, with no fixed rubric yet. But the report gave stakeholder a clear understanding of the benchmark and where we expect to see improvement with resources.
Evaluation During Development
Once I had resources allocated, the question became launch readiness - what’s the bar we need to meet in order to ship it?
Step one was establishing a baseline using professional judgment. I built an evaluation framework (how clinicians would review outputs) and a scoring rubric (how I'd translate that into numbers), then worked with the AI dev team to build a pipeline which supports customized prompts converting input into our target output format.
I took those outputs and made them reviewable - starting in Excel, then moving to a Word-plus-Excel format. Then I had clinicians manually read outputs and rate risk level. I'd group the feedback, feed it back into the next prompt iteration, and repeat. The cycle continued until feedback narrowed down to personal language preference rather than substantive errors.
This process worked, but it was entirely manual and didn't scale. That's what pushed us toward an automated evaluation platform and we landed on Arize to set up LLM-as-judge evaluator.
Setting up an LLM-as-judge evaluator was foreign at first, and I learned two important lessons from bad experiments:
Keep the rubric simple. The general recommendation - which I learned the hard way from attempting to mimic a human evaluator - is to focus each evaluator on one dimension, and on detecting rather than scoring. Flag "this is wrong," instead of trying to make it to flag, evaluate, score all at once. This ensures your evaluator each measures a different dimension and you could continuously calibrate one dimension at a time.
Calibrate the evaluator before you trust it. Before using an evaluator on real results, you need to know if the evaluator itself is any good. I built a human-labeled dataset from the manual review results - it’s a dataset with clinician feedback annotated across a few dimensions: accuracy, completeness, and language/style. I started with these three, the most common feedback themes, though there's room to break this down further. I ran the same dataset through the Accuracy LLM judge and compared the annotation of accuracy issues. The results showed partial alignment, with real mismatches in both directions - sometimes the LLM catches misspelling that the human clinicians miss, and sometimes it fails to understand a nuance. So: update the dataset, repeat, recalibrate, until the evaluator's judgments actually tracked with clinician judgment.
Evaluation After Launch
Being production-ready means more than a good evaluator. It's unrealistic to manually export outputs, build a dataset, and run them through an evaluator every time. You need real observability: if the model starts underperforming or causing harm, you need to know immediately. If you ship a change and quality drops, you need regression testing — the same discipline as traditional software development.
That means traces and spans, and a monitoring dashboard. Arize lets you connect your pipeline directly, stream traces to the platform, and run your evaluators against live traffic — monitored across dev, staging, and production environments.
In the end, four things make up a real production evaluation pipeline:
Traces and spans: the footprint of every pipeline run
Code evaluators: deterministic checks that catch clear-cut errors early
LLM-as-judge evaluators: the harder-to-automate quality judgments
A monitoring dashboard : to track performance continuously, not just at launch
Getting to Launch
When it’s finally the time to launch - three core things make up the product: an intuitive UX for our users, an AI pipeline that meets our quality bar, and a full platform monitoring the LLM performance behind the scenes.
None of this was obvious going in. But if you're building something AI-powered for a high-stakes domain, evaluation isn't something to bolt on halfway through - think about it from day one. Start with what the ideal evaluation state looks like, then work backward into how you build toward it.
I'll go deeper on the specifics of LLM evaluation and traces in a follow-up post when I get the chance.
