When you build products around GenAI, you quickly realize the old rules don’t apply - traditional QA won’t cover you. The product isn’t binary; it’s probabilistic. A tiny tweak in a prompt chain can ripple through the entire experience.

When your product’s core value depends on an LLM’s ability to reason, "hoping it works" isn’t a strategy.

In this talk, I’ll share how we moved from vibe-based decisions to building real confidence in our product. I’ll walk through the messy, iterative process we went through, and how evaluations became a core part of how we build, test, and improve AI-driven experiences.