AI Evals For Engineers & PMs

About three weeks ago, I started the course "AI Evals For Engineers & PMs" on maven.com. In all honesty, I wasn't sure what I was getting myself into. Sure, I'm a seasoned, product-minded engineer, and I was certain that the material would prove useful, especially considering that proper evaluation is still quite niche, even more so in Spain. Becoming something of a local go-to person in that particular field sounded rather appealing. Oh boy, was I not prepared for this. I should have brought a bigger boat. The amount of material, thought processes, tips, tricks, war stories, tools, recommendations and anecdotes is astonishing. There’s just so much to unpack. I found myself rewatching every session several times to make sure I didn’t miss even the tiniest detail. When it all clicks in your brain, it suddenly feels so intuitive. Guess what? Thinking in terms of evals reinforces your product mentality. And it paves the road to continuous testing AKA true observability. I believe I'm a much more complete engineer now than I was before starting this course. I will never look at logs, traces or assertions the same way again. Thanks to the titans Hamel H. and Shreya Shankar for leading this course. I've learned a lot from you folks.

This course provided a hands-on path to understand how AI products can be improved and how critical evaluations are for any business. The best lesson from me was having to understand the nuances of evaluation systems, and how a deep understanding of your data can make massive improvements on your results. One of the best, if not the best, courses on AI!!

What I loved about this course is that it didn’t just explain concepts, it gave me concrete steps I could apply right away. Shreya and Hamel focus on what actually works in practice, not just theory. I now have a clear, structured process for evaluating and improving our AI systems, and I’ve already started using it in our workflow. If you’re tired of vague advice and want something you can actually do, this is it.

As a CEO of an AI agent company, the quality of our product is of highest priority. And without this course I was ending up spending too much time. This course easily saved my 50 hours of research and trial and error.

Exceptionally clear with a relentlessly practical focus. Hamel and Shreya don't sell you on fancy tools or generic benchmarks. Instead, they distill hard-won experience from hundreds of client engagements into what actually works for building reliable AI applications. They show you exactly what steps you must never skip, how and where you can and can't automate, and how to move from hoping your AI works to systematic understanding of your pipeline's real behavior. The focus is on looking at your actual data and identifying the failure modes that matter for your specific use case. The spicy takes and real-world case studies make it clear this isn't theoretical, and the accompanying course reader is invaluable. No fluff, no jargon, just the systematic framework you need to build AI systems that people can actually depend on in production.

Evaluations are the fundamental building block for building AI systems (at least if you want them to actually work). Hamel & Shreya’s course is the best out there to learn build AI applications that actually work.  I highly recommend this course To anyone working in AI.

This course reframes evaluation not as a final step, but as a structured process that’s central to building reliable AI applications. For practitioners, this shift in perspective is incredibly valuable—it provides a clear path to understanding system behavior, tracking progress, and making informed decisions across the AI lifecycle. Shreya and Hamel combine clarity with practicality. Through real-world examples and thoughtful exercises, they show how to identify failure modes, perform targeted error analysis, and align evaluation with product goals—without relying on complex jargon. A key takeaway: asking “Is our system improving?” requires more than metrics—it requires a repeatable, insight-driven process. This course gives you the tools to build that process. Highly recommended for engineers, PMs, and anyone serious about delivering trustworthy, high-impact AI systems.

Hamel and Shreya teach absolutely must-have skills for AI that is rarely taught elsewhere.   There are many courses that teach how to use tools and frameworks, but there aren't many courses that teach you the right process to follow.   The things they teach in this course are timeless and apply to any AI problem that you might work on now or in the future. 

This is one of the most practical courses I've ever taken and the LLM Evals companion book is fantastic! I was able to put the concepts directly into action throughout the course, which has already improved our AI evals dramatically.

This course is the basis for how we are designing evals for our AI SRE. The course let us go from vibes to quantitative metrics that let us systematically improve the AI.