Playing from timestamp
Video starts at 16:10
Channel
Interviewed Person
Malte Ubl
"Does this program compile? Does this program start up? Does this program render the first page without client-side error? These very trivial evals are extremely valuable."
"You're in this constant loop that is actually very vibes driven, because there's the ground truth of what actually matters driven by the failures and successes of your production users."
"We try to upgrade failure modes that we observe in production into something where we can measure how well we do based on those previous failure modes."
"You use the agentic loop for fixing difficult things and not for fixing trivial things."
"The agent might find out it made a mistake and totally knows how to fix it, but it's fixing it using a model that has 900 tokens per second which is still snail's pace for trivial errors."
Hi, welcome to the Lin Space Lightning Pod. We are here with for a very special episode because this is not really about a company or some launch that someone's doing. This is actually just a debate that I accidentally set off about coding agents mostly not having evals or they do have evals. It's mostly it's not really like standardized or and entirely it's a lot of it's vioded and I think a
lot of it is based on the fact of this clip that I had from Boris of cloud code saying that they basically just vibe it and that's a 500 probably higher than that $500 million business uh that doesn't have evals. So I wanted to invite Anker and Malta from Brain Trust and Verscell uh to just talk about like the state of EVAL's here and like the EVEL's debate. You guys have your own perspective. Anker, you got a bit in a in a bit of like a blogging back and forth on evals. What are you guys feeling like what's maybe a position
statement from each of you? >> Yeah, I I mean I can start a little bit like I think the like one maybe important background on me is also that I worked on Google search just before leaving for Vercel. It's almost four years ago, but like I do vaguely remember it. And obviously that was also before JGBD moment and before kind of engineering becoming a mainstream thing. And so I left a world in which I was doing evolves every single day and kind of thinking that that would be it, right? Like that I wasn't really seeing
that in my future as someone joining an infrastructure company. And so it is both kind of fun to see it coming back and I think it does give me a little bit of extra experience for uh what like how it helps and how it doesn't help. One thing I tell people for sure is that like you have to put in the work, right? And and even vibe checking like as the eval like is important like if you are a product manager in Google search, you're probably doing like 500,000 searches a day. Um and if you are but then you want
to do revals because if you want to do 50,000 searches, you're not going to do that yourself. And I think there's something very similar here where essentially you just to see like, hey, I want to know if I'm doing well and how fast can I find out and like, yeah, the vibe check can tell you really fast, but if you want to know tomorrow, then the only way to do it is to have someone do the 50,000 searches or the 50,000 coding exercises, right? And if you want to know in in let's say in in 3 weeks, you can write an AB test. And if you don't care, then you you know do neither of those things. But but it is like I I think like there the way I think about
EVA is essentially like it's the thing that that can tell me tomorrow whether my change is good and um I can operate without that knowledge. But it's but it's super super helpful. >> Yeah, I have very little to add to that. I I um strongly agree. I think like fundamentally if you're building an AI product, you're dealing with this like non-deterministic magic and you don't you can't like type whether it's a model or prompt whatever you can't really type something and know what's going to happen. So you need a feedback loop and I think that fundamentally evaluate
was about are both the most challenging to build feedback loop and also the most efficient once built. And then I think AB tests are a little bit less challenging to build and a little bit less efficient. And then like pure vibes are basically like zero effort to build but then also the the least efficient. And you have to do a little bit of all of these things. And I think what I've seen both firsthand from building Loop, which is our own agent and brain trust, but much more secondhand from working
with Malta and a bunch of really great companies building AI products, is that the best people are extremely deliberate about the investment in each of the feedback loops and they're constantly questioning whether they've put enough energy into each of the feedback loops and making them um efficient. And then I think coding is it's like such an interesting topic for a few reasons. The first is it is the use case that has the most product market fit in AI I think other than ChatGPT and so it is just an important and interesting use case. The
