Prática de Shadowing: Building the best agentic analytics harness: Powered by Claude, built with Claude Code - Aprenda a falar inglês com vídeo

Thanks, everybody.

Great to be here today.

Omni is an AI analytics platform, platform, and today I'm going to talk a little bit about how we build with Claude

and what we've built with Claude and how Claude powers that.

So start just with how Claude has enabled us.

I'm the CTO.

I run the engineering team at Omni.

We have a team of about 25 engineers.

This is a slide of our commits to the main branch of our repository over time.

I think it kind of speaks for itself.

One of the things hidden in here, kind of a very small piece of that line, is my own commits.

I think as CTO of a growing company with hundreds of customers, I sort of assume that at some point I'd have to stop writing code,

and I thank Claude very much for making me still able to do some software engineering from time to time.

So that's been a really fantastic, sort of unexpected benefit of this rollout.

And to speak a little bit about how this went, earlier in 2025, we said to the team, I don't know when and I don't know how, but I know our jobs are changing.

So let's just start experimenting, start using these tools, figure out what works.

And we did a bunch of it with some sort of fits and starts.

And when Cloud Code with the Opus model released, that was when some of our senior engineers said, wait a minute, no, this is real.

This is actually helping consistently.

And it's kind of been off to the races since then.

And starting around, I sort of felt like everybody went away for the holidays

and then came back in January and had sort of skilled up

and figured it out and was ready to start hitting the ground running with Cloud Code.

And you can see the slope of the line from there.

So that velocity is a big part of our culture at Omni.

We have a core value called Ship It.

We also have a core value around transparency.

So you can actually see, if you go on our website, omni.co, there's a top line navigation page to our demos.

Every Friday at Omni, we have an all-hands meeting.

It's the most important meeting of the week.

We do about 10 minutes of announcements, shout-outs to each other.

And then we do 50 minutes or increasingly more than 50 minutes of demos.

We record all this and our CEO's favorite job is Saturday morning, he wakes up, he cuts those demos,

posts them on YouTube, and shares them with the world to see.

So if you're curious about what we're up to, you can go to omni.co.

This has been a really cool way to show our customers, our prospects, the community around us sort of how we're thinking and how we're building.

And that speed and velocity has been a huge benefit for us as a company.

So let's talk a little bit about what we have built with Claude.

So Omni's AI analytics, we let you talk to your data, right?

So how does that actually work?

So a user comes in and asks a question.

In Omni, we're using Claude to actually translate that question into a semantic query.

I'll explain more what that means later.

We have a semantic layer.

And that is, think of that as sort of a translation layer that sits on top of your data warehouse, database, or maybe multiple of those,

to actually provide some additional benefit about how to use the data, enforce it, sort of give it a map for how to actually translate that data,

or excuse me, that query into SQL that then runs against the warehouse.

We'll go into more of this later, but that's just a high level view of kind of how the system works.

And this is important because Claude is incredible at answering questions, but you need to tell it more about your business if you want it to answer questions about your business, right?

It can tell you incredibly deep insights about how businesses work generally, but if you want it to know about how your business works,

you need to tell it not only how the business works, the terminology you use, and then of course, if you're asking a data question,

you need to tell it how your data looks and how that works.

And so that's where we come in.

That's the problem we're solving.

And it's subtly difficult, right?

Like, even last quarter means drastically different things at different companies.

Even in our own company, you know, in the product and engineering organization, last quarter refers to the calendar year.

In our sales team, it refers to our fiscal quarter.

So all of that needs to actually get coded in to the context

and awareness and even the sort of data layer

and definitions of the data so that it can be used appropriately to ultimately get you the right answer to your question.

So this semantic layer, what's it doing?

Like I said, it's a translation layer that sits on top of the database.

It's doing a few things.

One, it's curating the data.

It's very easy to sort of come up with a toy demonstration of how an LLM or,

frankly, a human can get correct answers on top of 10 data sets in a database.

Real company data warehouses and databases are not like that.

They have tens of thousands, hundreds of thousands of data sets, sometimes more, and all of them have, there's a hundred revenue tables, right?

There's a hundred opportunity tables.

It's very unclear how to actually stitch those things together and use them in the correct way.

So this is one of the benefits that our semantic layer provides, is it allows you to define how to use these things together and also curate it.

Say, hey, listen, this is the one that matters, ignore those other ten.

With that, it's a way to encode the context.

And I think one of the things that we've learned is that context is great,

but context localized to what the actual definition that the context refers to makes it all the better, right?

And so I think about this, we're at code with Claude.

If you use Claude code, which I assume most of you do, you have your Claude.md files, right?

And so the more you can do to sort of localize

that context context next to the parts of the code that it applies to, the better results you're going to get.

That's what our semantic layer does as well, is it helps you provide that context next to the field definition that it applies to, rather than in a separate file over somewhere else.

And then finally, permissions, right?

It's a permissions layer.

Make sure that people see the data they're supposed to see and don't see the data they're not supposed to see.

And inside of our application, this feedback loop is an important part of how this stays current and accurate.

Because guess what?

In a real organization, this stuff changes constantly.

And so our application provides a feedback loop where the next question

that gets asked of the data can then be fed back into the definitions, into the context for a continuous learning loop.

So this is our agent.

Name's Blobby.

If you look at Blobby, you could probably see this is a mature, professional, refined data analyst.

And that is what Blobby is today.

But Blobby hasn't always been that refined.

started building Blobby, let's say, about 18 months ago, and we've learned a lot along the way.

You know, Blobby's grown up quite a bit in the past 18 months.

So what I want to talk about is a little bit of how we develop the different phases

and also what we learned along the way that kind of helped increase the quality and capabilities of Blobby.

So just to ground this conversation, I just want to show you a really quick demo video of what Blobby does in real life.

100

So if we can cut to the video quickly.

101

So what we're going to see here is, like I said, ask questions of your data, blobby is sitting there dutifully waiting to answer your question.

102

And as we do this, you'll start to see some of the phases that we go through to actually break down the question and then answer it.

103

So, right, the first thing Blobby's doing is saying, all right, you're asking about PRs.

104

Blobby's very smart because Claude's very smart.

105

It knows that PRs refer to GitHub pull requests.

106

Let me go find that in your semantic model and figure out what data you're actually referring to.

107

Then it goes and looks at the values of the dataset, because we said we only wanted it from a particular repository, so it has to apply a filter.

108

It needs to know what that filter value is.

109

But when I type out questions to an LLM, I make typos all the time.

110

So it needs to do a little bit of fuzzy matching to make sure that it's actually finding the right thing.

111

And then it goes through, it generates a query, runs that query against the data warehouse, gets the results, provides a nice visualization,

112

and then does a nice little summary at the end to tell you what you're seeing.

113

Cool.

114

So just grounding you a little bit in what the actual experience is like.

115

So what did we learn along the way?

116

So the very first version of Blobby was basically single question, single answer.

117

And we quickly realized

118

that we needed to give a lot more metadata about how to use the data and how the data is typically used.

119

So we always had these label and description fields in our definitions, but we needed to add some additional context.

120

So we added this AI context concept, which is specifically for an LLM, hey, how should you use this?

121

When you're asked a question about this, you might want to use this reference

122

or this field just to kind of help the data team

123

and the administrators of this to actually steer in the right direction and ensure that you get a quality answer.

124

And then sample queries, very self-explanatory, really helpful to kind of ground it in, hey, this is a typical use case.

125

This is the query you would run to answer a question that like X.

126

And then finally, values.

127

This one was kind of subtle.

128

What we realized is, again, back to that example of, I'm asking a question about a certain repository, or in this example, asking a question about a region.

129

It's really helpful to give the LLM just a taste of what the values of that field are.

130

Because you can see, region, all values are EMEA, NAM, APAC.

131

So it can infer the next 10 values, because it sees, all right, These are abbreviations of regions of the world, right?

132

But it's useful for it to know, like, okay, this is an abbreviation, so if somebody asks for United States, I can just put US in there.

133

So this did a nice job of helping to improve the quality of the question and answers that we were getting.

134

But at this point, Bobby was still really not an agent.

135

So that was the next big leap here, was adding an agentic loop around this.

136

This is a big engineering effort.

137

We built our own agentic harness.

138

And it included this concept of tasks like all good agents have, included a lot of other stuff too.

139

I think one of the biggest things we learned was that the agentic loop was really great at recovering from errors.

140

So one of the earliest massive quality increases we made was to just A,

141

tell Blobby how to recover from errors and give it some budget to go do that.

142

And B, then go invest in providing great error messages

143

that were descriptive about what was happening and how you might fix it.

144

And that alone allowed the quality score to increase dramatically.

145

We saw our evals, like a lot of our more difficult evals, got a lot better once we did that.

146

At this time, though, because we were sort of in this mode of question and answer, we were using the Haiku model.

147

And the Haiku model is great, but once you get into these more elaborate, agentic conversations, it's just not designed for those, right?

148

And so we switched over to Sonnet, and we're showing a graph here of token consumption.

149

And the reason is twofold.

150

One, these are longer conversations, they're more complex, they consume more tokens.

151

That was by design.

152

Two, this was a really big unlock.

153

So we all of a sudden started getting our customers saying, wow, I just asked this question that either I never would have been able to answer myself, or even if I did,

154

it would have taken me hours.

155

And it just nailed it in two minutes.

156

And so the usage of Blobby started dramatically taking off at this point.

157

So at this point, our CEO, who is our loudest and most critical user, like all good CEOs are,

158

was telling us, listen, guys, I know this thing's really good, but it screwed up this question, go fix it.

159

And we all said, you know, Colin, LLMs are a little unpredictable.

160

Like you're just gonna have to accept that it's not always going to be perfect.

161

He said, not good enough, go fix it.

162

So, okay, fine.

163

And where this led us was, I think one of the big efforts that we undertook at this point was to say,

164

okay, let's really invest in understanding the traces and being able to see the traces of these, you know, quote-unquote, bad sessions.

165

And this led us to a series of major surgeries that we refer to as the Blobotomies.

166

And the Blobotomies really were traced back to what we were seeing in these traces.

167

When you look at the traces, you kind of get to see the inner workings of how the agent is sort of talking to itself

168

and reacting and responding in these loops.

169

And that really clarified why some of these seemingly kind of just bad random sessions,

170

you could actually start to see where they were rooted in real problems.

171

So an example of this was the original design of our agent might have been a little too clever.

172

And we had sort of an outer agent that was responsible for producing the task list.

173

It understood where all the data available to it.

174

but it was not in charge of query generation.

175

It had a sub-agent that was in charge of query generation.

176

And it felt like a reasonable design.

177

It was also handy because we could use that query generation sub-agent in a few other contexts.

178

But what we found once we started digging into these traces was that the sub-agent, its job was to generate one query based on whatever it was asked.

179

And the outer agent didn't know what was actually able to be answered in a single query.

180

So it would say, hey, subagent, go answer me a question about GitHub pull requests and support data,

181

and summarize these things.

182

And then the subagent would say, I can't answer that in a single query.

183

I can only, I would have to run multiple queries.

184

And so it's sort of, the light bulb went off when we started seeing this, that we have to be really careful about how we separate the information

185

and the knowledge of the outer agent and inner agent.

186

And what we ended up doing was what our engineer Joel referred to as consolidating the brain.

187

You want to be careful not to have a split brain between any sub-agent system and outer agent system.

188

And so we just pulled these tools up into the outer agent harness.

189

And this got rid of a lot of this seemingly unpredictable, surprising behavior.

190

So this is a really big learning, dramatically improved a lot of our kind of more complicated evals.

191

So the next phase was we started saying, okay, that's great, but guess what?

192

When I actually go use Claude to generate SQL, it can answer some really impressively hard questions that,

193

candidly, sometimes Blobby doesn't really do very well on.

194

So we started thinking about this, and there's a really interesting backstory here.

195

The short version of it is when Omni was first built, we had actually built a full SQL parsing engine into it.

196

And we ended up discarding it because it wasn't reliable enough.

197

People would throw random SQL at it, and it just couldn't handle every possible permutation.

198

And so that had been sitting on the shelf for years.

199

But we sort of got thinking.

200

We said, listen, if Claude can generate this really powerful, expressive SQL, and we can parse this SQL,

201

as long as it sort of fits into a general form that we understand, maybe there's an opportunity here.

202

And we also kind of said, I think it's probably a safe bet to assume

203

that the good people at Anthropic are investing heavily in making Claude really good at SQL.

204

So that seems like a good bet to put our chips on.

205

So our engineer, Steven, sort of dusted off that old parsing code

206

and really just fundamentally changed the interface of how we were enabling or how we were exposing our query generation to Blobby.

207

Initially, it was this sort of JSON-ified form of a query that was like highly structured, and then we switched it to this SQL parsing mode where we said,

208

said, listen, you can now produce SQL, and we can parse through it.

209

And we were able to sort of give it some of the guidelines that prevented us, our parser, from falling over.

210

And this enabled Blobby to now take a lot of questions

211

that it might have taken three or four attempts to actually answer, or sometimes it would have to chain together three or four queries in sort of awkward ways,

212

and actually write it in a one-shot query.

213

And I think one of the things I noticed is it seems like Claude really likes to write SQL with CTEs, common table expressions, for any of the SQL nerds in the room.

214

And our parser was really good at parsing those, actually.

215

So this turned out to be a really nice sort of

216

marriage where we found the efficiency of the system went way up because we were able to,

217

A, we didn't have to teach Blobby about this proprietary JSON form of a query that we had invented.

218

We were just telling it to write SQL, which it already knew about.

219

and B, it was able to just produce much more efficient queries, so it didn't have to do two or three shots on it.

220

So this is where we are today.

221

We have our agentic system.

222

We have this kind of outer loop that's in charge of

223

checkpointing our executions to make sure that we can recover from any failures.

224

And then we have this inner loop where we have a bunch of tools available.

225

That set is growing dramatically.

226

In addition to the examples I just talked about, we have tools for generating dashboards, we have tools for generating visualizations, we have some validation tools.

227

I'm gonna hopefully do a live demo at the end if the gods are with me and we'll show some of those.

228

And the surface area of those tools is constantly growing.

229

We also have tools that enable Blobby to actually do the data modeling and improve that semantic layer.

230

We also have an eval system.

231

We have an internal eval system.

232

We're also building an eval system for our customers

233

because it's really important that one of the key benefits that we provide is predictability and quality.

234

The CEO asks the question, needs to get the right answer, needs to get the same answer every time.

235

Right, evals.

236

I actually, I love evals.

237

I think I love evals for a different reason than most people love evals.

238

Like I said, my favorite thing about the evals is just having that raw trace data.

239

It's really like the observability part of evals to me was enlightening.

240

Maybe this is a personality trait of being a brute force type person, but I really like just being able to go and say, okay, this was bad, why?

241

And look through that data.

242

And then obviously sort of capturing that into a judge is a nice efficiency gain as well.

243

And this is a big one.

244

So I talked at the beginning about how we build with Claude.

245

When you're building an agentic system, when you're building any system,

246

it's really important for your engineers to have an understanding of what the users of that system actually care about, right?

247

Like, it's really hard for somebody to build a system

248

that they couldn't themselves picture using or can't relate to the users of.

249

So I actually think that, you know, beyond just the massive productivity games we got with Cloud Code,

250

being users of Cloud Code helped us understand what a good harness looks like.

251

And then we can take some of those lessons and bake them into our harness.

252

And so, hey, should we go build a new way for us to go explore the semantic model?

253

Well, let's see what Cloud Code does, because guess what?

254

The semantic model is not that different from a code base.

255

So maybe we should sort of tap into some of the ways that the Cloud Code harness does this.

256

And I feel like that actually helped our engineers really relate to the problem deeply and see some really, you know, see kind of what the latest and greatest techniques were for solving it.

257

All right, I'm gonna switch to live demo mode, and if all goes well,

258

I'll show you just a quick glimpse at how Blobby actually works live. If all goes well.

259

So, first I'm gonna create a dashboard.

260

So, create a dashboard of engineering activity in the Omni repository.

261

So creating a dashboard creates a lot of queries.

262

It also does a lot of thinking about how to lay out the dashboard.

263

It can sometimes take a little while.

264

So while that's running, we'll take a look at the starting point here.

265

But right, it's going to create a bunch of...

266

It's got a plan.

267

We're just going to go through and look at the different relevant topics.

268

An Omnia topic is like a domain of data.

269

So think of it as a big wide data set that combines all the other sub data sets together.

270

And it's found the relevant topics and it's gonna go start actually building this dashboard.

271

While it's doing that, I will switch over to just like that demo that I showed earlier.

272

Same exact type of query.

273

I think one of the other things that we're tapping into here is our philosophy from a product perspective is

274

AI to build, UI to sort of validate and troubleshoot and refine.

275

And so that's baked into this chat experience in a really deep way.

276

So this is pretty much the same exact session.

277

And let's say I wanna actually go sort of touch this data, understand it in a more deep way.

278

Instead of having to kind of squint at a SQL block, I can actually open this in a workbook.

279

And in Omni, a workbook is just a way to actually generate queries and manipulate your data.

280

So I can see, okay, so these are the, you know, this is the GitHub pull request data set.

281

It's correctly filtered for the Omni repository.

282

It's correctly looking at the main branch.

283

And it's looking for all of the merged pull requests.

284

And so, you know, I can go manipulate this chart if I want to.

285

And then additionally, I can start actually looking at other aspects of this data.

286

So let's say I actually just want to look at a specific user.

287

Or you know what?

288

We'll do a little simpler here.

289

I can just look at a different repository, right?

290

And just very quickly, like, let's look at our docs repository instead.

291

So just becomes a much easier way

292

if you want to then go sort of riff on what Blobby has done for you, you can do that.

293

Get back to our dashboard here.

294

So if you look, let's see how we're doing.

295

Oh, great, okay, so it's built a dashboard.

296

Let's see, it's giving us a little summary.

297

Engineering activity, key metrics, cool, okay, let's go check this out.

298

So I'm gonna preview this, pulls open the dashboard in a split pane, and nice, it's done a very nice job here.

299

So past three months, top PR authors, oh man, it's gonna go to the heads of those engineers.

300

PR volume over time.

301

You can see some of this trend.

302

Actually, if we look further back, let's look back over the past 12 months, and we should actually see some of that trend that I showed in the original, oh yeah, there you go,

303

in the original slide deck about, or the original slide about our activity.

304

Oh, not surprisingly, AI is a very hot topic at Omni today.

305

Oh, looks like some of the workflow data didn't come through quite correctly here. That happens.

306

So yeah, and then, as I was saying, I can go through and actually just troubleshoot this live.

307

So maybe I just want to go understand, okay, why is this chart blank?

308

There we go.

309

And I'll be honest, I don't know at a glance why it's blank.

310

I actually don't think we have that data populating very well.

311

So anyway, we'll go back to the slides for now.

312

Thank you for indulging me in the live demo.

313

To wrap up, Omni AI analytics platform powered by Claude.

314

We have specifically designed our harness to be optimized for Claude and the Claude family of models.

315

Have some great customers, fortunate to be surrounded by phenomenal engineers and other teammates.

316

We're based in San Francisco.

317

If anybody wants a blobby sticker after the talk, I'd love to chat with you and give you one.

318

Thank you.

📺 Mesmo canal

✨ Vídeo recomendado

762 frases

Building Anthropic | A conversation with our co-founders

Business EnglishUniversity StudentsWorking Adults

321 frases

How One of NYC’s Fastest-Growing Startups Scaled to $5 Billion

Business EnglishUniversity StudentsWorking Adults

345 frases

What the heck is the event loop anyway? | Philip Roberts | JSConf EU

Business EnglishUniversity StudentsWorking Adults

444 frases

Here's what's REALLY Happening...

Business EnglishUniversity StudentsWorking Adults

58 frases

The mystery is solved... and the answer is 40x cheaper than Claude

Business EnglishUniversity StudentsWorking Adults

128 frases

Demystifying MCP | Architect Insights

Business EnglishUniversity StudentsWorking Adults

239 frases

My 4.9-Star iOS App Is Actually a Website: Here’s How I Built It

Business EnglishUniversity StudentsWorking Adults

12 frases

Rapid Engineering With AI

Business EnglishUniversity StudentsWorking Adults

157 frases

Claude Design just got MASSIVE Upgrades (2.0 Update)

Business EnglishUniversity StudentsWorking Adults

43 frases

Translating Claude’s thoughts into language

Business EnglishUniversity StudentsWorking Adults

853 frases

Why Coding Agents Keep Making Your Codebase Worse

Business EnglishUniversity StudentsWorking Adults

O que é a Técnica de Shadowing?

Shadowing é uma técnica de aprendizado de idiomas com base científica, originalmente desenvolvida para o treinamento de intérpretes profissionais. O método é simples, mas poderoso: você ouve áudio em inglês nativo e repete imediatamente em voz alta — como uma sombra seguindo o falante com 1-2 segundos de atraso. Pesquisas mostram melhora significativa na precisão da pronúncia, entonação, ritmo, sons conectados, compreensão auditiva e fluência na fala.

Técnica de shadowing: leia o guia completo passo a passo →

Prática de Shadowing: Building the best agentic analytics harness: Powered by Claude, built with Claude Code - Aprenda a falar inglês com vídeo

Plan smarter with Claude for Teachers

Can you trust what AI tells you?

Build a proactive agent workflow with Claude Code

Tag Claude in, right where you already work

Build data-driven lesson plans with Claude for Teachers

The future of work with @Claude

How AI gets its character

How Claude Code Works

Building the future of agentic infrastructure

Building Anthropic | A conversation with our co-founders

How One of NYC’s Fastest-Growing Startups Scaled to $5 Billion

What the heck is the event loop anyway? | Philip Roberts | JSConf EU

Here's what's REALLY Happening...

The mystery is solved... and the answer is 40x cheaper than Claude

Demystifying MCP | Architect Insights

My 4.9-Star iOS App Is Actually a Website: Here’s How I Built It

Rapid Engineering With AI

Claude Design just got MASSIVE Upgrades (2.0 Update)

Translating Claude’s thoughts into language

Why Coding Agents Keep Making Your Codebase Worse

O que você vai aprender

Ouça esses sons

Fale como um nativo

O que é a Técnica de Shadowing?