Практика Shadowing: Building the best agentic analytics harness: Powered by Claude, built with Claude Code - Изучайте разговорный английский по видео
Создание урока...
1
Thanks, everybody.
2
Great to be here today.
3
Omni is an AI analytics platform, platform, and today I'm going to talk a little bit about how we build with Claude
4
and what we've built with Claude and how Claude powers that.
5
So start just with how Claude has enabled us.
6
I'm the CTO.
7
I run the engineering team at Omni.
8
We have a team of about 25 engineers.
9
This is a slide of our commits to the main branch of our repository over time.
10
I think it kind of speaks for itself.
11
One of the things hidden in here, kind of a very small piece of that line, is my own commits.
12
I think as CTO of a growing company with hundreds of customers, I sort of assume that at some point I'd have to stop writing code,
13
and I thank Claude very much for making me still able to do some software engineering from time to time.
14
So that's been a really fantastic, sort of unexpected benefit of this rollout.
15
And to speak a little bit about how this went, earlier in 2025, we said to the team, I don't know when and I don't know how, but I know our jobs are changing.
16
So let's just start experimenting, start using these tools, figure out what works.
17
And we did a bunch of it with some sort of fits and starts.
18
And when Cloud Code with the Opus model released, that was when some of our senior engineers said, wait a minute, no, this is real.
19
This is actually helping consistently.
20
And it's kind of been off to the races since then.
21
And starting around, I sort of felt like everybody went away for the holidays
22
and then came back in January and had sort of skilled up
23
and figured it out and was ready to start hitting the ground running with Cloud Code.
24
And you can see the slope of the line from there.
25
So that velocity is a big part of our culture at Omni.
26
We have a core value called Ship It.
27
We also have a core value around transparency.
28
So you can actually see, if you go on our website, omni.co, there's a top line navigation page to our demos.
29
Every Friday at Omni, we have an all-hands meeting.
30
It's the most important meeting of the week.
31
We do about 10 minutes of announcements, shout-outs to each other.
32
And then we do 50 minutes or increasingly more than 50 minutes of demos.
33
We record all this and our CEO's favorite job is Saturday morning, he wakes up, he cuts those demos,
34
posts them on YouTube, and shares them with the world to see.
35
So if you're curious about what we're up to, you can go to omni.co.
36
This has been a really cool way to show our customers, our prospects, the community around us sort of how we're thinking and how we're building.
37
And that speed and velocity has been a huge benefit for us as a company.
38
So let's talk a little bit about what we have built with Claude.
39
So Omni's AI analytics, we let you talk to your data, right?
40
So how does that actually work?
41
So a user comes in and asks a question.
42
In Omni, we're using Claude to actually translate that question into a semantic query.
43
I'll explain more what that means later.
44
We have a semantic layer.
45
And that is, think of that as sort of a translation layer that sits on top of your data warehouse, database, or maybe multiple of those,
46
to actually provide some additional benefit about how to use the data, enforce it, sort of give it a map for how to actually translate that data,
47
or excuse me, that query into SQL that then runs against the warehouse.
48
We'll go into more of this later, but that's just a high level view of kind of how the system works.
49
And this is important because Claude is incredible at answering questions, but you need to tell it more about your business if you want it to answer questions about your business, right?
50
It can tell you incredibly deep insights about how businesses work generally, but if you want it to know about how your business works,
51
you need to tell it not only how the business works, the terminology you use, and then of course, if you're asking a data question,
52
you need to tell it how your data looks and how that works.
53
And so that's where we come in.
54
That's the problem we're solving.
55
And it's subtly difficult, right?
56
Like, even last quarter means drastically different things at different companies.
57
Even in our own company, you know, in the product and engineering organization, last quarter refers to the calendar year.
58
In our sales team, it refers to our fiscal quarter.
59
So all of that needs to actually get coded in to the context
60
and awareness and even the sort of data layer
61
and definitions of the data so that it can be used appropriately to ultimately get you the right answer to your question.
62
So this semantic layer, what's it doing?
63
Like I said, it's a translation layer that sits on top of the database.
64
It's doing a few things.
65
One, it's curating the data.
66
It's very easy to sort of come up with a toy demonstration of how an LLM or,
67
frankly, a human can get correct answers on top of 10 data sets in a database.
68
Real company data warehouses and databases are not like that.
69
They have tens of thousands, hundreds of thousands of data sets, sometimes more, and all of them have, there's a hundred revenue tables, right?
70
There's a hundred opportunity tables.
71
It's very unclear how to actually stitch those things together and use them in the correct way.
72
So this is one of the benefits that our semantic layer provides, is it allows you to define how to use these things together and also curate it.
73
Say, hey, listen, this is the one that matters, ignore those other ten.
74
With that, it's a way to encode the context.
75
And I think one of the things that we've learned is that context is great,
76
but context localized to what the actual definition that the context refers to makes it all the better, right?
77
And so I think about this, we're at code with Claude.
78
If you use Claude code, which I assume most of you do, you have your Claude.md files, right?
79
And so the more you can do to sort of localize
80
that context context next to the parts of the code that it applies to, the better results you're going to get.
81
That's what our semantic layer does as well, is it helps you provide that context next to the field definition that it applies to, rather than in a separate file over somewhere else.
82
And then finally, permissions, right?
83
It's a permissions layer.
84
Make sure that people see the data they're supposed to see and don't see the data they're not supposed to see.
85
And inside of our application, this feedback loop is an important part of how this stays current and accurate.
86
Because guess what?
87
In a real organization, this stuff changes constantly.
88
And so our application provides a feedback loop where the next question
89
that gets asked of the data can then be fed back into the definitions, into the context for a continuous learning loop.
90
So this is our agent.
91
Name's Blobby.
92
If you look at Blobby, you could probably see this is a mature, professional, refined data analyst.
93
And that is what Blobby is today.
94
But Blobby hasn't always been that refined.
95
started building Blobby, let's say, about 18 months ago, and we've learned a lot along the way.
96
You know, Blobby's grown up quite a bit in the past 18 months.
97
So what I want to talk about is a little bit of how we develop the different phases
98
and also what we learned along the way that kind of helped increase the quality and capabilities of Blobby.
99
So just to ground this conversation, I just want to show you a really quick demo video of what Blobby does in real life.
100
So if we can cut to the video quickly.
101
So what we're going to see here is, like I said, ask questions of your data, blobby is sitting there dutifully waiting to answer your question.
102
And as we do this, you'll start to see some of the phases that we go through to actually break down the question and then answer it.
103
So, right, the first thing Blobby's doing is saying, all right, you're asking about PRs.
104
Blobby's very smart because Claude's very smart.
105
It knows that PRs refer to GitHub pull requests.
106
Let me go find that in your semantic model and figure out what data you're actually referring to.
107
Then it goes and looks at the values of the dataset, because we said we only wanted it from a particular repository, so it has to apply a filter.
108
It needs to know what that filter value is.
109
But when I type out questions to an LLM, I make typos all the time.
110
So it needs to do a little bit of fuzzy matching to make sure that it's actually finding the right thing.
111
And then it goes through, it generates a query, runs that query against the data warehouse, gets the results, provides a nice visualization,
112
and then does a nice little summary at the end to tell you what you're seeing.
113
Cool.
114
So just grounding you a little bit in what the actual experience is like.
115
So what did we learn along the way?
116
So the very first version of Blobby was basically single question, single answer.
117
And we quickly realized
118
that we needed to give a lot more metadata about how to use the data and how the data is typically used.
119
So we always had these label and description fields in our definitions, but we needed to add some additional context.
120
So we added this AI context concept, which is specifically for an LLM, hey, how should you use this?
121
When you're asked a question about this, you might want to use this reference
122
or this field just to kind of help the data team
123
and the administrators of this to actually steer in the right direction and ensure that you get a quality answer.
124
And then sample queries, very self-explanatory, really helpful to kind of ground it in, hey, this is a typical use case.
125
This is the query you would run to answer a question that like X.
126
And then finally, values.
127
This one was kind of subtle.
128
What we realized is, again, back to that example of, I'm asking a question about a certain repository, or in this example, asking a question about a region.
129
It's really helpful to give the LLM just a taste of what the values of that field are.
130
Because you can see, region, all values are EMEA, NAM, APAC.
131
So it can infer the next 10 values, because it sees, all right, These are abbreviations of regions of the world, right?
132
But it's useful for it to know, like, okay, this is an abbreviation, so if somebody asks for United States, I can just put US in there.
133
So this did a nice job of helping to improve the quality of the question and answers that we were getting.
134
But at this point, Bobby was still really not an agent.
135
So that was the next big leap here, was adding an agentic loop around this.
136
This is a big engineering effort.
137
We built our own agentic harness.
138
And it included this concept of tasks like all good agents have, included a lot of other stuff too.
139
I think one of the biggest things we learned was that the agentic loop was really great at recovering from errors.
140
So one of the earliest massive quality increases we made was to just A,
141
tell Blobby how to recover from errors and give it some budget to go do that.
142
And B, then go invest in providing great error messages
143
that were descriptive about what was happening and how you might fix it.
144
And that alone allowed the quality score to increase dramatically.
145
We saw our evals, like a lot of our more difficult evals, got a lot better once we did that.
146
At this time, though, because we were sort of in this mode of question and answer, we were using the Haiku model.
147
And the Haiku model is great, but once you get into these more elaborate, agentic conversations, it's just not designed for those, right?
148
And so we switched over to Sonnet, and we're showing a graph here of token consumption.
149
And the reason is twofold.
150
One, these are longer conversations, they're more complex, they consume more tokens.
151
That was by design.
152
Two, this was a really big unlock.
153
So we all of a sudden started getting our customers saying, wow, I just asked this question that either I never would have been able to answer myself, or even if I did,
154
it would have taken me hours.
155
And it just nailed it in two minutes.
156
And so the usage of Blobby started dramatically taking off at this point.
157
So at this point, our CEO, who is our loudest and most critical user, like all good CEOs are,
158
was telling us, listen, guys, I know this thing's really good, but it screwed up this question, go fix it.
159
And we all said, you know, Colin, LLMs are a little unpredictable.
160
Like you're just gonna have to accept that it's not always going to be perfect.
161
He said, not good enough, go fix it.
162
So, okay, fine.
163
And where this led us was, I think one of the big efforts that we undertook at this point was to say,
164
okay, let's really invest in understanding the traces and being able to see the traces of these, you know, quote-unquote, bad sessions.
165
And this led us to a series of major surgeries that we refer to as the Blobotomies.
166
And the Blobotomies really were traced back to what we were seeing in these traces.
167
When you look at the traces, you kind of get to see the inner workings of how the agent is sort of talking to itself
168
and reacting and responding in these loops.
169
And that really clarified why some of these seemingly kind of just bad random sessions,
170
you could actually start to see where they were rooted in real problems.
171
So an example of this was the original design of our agent might have been a little too clever.
172
And we had sort of an outer agent that was responsible for producing the task list.
173
It understood where all the data available to it.
174
but it was not in charge of query generation.
175
It had a sub-agent that was in charge of query generation.
176
And it felt like a reasonable design.
177
It was also handy because we could use that query generation sub-agent in a few other contexts.
178
But what we found once we started digging into these traces was that the sub-agent, its job was to generate one query based on whatever it was asked.
179
And the outer agent didn't know what was actually able to be answered in a single query.
180
So it would say, hey, subagent, go answer me a question about GitHub pull requests and support data,
181
and summarize these things.
182
And then the subagent would say, I can't answer that in a single query.
183
I can only, I would have to run multiple queries.
184
And so it's sort of, the light bulb went off when we started seeing this, that we have to be really careful about how we separate the information
185
and the knowledge of the outer agent and inner agent.
186
And what we ended up doing was what our engineer Joel referred to as consolidating the brain.
187
You want to be careful not to have a split brain between any sub-agent system and outer agent system.
188
And so we just pulled these tools up into the outer agent harness.
189
And this got rid of a lot of this seemingly unpredictable, surprising behavior.
190
So this is a really big learning, dramatically improved a lot of our kind of more complicated evals.
191
So the next phase was we started saying, okay, that's great, but guess what?
192
When I actually go use Claude to generate SQL, it can answer some really impressively hard questions that,
193
candidly, sometimes Blobby doesn't really do very well on.
194
So we started thinking about this, and there's a really interesting backstory here.
195
The short version of it is when Omni was first built, we had actually built a full SQL parsing engine into it.
196
And we ended up discarding it because it wasn't reliable enough.
197
People would throw random SQL at it, and it just couldn't handle every possible permutation.
198
And so that had been sitting on the shelf for years.
199
But we sort of got thinking.
200
We said, listen, if Claude can generate this really powerful, expressive SQL, and we can parse this SQL,
201
as long as it sort of fits into a general form that we understand, maybe there's an opportunity here.
202
And we also kind of said, I think it's probably a safe bet to assume
203
that the good people at Anthropic are investing heavily in making Claude really good at SQL.
204
So that seems like a good bet to put our chips on.
205
So our engineer, Steven, sort of dusted off that old parsing code
206
and really just fundamentally changed the interface of how we were enabling or how we were exposing our query generation to Blobby.
207
Initially, it was this sort of JSON-ified form of a query that was like highly structured, and then we switched it to this SQL parsing mode where we said,
208
said, listen, you can now produce SQL, and we can parse through it.
209
And we were able to sort of give it some of the guidelines that prevented us, our parser, from falling over.
210
And this enabled Blobby to now take a lot of questions
211
that it might have taken three or four attempts to actually answer, or sometimes it would have to chain together three or four queries in sort of awkward ways,
212
and actually write it in a one-shot query.
213
And I think one of the things I noticed is it seems like Claude really likes to write SQL with CTEs, common table expressions, for any of the SQL nerds in the room.
214
And our parser was really good at parsing those, actually.
215
So this turned out to be a really nice sort of
216
marriage where we found the efficiency of the system went way up because we were able to,
217
A, we didn't have to teach Blobby about this proprietary JSON form of a query that we had invented.
218
We were just telling it to write SQL, which it already knew about.
219
and B, it was able to just produce much more efficient queries, so it didn't have to do two or three shots on it.
220
So this is where we are today.
221
We have our agentic system.
222
We have this kind of outer loop that's in charge of
223
checkpointing our executions to make sure that we can recover from any failures.
224
And then we have this inner loop where we have a bunch of tools available.
225
That set is growing dramatically.
226
In addition to the examples I just talked about, we have tools for generating dashboards, we have tools for generating visualizations, we have some validation tools.
227
I'm gonna hopefully do a live demo at the end if the gods are with me and we'll show some of those.
228
And the surface area of those tools is constantly growing.
229
We also have tools that enable Blobby to actually do the data modeling and improve that semantic layer.
230
We also have an eval system.
231
We have an internal eval system.
232
We're also building an eval system for our customers
233
because it's really important that one of the key benefits that we provide is predictability and quality.
234
The CEO asks the question, needs to get the right answer, needs to get the same answer every time.
235
Right, evals.
236
I actually, I love evals.
237
I think I love evals for a different reason than most people love evals.
238
Like I said, my favorite thing about the evals is just having that raw trace data.
239
It's really like the observability part of evals to me was enlightening.
240
Maybe this is a personality trait of being a brute force type person, but I really like just being able to go and say, okay, this was bad, why?
241
And look through that data.
242
And then obviously sort of capturing that into a judge is a nice efficiency gain as well.
243
And this is a big one.
244
So I talked at the beginning about how we build with Claude.
245
When you're building an agentic system, when you're building any system,
246
it's really important for your engineers to have an understanding of what the users of that system actually care about, right?
247
Like, it's really hard for somebody to build a system
248
that they couldn't themselves picture using or can't relate to the users of.
249
So I actually think that, you know, beyond just the massive productivity games we got with Cloud Code,
250
being users of Cloud Code helped us understand what a good harness looks like.
251
And then we can take some of those lessons and bake them into our harness.
252
And so, hey, should we go build a new way for us to go explore the semantic model?
253
Well, let's see what Cloud Code does, because guess what?
254
The semantic model is not that different from a code base.
255
So maybe we should sort of tap into some of the ways that the Cloud Code harness does this.
256
And I feel like that actually helped our engineers really relate to the problem deeply and see some really, you know, see kind of what the latest and greatest techniques were for solving it.
257
All right, I'm gonna switch to live demo mode, and if all goes well,
258
I'll show you just a quick glimpse at how Blobby actually works live. If all goes well.
259
So, first I'm gonna create a dashboard.
260
So, create a dashboard of engineering activity in the Omni repository.
261
So creating a dashboard creates a lot of queries.
262
It also does a lot of thinking about how to lay out the dashboard.
263
It can sometimes take a little while.
264
So while that's running, we'll take a look at the starting point here.
265
But right, it's going to create a bunch of...
266
It's got a plan.
267
We're just going to go through and look at the different relevant topics.
268
An Omnia topic is like a domain of data.
269
So think of it as a big wide data set that combines all the other sub data sets together.
270
And it's found the relevant topics and it's gonna go start actually building this dashboard.
271
While it's doing that, I will switch over to just like that demo that I showed earlier.
272
Same exact type of query.
273
I think one of the other things that we're tapping into here is our philosophy from a product perspective is
274
AI to build, UI to sort of validate and troubleshoot and refine.
275
And so that's baked into this chat experience in a really deep way.
276
So this is pretty much the same exact session.
277
And let's say I wanna actually go sort of touch this data, understand it in a more deep way.
278
Instead of having to kind of squint at a SQL block, I can actually open this in a workbook.
279
And in Omni, a workbook is just a way to actually generate queries and manipulate your data.
280
So I can see, okay, so these are the, you know, this is the GitHub pull request data set.
281
It's correctly filtered for the Omni repository.
282
It's correctly looking at the main branch.
283
And it's looking for all of the merged pull requests.
284
And so, you know, I can go manipulate this chart if I want to.
285
And then additionally, I can start actually looking at other aspects of this data.
286
So let's say I actually just want to look at a specific user.
287
Or you know what?
288
We'll do a little simpler here.
289
I can just look at a different repository, right?
290
And just very quickly, like, let's look at our docs repository instead.
291
So just becomes a much easier way
292
if you want to then go sort of riff on what Blobby has done for you, you can do that.
293
Get back to our dashboard here.
294
So if you look, let's see how we're doing.
295
Oh, great, okay, so it's built a dashboard.
296
Let's see, it's giving us a little summary.
297
Engineering activity, key metrics, cool, okay, let's go check this out.
298
So I'm gonna preview this, pulls open the dashboard in a split pane, and nice, it's done a very nice job here.
299
So past three months, top PR authors, oh man, it's gonna go to the heads of those engineers.
300
PR volume over time.
301
You can see some of this trend.
302
Actually, if we look further back, let's look back over the past 12 months, and we should actually see some of that trend that I showed in the original, oh yeah, there you go,
303
in the original slide deck about, or the original slide about our activity.
304
Oh, not surprisingly, AI is a very hot topic at Omni today.
305
Oh, looks like some of the workflow data didn't come through quite correctly here. That happens.
306
So yeah, and then, as I was saying, I can go through and actually just troubleshoot this live.
307
So maybe I just want to go understand, okay, why is this chart blank?
308
There we go.
309
And I'll be honest, I don't know at a glance why it's blank.
310
I actually don't think we have that data populating very well.
311
So anyway, we'll go back to the slides for now.
312
Thank you for indulging me in the live demo.
313
To wrap up, Omni AI analytics platform powered by Claude.
314
We have specifically designed our harness to be optimized for Claude and the Claude family of models.
315
Have some great customers, fortunate to be surrounded by phenomenal engineers and other teammates.
316
We're based in San Francisco.
317
If anybody wants a blobby sticker after the talk, I'd love to chat with you and give you one.
318
Thank you.
📺 Тот же канал
C12
Build a proactive agent workflow with Claude Code
Business EnglishUniversity StudentsWorking Adults
B25
Tag Claude in, right where you already work
Business EnglishWorking Adults
B22
Build data-driven lesson plans with Claude for Teachers
Business EnglishHigh School StudentsUniversity Students
B2The future of work with @Claude
Business EnglishUniversity StudentsWorking Adults
B23
Can you trust what AI tells you?
Business EnglishUniversity StudentsWorking Adults
C14
How AI gets its character
Business EnglishUniversity StudentsWorking Adults
B12
Plan smarter with Claude for Teachers
Business EnglishHigh School StudentsUniversity Students
C24
How Claude Code Works
University StudentsWorking Adults
C1Building the future of agentic infrastructure
Business EnglishUniversity StudentsWorking Adults
✨ Рекомендуемое видео
C1762 предложений
Building Anthropic | A conversation with our co-founders
Business EnglishUniversity StudentsWorking Adults
B21
321 предложений
How One of NYC’s Fastest-Growing Startups Scaled to $5 Billion
Business EnglishUniversity StudentsWorking Adults
C13
444 предложений
Here's what's REALLY Happening...
Business EnglishUniversity StudentsWorking Adults
C1115 предложений
Novanta CIO Sarah Betadam on managing risk around AI
Business EnglishUniversity StudentsWorking Adults
C11
218 предложений
Next.js 16.3 got good (if you care)
Business EnglishUniversity StudentsWorking Adults
C11
128 предложений
Demystifying MCP | Architect Insights
Business EnglishUniversity StudentsWorking Adults
B21
157 предложений
Claude Design just got MASSIVE Upgrades (2.0 Update)
Business EnglishUniversity StudentsWorking Adults
C11
126 предложений
How I use AI as a senior software engineer and 5 principles of AI explained in 10 minutes
Business EnglishUniversity StudentsWorking Adults
C11
346 предложений
Trust the Process: How to Design Teams That Actually Work
Business EnglishUniversity StudentsWorking Adults
B23
43 предложений
Translating Claude’s thoughts into language
Business EnglishUniversity StudentsWorking Adults
C11
853 предложений
Why Coding Agents Keep Making Your Codebase Worse
Business EnglishUniversity StudentsWorking Adults
Контекст и фон разговора
Видео представляет собой презентацию чиновника компании, рассказывающего о разработке аналитической платформы с использованием искусственного интеллекта. Диалог носит официальный, но разговорный тон, с элементами технической терминологии. Это отличная возможность для практики понимания профессиональной речи на английском, а также для работы над произношением и ритмом.
5 ключевых фраз для повседневной коммуникации
- "start experimenting" — "начать экспериментировать". Удобная фраза для обсуждения новых идей или инструментов.
- "hitting the ground running" — "начать действовать активно". Используется, когда нужно подчеркнуть быстрый старт проекта.
- "core value" — "основное значение". Полезна для описания принципов компании или команды.
- "shout-outs to each other" — "похвалы друг другу". Хорошая фраза для разговоров о корпоративной культуре.
- "deep insights" — "глубинные знания". Подходит для обсуждения анализа данных или ситуации.
Пошаговое руководство по shadowing для этого видео
Чтобы улучшить произношение английского и освоить ритм речи, используйте метод shadowspeak (или shadowing). Вот как это сделать:
- Послушайте фрагмент видео (10–15 секунд) и обратите внимание на интонацию и паузы. Техническая речь часто содержит длинные предложения, поэтому важно заметить, где спикер останавливается.
- Повторите фразу сразу после спикера, стараясь копировать его темп и ударения. Например, фраза "This is actually helping consistently" требует акцента на "actually" и "consistently".
- Запишите себя и сравните с оригиналом. Обратите внимание на звуки, которые сложно произносить (например, "analytics" или "semantic").
- Повторяйте каждый фрагмент 3–5 раз, пока не почувствуете, что ваша речь становится естественной. Это поможет улучшить не только произношение, но и практику разговорного английского.
Этот подход позволит вам адаптировать речь к разным ситуациям, включая профессиональные презентации, и сделать ваше произношение более уверенным.
Грамматика в этом видео
Конструкции, которые говорящий использует чаще всего, с точными словами из видео:
| Конструкция | В видео |
|---|---|
| Present Perfect have/has + причастие прошедшего времени — прошлое действие, важное сейчас | we've built · has enabled · has been |
| Пассивный залог be + причастие прошедшего времени — важно, что происходит, а не кто это делает | be fed · were traced · were rooted |
Что такое техника Shadowing?
Shadowing — это научно обоснованная техника изучения языка, изначально разработанная для подготовки профессиональных переводчиков и популяризированная полиглотом доктором Александром Аргуэльесом. Метод прост, но эффективен: вы слушаете аудио на английском от носителей языка и немедленно повторяете вслух — как тень, следующая за говорящим с задержкой в 1–2 секунды. В отличие от пассивного прослушивания или грамматических упражнений, Shadowing заставляет мозг и мышцы рта одновременно обрабатывать и воспроизводить реальные речевые паттерны. Исследования показывают, что это значительно улучшает точность произношения, интонацию, ритм, связную речь, понимание на слух и беглость речи — что делает его одним из самых эффективных методов для подготовки к IELTS Speaking и реального общения на английском.