Shadowing Practice: Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse - Learn English Speaking with Video

Kies een oefenmodus

Elke modus opent een volledig scherm oefenruimte.

612 zinnen
Als zinnen te kort of te lang zijn, klik op Bewerken om ze aan te passen.
1
Hey, my name is Raymond Weitkamp, and my talk today is originally titled Recursive Decision Models,
2
and what I think I've realized since submitting that title is what it's really about,
3
what we're really going to be talking about today is Operating System 1.
4
Okay, so maybe some quick definitions, or what does that mean? uh
5
so system one models the the um naming of system one models from type safe
6
when they released jev um not even two months ago now i think um came from uh this concept
7
that danny kahneman popularized uh in thinking fast
8
and slow this idea of like the system one part of our brains is making very fast and intuitive like judgment calls,
9
gut reactions, reflexes.
10
Maybe reflexes more purely neurological but that idea of like a lot of the decisions
11
that we make are not actually like purely rational and so system one is fast,
12
system two is slow, more deliberate like logical reasoning problem solving etc
13
and um i think what was a little unclear when they um
14
when they announced jeff was whether they intended system one to be like their proprietary name family of models
15
or like that's they were expecting all the other people to call them system one models
16
i would say most um most people are calling them decision models
17
and um and so i wasn't really willing to wait for the people to decide.
18
And so when I came up with recursive decision models, you know, it was just kind of using that.
19
So a rose by any other name, tomato, tomato, we don't really need to get too hung up on this idea of what are they?
20
Are they decision models or not?
21
Allie here argues that
22
Jeff makes the judgments, not the decisions your code makes the decision i think that's that's absolutely correct um
23
and yet i think that that much of the zeitgeist
24
and much of the folks working
25
this space outside of type safe are are kind of uh rallying around
26
this idea of um decision model being the term there is the holdout camp of
27
refusing to call them anything other than zero shot classifiers um anyways
28
i don't have a dog in the race the naming just trying to explain what i am talking about
29
and i think that one of the things
30
that is now connected here to this idea of system one system two dan economy
31
and all this stuff um maybe related to how we behave
32
but i think much more interesting from this perspective of building ai systems is what
33
if reasoning uh was actually many small decisions and um
34
So yeah, maybe a different way of saying this, like what if behaviors or tasks
35
that we might think of as requiring a lot of reasoning
36
or a lot of like deliberate cognitive processing could actually be solved in a different way with a lot of very tiny,
37
very, very fast decisions.
38
We're going to keep coming back to this question throughout the talk.
39
And I think it's really key in what's interesting here.
40
So recursive decision models.
41
So this is my concept that I've been developing.
42
It's at its simplest, perhaps the marriage of two things that I got really excited about.
43
One is just over a year old.
44
It's called recursive language models, RLM.
45
This is from MIT, Alex Singh and Omar their original blog post is just over a year ago.
46
Happy birthday to RLM, belated birthday.
47
And then like many others, I was also chomping at the bit to, I think it's actually champing at the bit.
48
Anyways, I was champing and chomping to get off the freaking Jev waitlist, which thankfully got off real quick.
49
They did a good job with that.
50
But I think that one of the things
51
that I've been sharing with people is I think
52
that part of what TypeSafe did so well and what captured the attention of so many engineers,
53
AI engineers, is this idea of JEV as convergence, right?
54
Like decisions converge.
55
And that is in direct contrast to generative AI, which has been, you know blowing up like crazy it's arguably like the fastest growing
56
um you know market in the history of the world
57
and um so much of the attention
58
and money in in the space has been about generation right
59
and that's awesome and there's amazing things that are possible generation
60
but if you really think about it like even if you
61
if you have infinite tokens right
62
you kind of have infinite divergence right like the generative stuff
63
it's actually hard to um converge
64
it's like we still have slop is someone still needs to decide like what's going to make it into production
65
and maybe you've got your like code factory set up
66
there all those like agents are doing those decisions for you um
67
but at the end of the day like the the important thing
68
that i've got me excited
69
and i think i think this is why many other people are
70
so excited about about jev these system one models these decision models is this is this idea of yin
71
and yang right it's a counterbalancing force um and
72
so the decision model cannot generate text it can only rank
73
and pick and choose amongst options um and
74
that is like in direct contrast to lms
75
which have this ability to just like spit out uh freewheeling open -ended um text
76
and in the early is actually hard to get them to structure their output
77
so i i see this as as yin and yang
78
um i am not going to go super deep into recursive language models
79
i would highly encourage
80
you to check out my talk from the summer from ai engineer world's fair
81
this summer recursive coding agents it's on their youtube channel it's also at recursivecodingagents .com My slides here are also a website,
82
and I'm going to make them available publicly after I record this.
83
That is going to go a lot deeper into why I think RLMs are the new reasoning models, why this is a really important paradigm.
84
So with that, we do need to sort of just draw the direct analogy here and
85
so in the rlm idea right the root recursive language model has the ability to call sub recursive language models
86
and those could call sub recursive language models
87
and then you can you have to sort of like cap the depth of recursion to just not um
88
if you ever want to like come come back from the depths um
89
and so here we say like okay depth two then it's an llm
90
so the the leaf node in this graph in this tree would would ultimately be an LLM call
91
we can make that exact analogy
92
that exact map onto system one models onto decision models
93
and say okay we've got a root and
94
that can decide to call a sub RDM
95
and then there's some max depth at
96
which point we just have to make like a single call a single system one call a single decision call
97
but we also have to think about composing these things together right
98
and the um i am going to show some examples as pure decisions all the way down
99
that's very important um this bottom left of what i'm i'm calling decision as a tool um
100
so you could you could also
101
this is what my recursive coding agent talk was about it's
102
like replace rlm with like agent right coding agent okay
103
you got the root coding agent you got the sub agent
104
so i would say like this decisions as a tool
105
or jev as a tool was is like the obvious one
106
and within you know 24 hours of people getting access to jev there's tons
107
and tons of examples here right people using jev for compaction people using jev as the thing
108
that decides whether a tool call can execute
109
or not people using jev as a router to decide
110
which um which model is going to get this particular request right
111
so that stuff's um it's all very cool there's some really interesting cases there
112
and people just went buck wild going super fast um figuring
113
that out i think to me what's
114
i'm more interested in is um is not as obvious how you would set it up uh
115
and that's where the root is a decision model
116
and you only can you could choose as a decision to call to a language model
117
or an agent as one as one possibility so almost everything
118
that i'm going to be talking about with rdm is going
119
to be in this paradigm where we're actually letting the root
120
or the parent of this system be the system one model the decision model okay
121
so now i want to show this for a couple reasons
122
this you might recognize this this is the jeff plays doom it was part of like the launch tweet storm
123
and got a lot of attention and yet despite
124
that i would say it's also one of the most um under appreciated
125
or at least i would say underdeveloped um uh examples here and
126
so i'm just going to show a very short clip from
127
this where we compose their answers into the player's next action
128
and keep repeating
129
that loop as the game changes
130
this is what we call a composition of ai primitives composition of ai primitives
131
now this composition handles the moment to moment play
132
so basically the first thing that they're showing is this harness
133
that they built of decisions to handle like the first person
134
shooting aspect is it going to hold carry on walk dodge blah blah blah
135
it has these actions right this is where i i it's so simple
136
and it's very much in line with this idea of composing
137
systems together recursive decision models right a graph of different types of decisions um here we go
138
so this is the second composition composition that decides where to explore next
139
and they can work together to make it through a level okay
140
so now you have two compositions one is the first person perspective one is this like bird's eye
141
or like map aware perspective
142
and by composing those two things together you have a doom playing decision agent
143
and now to me i don't know maybe i missed something
144
but i would say like um i have seen almost nothing very very very few um examples
145
that are actually like taking this uh forward that are
146
that are moving this is a super powerful idea of composing these two systems together it's really insane
147
that with like essentially two separate um but coupled um
148
so two distinct
149
but coupled decision agents right you now have with pure jev calls um a doom playing agent right like
150
that is really wild and i think really underappreciated despite the fact
151
that probably like i don't know millions
152
or tens of millions of people saw this video
153
and it comes back to this question what if reasoning was actually many small decisions, right? And, um,
154
or what are the types of activities or categories of things that we might naively assume would need reasoning when in fact,
155
what we need is a smart way of architecting small, fast decisions.
156
Okay.
157
So now onto recursive decision models.
158
I think the, the glib tagline for this would be like a decision that can decide to make another decision.
159
And, that's certainly an important element of it.
160
While I was developing this idea, this concept, I've given myself three rules.
161
The three rules of RDMs are, one, that the context stays outside the model.
162
And so the code can cut it into slices.
163
So in the Doom example, the full state of the game is outside
164
then those two little sub harnesses the first person shooting harness has some specific subset of
165
that the the map exploring bird's eye view decision model has another view of
166
that the second thing is this
167
and now this is where it's a little different from that demo is the decision can choose to call another decision.
168
And again, up to this maximum depth, we talked about this recursive tree.
169
Now, these two things are the most important fundamental aspects of RLM as well, right?
170
The context is externalized.
171
The agent can make sub -agents.
172
The RLM can call sub -RLMs and sub -LLMs, right?
173
So those two things are like copied straight from the RLM idea.
174
And it's applying that directly to, to system one models, decision models.
175
Uh, the third piece is first, it must work with no generative model at all.
176
And this was very important to me.
177
Um, because I, again, I'm like all this, like Jeff as a tool for a normal coding agent is, is, is cool.
178
And there's some really great examples, but it's, it's obvious to me what was really interesting in terms of
179
novel what might unlock some novel behaviors some novel capabilities um is what
180
if there's no generative model at all it also proves
181
that the that the rdm architecture is actually uh correct like it should be able to work with no generation
182
And at the same time, it must compose with LLMs and with agents, right?
183
So those are my three rules in developing this concept of recursive decision models.
184
So first, I want to show the decisions all the way down example, okay?
185
So decisions all the way down.
186
You might be familiar with this idea of a Galton board.
187
And maybe you learned about this in the context of learning about the Gaussian distribution or what do they call it?
188
Normal distribution.
189
So like each ball has like a 50 -50 chance at each pin of falling.
190
And then you build this like bell curve Gaussian distribution distribution
191
so the simplest way
192
that i could think of to to visualize this idea of um like recursive layers of decisions is now what
193
if the pins could decide
194
so you could think of this kind of like a binary search like
195
so each um numbered ball just like oh is this one gonna go higher
196
or lower higher or lower um and
197
so this pin is like splitting them in half this pin spitting them in to its half etc um
198
and again this is kind of like a silly simple example uh but now
199
when those are smart then we separate them perfectly
200
now system one model like jev decision model whatever
201
we want to call them uh well
202
we can ask about we can ask many questions at once
203
we don't have to just ask one on one axis we could ask in multiple dimensions
204
so we can now visualize
205
that in 3d as like the ball's falling they're being sorted by color
206
and one axis by number and another we could put the cross -sectional views here
207
and see that in the case that this is a perfect classifier right
208
it's perfectly classifying the numbered bucket
209
that it should go in in the end the color bucket
210
so again this is very much a toy example in the sense
211
that like actually what you you could do all
212
this in one classifier call you could do all this in one decision call you don't need
213
this like recursive tree or like iterative tree to do it
214
but I'm I want you to keep the visualization of this
215
because it's exactly what's going to happen in some of these more specific examples of recursive decision models.
216
Okay.
217
So hierarchical search was one of the launch day cookbooks of TypeSafe.
218
And what they did is they took this Shopify database of 12 ,000 product categories
219
and they implemented like a beam search algorithm with hierarchical calls
220
to jev to figure out from from a user query
221
which product category the user's query was describing
222
and what is unique as a simple demonstration of what's possible
223
with recursive decision models is you can do the beam search without the beam.
224
So essentially at each layer here,
225
we're simply asking, well, of the next sub layer of products,
226
what is the next one that we're interested in exploring?
227
And so in this example, we're not exploring all 12 ,000 categories at once
228
that would like overload the context window we're taking the parent category
229
we're deciding okay the user said laptop backpack okay which category am I going to explore next and and iterating along from there
230
and we could decide in the code how we implement
231
it do we do greedy do we pick like only the top one right do we explore top three
232
that are rated with the different probabilities etc
233
but you could kind of again it's like similar to a binary search algorithm in theme except
234
that like the compositional unit of it is a classifier is
235
a judgment is a decision model is a system one model again too many names
236
i don't want to fight about the names okay visually um we can show this um as well and
237
so the uh jev quickly became uh multimodal uh just a
238
few weeks ago about a month after the initial uh launch
239
and um lily over here uh posted like the first variation of this which is like wherever you are getting a hit,
240
you are now then kind of like zooming in to have like a feedback loop and like figure out.
241
you know ask again on on a smaller box okay is this really uh in her her case looking for beetles
242
right and and then nathan from the typesafe team kind of followed up on this
243
and um what's interesting about his implementation implementation is it's much more like um
244
this um binary search in the sense
245
it's like a recursive halving of the of the image right
246
like a like a golden um um golden spiral style like having right
247
and then within those uh blocks within those buckets um uh like what what is it
248
that we're um seeing so again this is this is the um these are simple simple examples
249
that you you might not need recursion here like a you
250
know a good vision model could probably answer the question in one shot
251
and i think some people have have pointed that out clearly,
252
but I'm trying to give a number of different visual examples of how a recursive decision model could work
253
and where different ways of you seeing what that might look like in terms of the recursion.
254
So now let's get back to playing video games.
255
And so this is me playing factorial with a recursive decision model
256
so we've got all of these um these lines
257
that are making our steel or whatever i'm i'm not very good at factorial um
258
so apologize uh i actually would like to um my goal
259
actually with this is to have my recursive decision model teach me how to play factorial
260
because i've never really played before
261
but it's a it's a pretty cool like very open -ended um way of evaluating these systems benchmarking systems
262
so you can see here it's got all these lines
263
it's figuring okay this group is though the one that's um got a problem
264
uh this line is the one that's got a problem it's this element the inserter
265
that i think i need to go uh fix
266
and i need to replace it so now it's going to run over
267
and go do that and now the thing that I want to point out here is there is no reason here.
268
There's no large language model here.
269
There's nothing generative here, right?
270
So it's kind of, I don't know, maybe it's a little bit of a toy example in that it's literally like a matrix of these things,
271
and they're all the same.
272
But then it moves on to the next one.
273
You'll see, actually, I should have maybe added some pathfinding composition of recursion
274
because it ran like the long way around to go the next one
275
that was not totally perfect right but this is just an example of what
276
if you just looked at this without knowing it i think
277
you would i don't know at least i would assume
278
that you need some kind of reasoning model here you need um
279
and um yeah you like this is like i don't know like cloud plays pokemon level uh thing
280
and to me it's just absolutely fascinating that this can be done
281
without any generative language model in there at all
282
so again we come back to this idea what if reasoning
283
or tasks that we might have assumed
284
and naively required reasoning can actually be done with these many many many small decisions all right
285
so the the weakness
286
or the downside um the limitation of decisions all the way down right is that something has to have made
287
those decisions like what what um are the allowed decisions
288
and um for any of you who have a type safe account
289
you might remember the the quiz you have to pass a quiz to get an api key
290
and the question is can jeff generate text or not
291
and the answer is no and
292
if you put yes they won't give you an API key so
293
so the um you something like programmatically um whatever like a
294
script has to do it another agent has to do it
295
the application itself has to have pre -enumerated all of it right um
296
if you can only have decisions then there's no generative aspect um
297
and what's crazy to me is that for games for browsers
298
and i think even for a lot of elements of computer use
299
that actually might be enough right and i think
300
if you kind of visualize now back all these examples
301
that we just showed uh
302
you hopefully can understand why with a tree of many many many small decisions we can actually um find things
303
so you imagine a game the number the total possible um
304
actions available to a user the total possible state of the game can be like enumerated programmatically
305
and recursively addressed to make judgments make decisions about what what the next move is
306
um in a browser
307
so these slides are there's actually a website um in this particular website
308
you can go up you can go down you can start a video
309
so there's like three interactive elements in the dom um the
310
the you think just think about it on your computer like really
311
if you're on your computer right
312
now like how many um how many actions are available to you on the current screen uh
313
or with like the hotkeys like it's it's a lot and if
314
and typically computer use has been this benchmark where you think
315
of oh only the smartest frontier reasoning models can actually figure this out um well maybe not like maybe actually
316
if you could just enumerate like all of the possible options to you, you can navigate it with very quickly.
317
And we're going to get towards real -time machine learning examples in a little bit.
318
Okay.
319
So it also must compose generatively.
320
Okay.
321
So there is a weakness with decisions only.
322
And one way that you might think about this is um like the the symmetry
323
so so um not just the symmetry
324
but like the the pre -definition two of the tree
325
so like in these examples of like the halving search
326
or this like hierarchical search
327
so like here i'm calling it outline walk it's a similar example
328
you find the title you find the um the h1 heading
329
you find the subsection heading you find the paragraph you find the sentence you find the word right um
330
so at each one of those those layers of abstraction you can call another one
331
so that's not like super symmetric like this like pure binary search um having one
332
but the paths are are essentially like predefined um
333
or need to be programmatically defined um and what's very interesting about then
334
when you can um now have a generative element
335
which i'm showing as like the square here
336
so like in this previous one
337
it was like the generative ones diverge the um the decision models converge generative ones diverge decision models converge
338
it's kind of an animation similar idea here now you can start to have
339
so these are actual real call trees from the demo
340
i'm about to show you the smart house you have these much more organic shapes
341
and these much more like emergent graphs
342
and i think there's this um i have this idea
343
that like that pairing the yin and yang of the decision model
344
and the generative model itself is kind of this very powerful automaton right
345
that can kind of ratchet through the system and and generate emergent behavior complex systems from the pairing of these things.
346
And you can get some sense of
347
that from the different geometries of these graphs that you're looking at without seeing the actual behavior.
348
TypeSafe had a really cool cookbook that they also launched with,
349
or maybe it was called a demo or whatever, of a smart home.
350
And so the key thing from the smart home demo
351
that they showed was when the problem cannot be solved purely with a decision,
352
then it can go to an LLM to do generative element.
353
So what I've done is adapt the smart home demo, make it many more rooms in the home,
354
many more appliances and things like that to show how the recursive decision model can interact with the home.
355
You can express a very high level thing like we're off to bed
356
and then you can ask that
357
and it will do all of the sub steps very very quickly to get you to door locked
358
and lamped on uh lamp on everything off
359
and then you can have um these more like oblique type things
360
that it's learning how to do over time let's watch a film um
361
and then we will ask now it's going to uh like
362
work through all those things in order for the decision model
363
and the language model to talk to each other we need some kind of shared interface
364
and so this is what i'm calling the map and
365
so again the decision model can only point it can judge various things
366
that were given to it
367
and lms in their sort of most raw form have no structure to their output
368
or it's perfectly open -ended stream of tokens, right?
369
And so what we need is to kind of have a shared interface.
370
And so I call this the map. And the map is...
371
you could think of it as a set of like pointers in the sense
372
that it could literally have like a name like tv on
373
that has a very simple description and then some function
374
or in this case it's like flipping this boolean to be on right it might have a much more complicated
375
function underneath it like this sort of like loop here
376
but the key idea is that the the decision model can decide from the map
377
which things are the right thing it doesn't
378
so it can decide lights on the code underneath
379
it can implement the light can implement the lambda function lights on okay the llm
380
now has a structured interface to add and edit
381
and modify the map so the llm can basically be learning from new user behavior
382
that okay this movie time is now the combination of turning on the tv
383
and turning off the lights and so now there is a shared interface, the shared substrate, that both can talk to.
384
I'm not going to go into it here because it gets a little bit more complicated,
385
but I also want to point out that the map itself can be multidimensional,
386
so it can essentially be representing embeddings. For example,
387
Kieran Clausen from Every had a really interesting idea of using JEV to create low -dimensional but highly semantically meaningful embeddings.
388
So this map that I built into the RDM codebase can handle that as well.
389
The last property or the last primitive or behavior
390
that I have included in my recursive decision models is what I'm calling the arbiter.
391
And here's why this is important, which is JEV is fast and cheap.
392
There's decision models
393
that can run on your phone on a crappy 10 year old laptop on um obviously
394
you can get a better one on your new um your
395
new macbook pro whatever um these things are very fast they're very inexpensive either from a compute perspective
396
or a dollars perspective and that means
397
that we can run many at once we can we can actually speculatively run a bunch of them
398
when we're uncertain all in parallel and now the thing is
399
that we need to if we just let all of those act on the state
400
we're going to have a lot of collisions we're going to have a lot of challenge so
401
an example from how your cpu is working would be like that there can be many things reading in parallel concurrently,
402
but now we need some kind of lock when we write because we don't want to collide, have colliding writers.
403
There's also an interesting idea also from Alex Zeng at MIT in Omar's group, The Adventure of RLMs, about speculative tool calling,
404
speculative programmatic tool calling for agents.
405
And this is, you know, informed my thinking here.
406
But here's here's this idea applied to audio, which is like we have, uh, I'm speaking, uh, these sentences we're having,
407
we're making decisions where I haven't even actually like finished the sentence.
408
And so we essentially have, um, at various times, the arbiter is like deciding,
409
okay, with the given information that I have across all of these different channels, what is the decision that I'm going to allow through?
410
So you could, you could imagine
411
that like each one of these lanes might like want to might be voting for like a certain action
412
and the arbiter's job is to figure out like where it has the confidence to actually like allow
413
that action to change the state because again
414
if we let them all act
415
we could have a lot of collisions let's say just a simple example right there's two there's two
416
that are making uh they're different models they're making the same kind of decision um
417
and one of them uh goes a little bit faster changes
418
the state um before the second one makes a different decision
419
that would have changed the state the other way you can imagine all kinds of problems
420
or collisions that could happen we do that
421
so this this pattern of the arbiter is what i've developed um as a way of like um um
422
it's like a gate for what can actually now act on the system change the state of the system
423
and it's very useful when you want to run many of these things in parallel okay
424
so i'd love you to try it my very first implementation of rdm was as a dspy module
425
this is in part
426
because the dspy team in particular the the folks at compound
427
ai were very very quick to add support for system one api calls within dspy
428
and secondly dspy is a very opinionated way of working.
429
It's a very structured way of working, and I found this to be a very useful playground for essentially ensuring
430
that the ideas of recursive decision models were structured in the right way,
431
that the modules were behaving in the right way.
432
I have also made a standalone
433
Rust package, RDM, which is great when you want to bring very,
434
very fast and portable RDMs to all kinds of systems.
435
So yeah, check it out in either my DSPy fork, if you're a DSPy nerd,
436
or the RDM package if you want to play around with Rust.
437
okay so this is where we move into operating system one
438
and i mean this uh with both of these phrasings right um
439
this is an operating system one it's operating system one
440
and we're also learning how to operate system one models right
441
so that's sort of the plan words here with operating system one
442
and what's really important to me um personally is this physical enmeshment
443
that we have with with our devices right um
444
i got like technic right like my head's always down i've got this like crazy,
445
like shoulder problem and tension from mousing all the time.
446
When I was in grad school with really stressful deadlines, I was having all kinds of, um, repetitive stress issues.
447
And I've got every crazy vertical mouse and track ball
448
and ergonomic keyboard that you could possibly imagine
449
but the like there is great um power
450
and economic benefit in working with computers in working with devices in working with um our phones
451
and our tablets in terms of being interconnected
452
and in terms of being able to just do our normal jobs
453
and obviously anyone who is an ai engineer is going to need a computer to do their job
454
and at the same time like there is just all of this um i would say unnecessary suffering
455
that is caused by the coupling of our physicality like the
456
coupling of our physical bodies to the physical computer like through the traditional interfaces
457
that have made their way in as the popular ones of society the QWERTY keyboard, the mouse, the touchscreen, etc. Okay,
458
so some of you might recognize this sequence.
459
I used it in my Recursive Coding Agents talk, and it's also on the back of a t -shirt
460
from the AI Engineer code event in New York a couple years ago.
461
And so here we go.
462
We've got the evolution of the AI Engineer, you know, typing on the laptop, not great posture.
463
You know, now here it's got like the ai uh sparkle features
464
so we're vibe coding here maybe um you know in the third one
465
you can see there's no more computer anymore
466
so now we're just like talking to this like orb um that's pretty cool
467
and uh then in this ultimate like state of enlightenment right we just like think
468
and and things are manifested uh in in our world
469
or i don't even know what kind of uh wavelengths are coming out of this guy's head
470
so i'm going here i'd like to get to the the right um making some progress
471
but i think right now with uh with operating system one
472
and my recursive decision models i'm i'm much closer to to the audio piece um and
473
so i want to um
474
i want to sort of take a um to go back in time a little bit um
475
this these videos are um i'm getting old
476
they're they're coming up on not quite 20 years old it's like 17 18 year old videos.
477
And I want to show you some experiments in what I call radical human computer interfacing
478
from some of my work in music.
479
I'm just going to show these really quick and I'll explain a little bit what's going on.
480
So over here. So just watch here.
481
I really set this up.
482
all right so we've got like hitting these four different locations on the laptop and that is
483
being classified to make a drum kit right
484
so the laptops used to have accelerometers built into them back
485
when we're still have disk spinning disk hard drives this one i'm not going to play um
486
so much of but you can um uh in
487
that left one i'm like triggering the drum kit here
488
you can use the tilt of the um laptop to control different effects
489
i'm using the hit of this thing as the sidechain compression um
490
so it's like a ducking compressor every time i hit it uh
491
and this is much longer but this is um i want to bring this up um
492
because it's personally meaningful to me is also the way
493
that I actually learned how to program and the way that I learned machine learning.
494
And this is through the Princeton Laptop Orchestra.
495
It just celebrated its 20th anniversary this spring.
496
And really the idea of the Princeton Laptop Orchestra was what does it mean to have the laptop itself be the instrument?
497
So there were pieces we were performing where the trackpad was the instrument
498
there were pieces where the keyboard was the instrument
499
there were pieces where all the computers were networked together on wi -fi
500
and like syncing super super quick like low latency clock times to send messages back
501
and forth to each other
502
and I was able to work with this brand new cutting
503
edge real -time machine learning framework called the Wekinator developed by Rebecca Feebrink, um, to actually develop, um, and again, this is like,
504
this is like 19 years ago, 18 years ago, um, these, um, very, very small, um, you know, K nearest neighbors,
505
um, like single layer neural nets,
506
um, you know, things that look much more like principal component analysis than like what we think of as like transformers, large language models.
507
And, and, And you can actually do what you might call like on -the -fly machine learning to classify
508
and map and do nonlinear interpolations.
509
It was just so much fun.
510
So I've been having a lot of fun with coming back to
511
that a little bit and kind of merging my excitement about RLMs, my excitement about JEV, these decision models,
512
this invention of recursive decision models and now towards, um, this kind of personal goal of just like,
513
like I want to be able to get things done through my computer and,
514
uh, work in this field without hurting, without like touching a keyboard all day, without touching a mouse all day.
515
Um, and so, uh, I'm going to show this example.
516
This is sort of a precursor to, to, oh, so that I called earshot and I want to set this up
517
and explain it because it's actually a little bit confusing video to watch.
518
So you can think of this as a recursive decision model.
519
We have one layer of the system that is transcribing the audio
520
and then choosing an action to take based on the transcribed word.
521
And so here you're just going to see it's going to be cycling through my different workspaces, one through eight.
522
Um, and then, uh, while that's happening, there is a very small, um, uh, based in Chuck,
523
there's a new thing called Chai.
524
That's like the Chuck AI.
525
That's sort of the successor to the Wekinator that I mentioned before.
526
Very, very small model model operating on a very, very small slice of audio, 40 milliseconds, 80 milliseconds, a hundred milliseconds.
527
Right and essentially trying to just map from some very crude binning of
528
so the features are just like some bins of audio frequency bins of audio
529
which is the right one now that's supervised right
530
so it knows nothing in the beginning and we can't trust it in the beginning so So...
531
There is one layer on which there's like a teacher model that's teaching this small real -time model, right?
532
And then there's a second layer on which the system is independently figuring out
533
when it actually trusts the small model enough to follow it.
534
So we're going to watch it now.
535
And what you're going to see is in the beginning, there's a big delay between the word.
536
And this comes from Philip Glass's Knee Play 1, part of Einstein on the Beach and um so you're gonna see
537
that it's going look up here for the one two three four
538
so there's a big delay right
539
now three four five six seven eight one two three now it's it's supervised learning
540
so this early numbers where it's hearing them more one two three four we're gonna learn a little faster
541
so now see how it's speeding up a little bit for the early ones
542
so there's a lack in the leading ones by the end of this thing you're going to see
543
that we've got down to essentially 45 millisecond time here
544
we go one two three four one two three four five six one so now for those really locked in.
545
So this is recursive decision model deciding when to trust this smaller supervised model
546
that is learning on the fly and operating on tens of milliseconds of information on the pure audio.
547
5, 6, 7, 8, 1, 2, 3, 4, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6,
548
1, 2, 3, 4, 5, 6, 7, 8.
549
Anyways, to me, that's super exciting.
550
Really, really fun to come back to these real -time, on -the -fly machine learning algorithms,
551
which is actually how I first learned about machine learning, and I first learned programming through music so um i also want to
552
so that's the audio example
553
i also want to give a shout out to this project called hunch uh matt mastrocki
554
i'm not sure exactly how to pronounce
555
it he sent something really crazy visually messing with just embedding gemma 2 um
556
and essentially having this thing play pictionary
557
so he has this thing classifying your doodles um on your machine in web gpu in like,
558
look at these numbers, like 150 milliseconds.
559
So again, I want, I just want to give this a shout out.
560
I was like, I'm not the only one working in this space.
561
Audio isn't the only, um, way in, but these, these very, very, very small, um, embedding models, encoder models, classifier models, right?
562
Like there's a whole resurgence of this.
563
And if you can, uh, structure them and architect them into a bigger space, you can do really, really powerful things.
564
Okay.
565
So So with this, this is kind of where I'm going.
566
It's like, how can we decouple our bodies from our devices?
567
What would it mean to work on a computer without touching it?
568
What would it mean to work on a computer without being able to see it, right?
569
And that's really what I'm building up towards with this project that I have called Oso, O -S -O dot computer.
570
It's short for operating system one it's not quite ready um
571
but that earshot demo
572
that i showed you where we're very quickly from a single word even before uh even at a speed
573
that would be faster than we can actually transcribe what we're saying we're able to react
574
and respond um to to our voice
575
so we're starting starting with voice as the first input um
576
and yeah just like a little bit about oso as like a teaser
577
and i hope you'll join the wait list and try it out
578
when it's ready is like it is a computer use agent
579
but it's built on recursive decision models so yes there are regular agents
580
and rlm agents and you can actually um uh you can um sort of farm out jobs
581
or um give them to like your normal coding agents like cloud code
582
and codex um but it's it's decision first right and
583
so you can also do things
584
that are super fast without touching the keyboard without touching the mouse
585
it's recursively improving it's constantly trying to learn what the preferences are of the user
586
and it's also trying to essentially always like
587
that um earshot demo the the philip glass demo
588
it's also trying to like farm out the decision to the the fastest smallest um available model once
589
it can build trust in that
590
so that's this idea of the arbiter right
591
it's like figuring out okay where have we seen this thing do well before
592
and then now how much can i trust it over time
593
so it's a it's an evolving system
594
it is a tool to control your computer without touching
595
it again we're starting with voice
596
it's an experiment in what i consider radical human computer interfacing let's like rethink everything
597
i think there's this really interesting thing about everyone going into the terminal, um, with cloud code.
598
And I don't think it's actually anything, um, special about the terminal.
599
That's why, um, that happened.
600
The terminal is clearly not like the best interface for it.
601
I think a big part of what happened over the last
602
couple of years of us all going into the terminal was really about,
603
uh, just realizing the limitations of these default interfaces that we've been given these default application interfaces,
604
these default visual, um, interfaces, touchscreen interfaces mice keyboards all this stuff um
605
and then yeah lastly it's just like um i'm having
606
so much fun uh coming back to uh what i consider my roots in in real -time machine learning
607
and and it's really fun to be building this this computer use agent also
608
so anyways i hope you will check it out
609
i'm really grateful for your attention this is a long talk
610
i'm using the opportunity to do the virtual talk to to really go deep
611
and yeah I hope you'll check out the recursive decision model repositories
612
and I hope you'll sign up for this waitlist when OSO is ready at oso .computer thank you so much

App downloaden

Alles wat je nodig hebt om vloeiend te spreken

AI UitspraakScoor elke zin
IPA OefeningBeheers elk geluid
WoordenschatBouw je woordenschat op
WoordenspelLeren terwijl je speelt

Zelfde Kanaal

FOR YOU

Aanbevolen Video

Woordenschat en spreektips bij deze les

Deze spreekles is gebaseerd op de video “Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse”. Deze woorden komen het vaakst terug: decision, model, recursive. Deze video bevat 610 zinnen en 7615 woorden om na te spreken. Het gesproken deel duurt 51:48. De spreker praat in een gelijkmatig tempo van ongeveer 147 woorden per minuut, prettig om te shadowen. 82% van de woorden hoort bij de 3.000 meest gebruikte Engelse woorden; de rest kun je beter vooraf bekijken.

Belangrijke woorden in deze video

15 woorden uit de video die het leren waard zijn, met uitspraak en betekenis:

WoordUitspraakBetekenis
interface zelfstandig naamwoord/ˈɪn(t)ɚˌfeɪs/koppeling, aansluiting
essentially bijwoord/ɪˈsɛnʃəli/essentieel
layer zelfstandig naamwoord/ˈleɪ̯ɚ/laag
laptop zelfstandig naamwoord/ˈlæp.tɑp/laptop, notebook
demo zelfstandig naamwoord/ˈdɛm.oʊ/demo, demonstratie
composition zelfstandig naamwoord/ˌkɑm.pəˈzɪʃ.ən/samenstelling
explore werkwoord/ɪkˈsploɹ/onderzoeken, bestuderen
keyboard zelfstandig naamwoord/ˈki.bɔɹd/toetsenbord, klavier
element zelfstandig naamwoord/ˈɛləmənt/element
mouse zelfstandig naamwoord/maʊs/muis
judgment zelfstandig naamwoord/ˈd͡ʒʌd͡ʒ.mənt/oordeel
implement werkwoord/ˈɪmpləmənt/implementeren, (in de praktijk) toepassen
graph zelfstandig naamwoord/ɡɹæf/grafiek
binary bijvoeglijk naamwoord/ˈbaɪ.nə.ɹi/binair, binaire
cloud zelfstandig naamwoord/ˈklaʊ̯d/wolk

Phrasal verbs die je zult horen

WoordUitspraakBetekenis
figure out werkwoorduitvogelen, er achter komen
check out werkwoordonderzoeken
get off werkwoordvan ... afgaan
point out werkwoordaanwijzen, aanduiden
build up werkwoordopstapelen
come up with werkwoordbedenken, verzinnen
go down werkwoordafdalen
look up werkwoord/ˌlʊk ˈʌp/opkijken

Grammatica in deze video

De structuren die de spreker het meest gebruikt, met de exacte woorden uit de video:

StructuurIn de video
Present perfect continuous have/has been + -ing — iets dat eerder begon en nog bezig isI've been developing · I've been sharing · I've been having
Present perfect have/has + voltooid deelwoord — iets uit het verleden dat nu nog teltI've realized · has been · have seen
Lijdende vorm be + voltooid deelwoord — het gaat om wat er gebeurt, niet om wie het doetis originally titled · is now connected · be solved
Betrekkelijke bijzinnen who / which + zin — extra informatie over een persoon of dingthis which is · important, which is · audio, which is

Uitspraak om op te letten

De spreker gebruikt 124 samentrekkingen en verkorte vormen, zoals I'm, we're, don't. Spreek ze kort uit, zoals je ze hoort.

  • De “th”-klanken: underneath /ˌʌndɚˈniθ/, algorithm /ˈælɡəɹɪðm̩/
  • De klanken “sh” en “zh”: essentially /ɪˈsɛnʃəli/, composition /ˌkɑm.pəˈzɪʃ.ən/, implementation /ˌɪmplɪmənˈteɪʃən/, collision /kəˈlɪʒn̩/, contrast /kənˈt(ʃ)ɹæst/
  • Lange woorden — let op de klemtoon: essentially /ɪˈsɛnʃəli/, composition /ˌkɑm.pəˈzɪʃ.ən/, implementation /ˌɪmplɪmənˈteɪʃən/, complicated /ˈkɑm.plɪˌkeɪ.tɪd/, experiment /ɪkˈspɛɹ.ə.mənt/

Klanken die Nederlandstaligen lastig vinden:

  • Stemhebbende eindmedeklinker — /b/, /d/, /g/, /z/, /v/ niet verscherpen: keyboard /ˈki.bɔɹd/, anyways /ˈɛn.iˌweɪz/, cloud /ˈklaʊ̯d/, complicated /ˈkɑm.plɪˌkeɪ.tɪd/, package /ˈpæk.ɪd͡ʒ/
  • /æ/ — opener dan de Nederlandse “e”: laptop /ˈlæp.tɑp/, graph /ɡɹæf/, parallel /ˈpæ.ɹəˌlɛl/, yang /ˈjæŋ/, contrast /kənˈt(ʃ)ɹæst/
  • /g/ — een harde plofklank, geen Nederlandse “g”: graph /ɡɹæf/, algorithm /ˈælɡəɹɪðm̩/, gate /ˈɡeɪ̯t/, organic /ɔːˈɡænɪk/, grateful /ˈɡɹeɪtf(ə)l/

Zo oefen je met deze video

  1. Luister de hele video één keer zonder te spreken en noteer de woorden die je niet kent.
  2. Spreek zin voor zin na op normale snelheid en herhaal elke zin tot je ritme gelijk is aan dat van de spreker.
  3. Neem jezelf op en vergelijk met het origineel; let daarbij op woorden als interface, essentially, layer.

Wat is de Shadowing-techniek?

Shadowing is een wetenschappelijk onderbouwde taalleermethode die oorspronkelijk is ontwikkeld voor professionele tolkentraining en gepopulariseerd door polyglot Dr. Alexander Arguelles. De methode is eenvoudig maar krachtig: je luistert naar native Engelse audio en herhaalt het onmiddellijk hardop — als een schaduw die de spreker volgt met slechts 1–2 seconden vertraging. In tegenstelling tot passief luisteren of grammaticadrills, dwingt shadowing je hersenen en mondspieren om echte spraakpatronen tegelijkertijd te verwerken en te reproduceren. Onderzoek toont aan dat het de uitspraaknauwkeurigheid, intonatie, ritme, verbonden spraak, luisterbegrip en spreekvaardigheid aanzienlijk verbetert — waardoor het een van de meest effectieve methoden is voor IELTS Speaking-voorbereiding en echte Engelse communicatie.

Shadowing-techniek: lees de volledige stap-voor-stap-gids →