शैडोइंग अभ्यास: Top 5 AI Gateway Use Cases | Solo.io - वीडियो के साथ अंग्रेजी बोलना सीखें

अभ्यास मोड चुनें

हर मोड पूरी स्क्रीन का अभ्यास कक्ष खोलता है।

159 वाक्य

यदि वाक्य बहुत छोटे या लंबे हों, तो समायोजित करने के लिए Edit पर क्लिक करें।

Continuing on the theme of AI related connectivity, let's talk about AI gateways.

A really common question we get is, why do I need an AI gateway?

And maybe I'll just build one myself or use my existing API gateway for that functionality.

So let's examine the top five use cases we see in customers adopting an AI gateway.

The first one involves credential management.

So without an AI gateway, a very common pattern you see is I'll create a key here in my LLM provider,

and then I'll go ahead and take that key and distribute it to the application teams,

the teams writing agents, and anyone maybe that they're using an agentic IDE and they need an API key there as well, and you can distribute it in all those locations.

Functionally this works.

The problem is when any one of these developers that has seen

that key maybe leaves the company or accidentally checks it into a public GitHub repository,

the blast radius of that is this entire infrastructure, including your account in the ALM provider.

One thing that an AI gateway helps out with immediately is

that I can have safe and secure storage for my API keys that only my AI gateway has access to.

The AI gateway then can, if we want, go ahead and issue individual keys for specific use cases for specific user groups.

Or I can also integrate with existing authentication mechanisms I have in my organization.

Like if an agent is presenting a JWT, I can use that and map it to the API key I'm providing to the provider.

This is critical just in terms of the blast radius limitation of like, hey, a key is exfiltrated,

or someone has hacked a certain piece of code and gained access to a certain piece of code.

I know exactly how to contain that threat.

And by containing it, I'm not impacting any of the use cases I have for that LLM provider.

I would say that credential management also plays a key role in the other use cases we're about to talk about now.

So let's talk about rate limiting.

Why is rate limiting important?

So maybe you've already seen this in your existing applications where let's say you have,

this app is sending five requests with a token size of let's just say 20K tokens, right?

And over here, this IDE is sending 20 requests with a token size, like a 45k token window,

like every minute or so, right?

Now you end up with turning an agent on and the agent goes bananas.

And all of a sudden it sends, you know, 20,000 requests and a massive context window that it's dumping in.

And it's got like a million tokens, let's say, just in a very short period of time, like in less than a minute.

Now in

that scenario, what's going to happen is the LLM provider is going to rate limit not only based on request count, but also by token count.

So those are both variables in that equation and evaluation.

And when you get shut off, you're going to get shut off, not just for the use case that's abusing it, but account wide, and you're going to get a backoff.

That means maybe this agent, I'm just testing around or it's an evaluation,

but I brought my entire account down and that's all API keys for that account across all the other use cases.

It's impacting developers, it's impacting my production apps that may be customer facing as well.

We have to find a better way to deal with this problem.

And the answer is with an AI gateway to actually support rate limiting

and token rate and rate limiting in particular directly in the gateway.

And that means when I have this same traffic here where I'm coming over and this is the bad actor.

I have our IDE friends over here.

This agent, since it's interacting with the AI gateway first,

the AI gateway can shut off that agent directly while allowing traffic from all other consumers to continue.

That means I have no production or service interruptions from the app or the IDE, and the agent gets really gentle back pressure

and gets turned off for a period of time and then gets opened up again.

It'll be very obvious to that team that they're getting rate limited locally, and then they can fix that problem and address it and come back online.

Or maybe it was a periodic or ephemeral issue with flooding a bunch of traffic, and as soon as they exceed the rate limiting window, then they'll come back online.

But the key thing in any gateway and why API gateways, traditional API gateways are not sufficient to handle this requirement is you need both request rate limiting,

but also token weighted rate limiting.

So I can compute the amount of tokens that are allowed given a certain time window in seconds, minutes, hours or days.

All right.

Use case number three, usage reporting.

So maybe this is familiar to you as you're consuming more and more of provider services.

end of the day you get a nice bill from your provide your LLM provider

and it's a million dollars okay okay who has actually spending

that money okay you may be lucky enough to have multiple API keys

so you get maybe some subdivision of that but again

if you're sharing API keys across the clients

that are using it then you're gonna that's gonna get washed

out more importantly you're not gonna have the very specific details of the identities the groups that are consuming it, the model and the provider used in

that internal detail of how you're actually going to show chargeback or showback to the teams consuming.

So what we want is the ability that we'll get the same million dollar bill over here.

Okay.

But in this scenario, every request that's going through the AI gateway is actually earmarked

or watermarked with the client actually consuming it.

That's how we're doing rate limiting.

And that's why we implement credential management in this layer as well, which means that when I get this $1 million bill,

the access logging and reporting here can actually tell me that this particular app is actually responsible for 50K

and this IDE is responsible for 50K.

Or in like, you know, the token count, it'll report the token counts and you can extrapolate the value there.

And then our friend, the agent that was acting up in terms of rate limiting, their token contribution was around 900K, right?

This is a very, very key attribute to like once you're trying to justify

that AI budget and who's consuming you're trying to even

if you're not charging back those teams you're trying to show

who's consuming consumption reporting in the gateway is a key feature

you'd certainly don't want to you don't have the right details in a traditional API management solution

or API gateway to basically track providers and clients

and token counts to show that charge back

but then also you won't have the granularity to isolate it to given teams all right number four for guardrails.

So for guardrails, what we're going to protect against is a number of things.

The request that's going in, this can be a hijack request, it could be a phishing prompt, it can be anything.

This is your attack surface area of what's going in via natural language prompt.

And they're going to be a common attack vectors that you're going to want to protect against.

Certain prompts that if they contain keywords or they're structured in a certain way, you always want to prevent.

Or perhaps you want to append a system prompt to every request that goes through.

So you can use some semantic guardrails there in line with the request.

Likewise, even if it wasn't a malicious request on the response path, I always have a danger here of data exfiltration, right?

So if I'm asking a very innocent question that's not a phishing attack or trying to be a Trojan horse, I still might get back some customer data,

account information, social security number, this type of thing that I don't want to expose to clients.

And in that case, I have no way of protecting against either of these.

The only thing I can rely on is the actual LLM providers' moderation services, which are definitely valuable but on the request path, I'm never limited.

Like any data can go all the way to the LLM provider here.

So that's actually leaving my network and is out on like open internet and available to the provider.

And I might not want that data even escaping my network, right?

Likewise, there could be very specific requirements I have where maybe the obvious things like credit card numbers and social security numbers,

you can pattern match on those and the provider services can filter those.

But what if I have very unique constraints of what I consider PII for my customer base, or I have intellectual property that I don't want to escape my network,

100

I don't want developers accidentally putting in key bits of code

101

or other IP materials that actually might throw to an LLM provider, that I need to be able to instrument for advanced guardrails as well.

102

So let's look at how an API or an AI gateway solves for that.

103

In the AI gateway, you actually have three controls available.

104

So you have basic guardrails.

105

And what basic guardrails do is the first use case I said, that acting directly on the prompt and the response, it can introspect the request, it can identify keywords, identify patterns,

106

and reject those and reject that request to not allow it to flow through to the upstream LLM provider.

107

Likewise, in the response path, it can inspect the data and it can either reject

108

that outright or it can mask that data as it comes back to the client.

109

The key value there is that I don't have to implement it all of the client apps, these guardrails.

110

I have a centralized point of control.

111

This becomes very important if after going to production, you discover that some key data is being exfiltrated, or you're open to some attack vector.

112

What is the fastest path to remediating that exposure?

113

And it's not rolling out new updates to all the apps you've deployed, and it's not working with your upstream LLM provider to add that to their moderation endpoint.

114

that's actually implementing that in the AI gateway and getting it pushed to production in minutes.

115

The next feature I'll mention is moderation endpoint integration.

116

So again, this is a key feature that the upstream LLM providers all offer, but why would I want to wait necessarily for the full request to go to the LLM

117

and then catch that in the processing path?

118

What I really want to do is inside my network, basically contact the moderation point, and if there's any problem with that request,

119

I want that to actually be cut off in my network and not go farther downstream.

120

A key principle here is trying to eliminate any threats or any potential data exfiltration as soon as you can detect it.

121

And this is a key aspect of that.

122

And then finally, we support what we call advanced guardrails.

123

And really what that is, as an example, is if you wanted to train a small language model, maybe you take like an 8 billion parameter model

124

and you fine-tune it with maybe 500 examples of various types of data

125

that you don't want to see in a request or different types of prompts that you don't want to see,

126

that can be very, very useful to execute inline in the request path as more of a semantic guardrail.

127

So you're not even matching on specific data patterns or data signatures.

128

you're doing semantic controls of, is this suggestive of this, or is this like these other things?

129

And that can be done with a small language model that can be integrated directly with the AI gateway.

130

The last point I want to make is around abstraction.

131

So in this case, we have individual apps, agents, and IDEs going directly against the LLM provider.

132

The impact of that means that they are binding to the very specific API of that provider and also they are actually,

133

if you look behind the scenes, most customers, while they may be consuming the public endpoint,

134

the generalized endpoint for OpenAI, for example, that we all use when we use it, but it's quite likely for sensitive information or for their enterprise use cases,

135

you're consuming a private endpoint there.

136

So you've isolated that LLM in specific infrastructure for your company, and you need to access that endpoint or a specific model.

137

And in that scenario, now the app is going to go to that specific endpoint.

138

There are definitely cases where there can be endpoint availability issues

139

there for a very specific endpoint where it's unavailable for a certain period of time.

140

Now in that case, if that's tied to a production experience, so like a chat bot or any customer facing functionality,

141

then that customer channel is down in that period.

142

Another potential exposure here is if you get rate limited in a given model, or even availability of a given model is down,

143

then you might want to fall back or fail over to another model.

144

But it is very hard to instrument for this.

145

You actually have to build that logic directly into each application agent

146

and you're hoping your IDE supports it that you're using up here, but with an AI gateway we can do that seamlessly.

147

If a given endpoint, private endpoint is down, the LLM will seamlessly fall over based on your configuration to a backup endpoint

148

and this can cascade down to multiple end points, multiple models, and even multiple providers.

149

When we talk about abstraction, if your AI gateway supports a canonical API as we do in Agent gateway,

150

all of your clients can just talk OpenAI's API, and that can be mapped to any provider underneath the cover,

151

whether that's Claude or Vertex or Bedrock or OpenAI itself.

152

It can scale to all of these models while presenting a single canonical and consistent API to the consumers, which makes it much easier as you start to exchange models,

153

whether it's hosted providers or you're starting to run LAMA or another model, open source model internally.

154

you still don't impact the consumers of that because you're keeping that API abstraction the same.

155

So hopefully this was useful in terms of discussing the key use cases

156

and the key value we see with customers in adopting an AI gateway

157

and gives you a good idea of how daunting this task is

158

if you were gonna embark on the path of implementing it on your own.

159

Check out Agent Gateway today.

ऐप डाउनलोड करें

धाराप्रवाह बोलने के लिए ज़रूरी सब कुछ

AI उच्चारण स्कोरहर वाक्य का स्कोर

IPA अभ्यासहर ध्वनि में महारत

शब्दावलीअपना शब्द भंडार बनाएँ

शब्द गेमखेलते हुए सीखें

QR स्कैन करें

Download on theApp Store Get it onGoogle Play

FOR YOU

अनुशंसित वीडियो

124 वाक्य

Agentic AI Frameworks Explained: Workflows, Multi-Agent, & Production

Daily ConversationUniversity StudentsWorking Adults

9 वाक्य

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Daily ConversationUniversity StudentsWorking Adults

101 वाक्य

Data Ingestion Best Practices! How to Ingest Data Efficiently at Enterprise Scale!

Daily ConversationWorking Adults

105 वाक्य

AI Governance Explained: Ethical, Transparent & Accountable AI for the Future

Daily ConversationUniversity StudentsWorking Adults

114 वाक्य

Harnessing generative AI in architecture, engineering & construction

Daily ConversationUniversity StudentsWorking Adults

इस पाठ की शब्दावली और बोलने के सुझाव

यह C1 स्तर का स्पीकिंग पाठ वीडियो “Top 5 AI Gateway Use Cases” पर आधारित है। सबसे ज़्यादा दोहराए गए शब्द ये हैं: gateway, key, provider, request, Api। इस वीडियो में शैडोइंग के लिए 159 वाक्य और 2518 शब्द हैं। बोला गया हिस्सा 13:37 लंबा है। वक्ता तेज़ बोलता है, लगभग 185 शब्द प्रति मिनट, इसलिए जुड़ी हुई और कमज़ोर ध्वनियाँ ज़्यादा मिलेंगी। केवल 80% शब्द अंग्रेज़ी के 3,000 सबसे आम शब्दों में से हैं, इसलिए शब्दावली कठिन है।

इस वीडियो की मुख्य शब्दावली

वीडियो के 15 सबसे कठिन शब्द, उच्चारण और अर्थ के साथ:

शब्द	उच्चारण	अर्थ
consume क्रिया	/kənˈsum/	उपभुक्त
abstraction संज्ञा	/æbˈstɹæk.ʃn̩/	अमूर्तन, भावानयन
semantic विशेषण	/sɪˈmæntɪk/	अर्थसंबंधी
upstream विशेषण	/ʌpˈstɹiːm/	ऊपर को
functionality संज्ञा		क्रियात्मकता
vector संज्ञा	/ˈvɛktɚ/	सदिश
ephemeral विशेषण	/ɛˈfɛ.mə.ɹəl/	क्षणभंगुर
generalize क्रिया	/ˈd͡ʒɛn.(ə.)ɹə.laɪz/	व्यापक करना
parameter संज्ञा	/pəˈɹæm.ə.tɚ/	प्राचल
unavailable विशेषण	/ˌʌnəˈveɪləbəl/	अनुपलब्ध
subdivision संज्ञा	/ˈsʌbdɪvɪʒən/	उपखंड
inspect क्रिया	/ɪnˈspɛkt/	जाँचना, जाँच करना
malicious विशेषण	/məˈlɪʃəs/	द्वेषी
internally क्रिया विशेषण	/ɪnˈtɜːnəli/	आंतरिक
attribute संज्ञा	/ˈæt.ɹɪˌbjut/	गुण, सिफ़त

वीडियो में आने वाली फ़्रेज़ल वर्ब

शब्द	उच्चारण	अर्थ
come back क्रिया		वापस आना
go ahead क्रिया	/ˌɡoʊ əˈhɛd/	आगे बढ़ना

इस वीडियो का व्याकरण

वक्ता जिन संरचनाओं का सबसे अधिक प्रयोग करता है, वीडियो के असली शब्दों के साथ:

संरचना	वीडियो में
Passive voice be + past participle — ज़ोर इस पर कि क्या होता है, न कि कौन करता है	is exfiltrated · are allowed · is actually earmarked
Present perfect have/has + past participle — बीता हुआ काम जिसका असर अभी भी है	has seen · has hacked · you've already seen
Relative clauses who / which + उपवाक्य — व्यक्ति या वस्तु के बारे में अतिरिक्त जानकारी	well, which means · services, which are · consumers, which makes

ध्यान देने योग्य उच्चारण

वक्ता 60 बार संक्षिप्त रूपों का प्रयोग करता है, जैसे you're, don't, I'm। इन्हें वैसे ही छोटा बोलें जैसा आप सुनते हैं।

“th” की ध्वनियाँ: authentication /ɔːˌθɛntɪˈkeɪʃn̩/, underneath /ˌʌndɚˈniθ/
“sh” और “zh” की ध्वनियाँ: moderation /ˌmɑdəˈɹeɪʃən/, abstraction /æbˈstɹæk.ʃn̩/, credential /kɹɪˈdɛnʃəl/, phishing /ˈfɪʃɪŋ/, authentication /ɔːˌθɛntɪˈkeɪʃn̩/
लंबे शब्द — बलाघात सही रखें: moderation /ˌmɑdəˈɹeɪʃən/, canonical /kəˈnɒn.ɪ.kəl/, authentication /ɔːˌθɛntɪˈkeɪʃn̩/, ephemeral /ɛˈfɛ.mə.ɹəl/, extrapolate /ɛkˈstɹæp.əˌleɪt/

हिंदी भाषियों के लिए मुश्किल ध्वनियाँ:

/v/ और /w/ का फ़र्क़ — /v/ में ऊपर के दाँत निचले होंठ को छूते हैं, /w/ में होंठ गोल होते हैं: gateway /ˈɡeɪtˌweɪ/, likewise /ˈlaɪkˌwaɪz/, keyword /ˈkiː.wɜːd/, vector /ˈvɛktɚ/, suggestive /sə(ɡ)ˈd͡ʒɛs.tɪv/
/z/ — इसे “ज” (/dʒ/) न बनाएँ: likewise /ˈlaɪkˌwaɪz/, centralize /ˈsɛntɹəlaɪz/, generalize /ˈd͡ʒɛn.(ə.)ɹə.laɪz/, repository /ɹɪˈpɒz.ɪ.t(ə.)ɹɪ/, expose /ɪkˈspoʊz/
/æ/ — “ऐ” से ज़्यादा खुला: abstraction /æbˈstɹæk.ʃn̩/, semantic /sɪˈmæntɪk/, cascade /kæsˈkeɪd/, extrapolate /ɛkˈstɹæp.əˌleɪt/, granularity /ˌɡɹæn.jəˈlɝ.ə.ti/

इस वीडियो से अभ्यास कैसे करें

पहले बिना बोले पूरा वीडियो एक बार सुनें और जो शब्द नहीं आते उन्हें लिख लें।
0.75× गति से शुरू करें, एक-एक वाक्य की शैडोइंग करें, और सहज लगने पर सामान्य गति पर लौटें।
अपनी आवाज़ रिकॉर्ड करें और मूल से तुलना करें, ख़ासकर consume, abstraction, semantic जैसे शब्दों पर ध्यान दें।

शैडोइंग तकनीक क्या है?

शैडोइंग (Shadowing) एक विज्ञान-समर्थित भाषा सीखने की तकनीक है जो मूल रूप से पेशेवर दुभाषिया प्रशिक्षण के लिए विकसित की गई थी। विधि सरल लेकिन शक्तिशाली है: आप मूल अंग्रेज़ी ऑडियो सुनते हैं और तुरंत इसे ज़ोर से दोहराते हैं — जैसे वक्ता की छाया 1-2 सेकंड की देरी से। शोध से पता चलता है कि यह उच्चारण सटीकता, स्वर, लय, जुड़ी हुई ध्वनियाँ, सुनने की समझ और बोलने की प्रवाहशीलता में काफ़ी सुधार करता है।

शैडोइंग तकनीक: पूरी चरण-दर-चरण गाइड पढ़ें →

शैडोइंग अभ्यास: Top 5 AI Gateway Use Cases | Solo.io - वीडियो के साथ अंग्रेजी बोलना सीखें

अभ्यास मोड चुनें

ऐप डाउनलोड करें

Agentic AI Frameworks Explained: Workflows, Multi-Agent, & Production

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

BGP Overview

How To Solve The App Onboarding Paradox

IAM

Armchair Architects: Resiliency in the Cloud – Core Patterns

it’s boring, but it’s what bursts the ai bubble

Azure App Service and Virtual Network Integration Options

I Tried PewDiePie's Odysseus AI So You Don't Have To

Data Ingestion Best Practices! How to Ingest Data Efficiently at Enterprise Scale!

AI Governance Explained: Ethical, Transparent & Accountable AI for the Future

Harnessing generative AI in architecture, engineering & construction

इस पाठ की शब्दावली और बोलने के सुझाव

इस वीडियो की मुख्य शब्दावली

वीडियो में आने वाली फ़्रेज़ल वर्ब

इस वीडियो का व्याकरण

ध्यान देने योग्य उच्चारण

इस वीडियो से अभ्यास कैसे करें

शैडोइंग तकनीक क्या है?