Shadowing Practice: Design a File Upload Service Like a Senior Engineer - Learn English Speaking with Video

Bir pratik modu seçin

Her mod tam ekran bir pratik odası açar.

114 cümleler

Cümleler çok kısa veya çok uzun ise, ayarlamak için Düzenle'ye tıklayın.

How can you design a scalable file upload service?

In this video, I'm going to walk you through a system design discussion where

we are going to build a file upload service step by step, starting from a very simple scenario and then expanding it into a more reliable and scalable solution.

So let's start by listing out some requirements for our file upload service.

These are going to be our functional requirements

and we need to be able to upload

and download files, obviously we need to be able to work with large files we need to make our system secure

and we also have to allow for async processing of the files

so that our users don't have to wait too long

or any background work that we have to perform behind the scenes

so you may start simple where you have your clients

and they have some files that they want to upload to our API

so let's add a component here this is going to be our file upload API

and this is going to be their entry point for their initial request.

So they're going to send the request to our API

and we could have an endpoint like files and then slash upload for example.

We're also going to need some database and this is going to hold our files and metadata for our initial implementation.

It doesn't really matter which specific database we're using but let's say it's some sort of relational database.

So our API needs to be able to store the data inside of our database

and this could be our initial primitive implementation.

Now let's look at our functional requirements.

So we'll be able to upload and download files, store our metadata.

Now the ability to work with large files is somewhat questionable as all of the requests are flowing through our API.

So it's going to quickly become a bottleneck

if many users are attempting to upload a file at the

same time as we have to buffer the file in memory before we can store it in our database.

So I'm going to leave a question mark there even though we should probably fail this requirement.

When it comes to security we can still secure this just fine

and then async processing is somewhat possible with background services but if it's all still running on our single API instance, then we're going to quickly run into the same bottlenecks.

Another problem that we're going to run into is with the fact

that we are initially storing the files and the metadata in our database.

So most databases can store binary data just fine, but this is going to cause problems with page size explosion, as we can assume these files can be large,

and most databases have an upper limit on how large a single page can be.

So you can imagine a single file spanning multiple database pages, and this incurs more costs when writing to the database and also querying from the database.

So it's something to keep in mind.

And this is where we could reach out to a more specialized solution for working with files.

So we can introduce another component into our system, and we're going to call this the object store.

So these are specialized services.

So I'm going to give you a couple examples.

So let's say something like SV or Azure Blob Storage, and they are designed to work with files efficiently.

So with the object store now part of our system, we're going to slightly alter how clients are interacting with the file upload API.

So what's going to happen is instead of having an endpoint to directly upload the file to our API, we're going to introduce a different API endpoint.

So let's add it here and let's call it something like files pre-signed upload.

So the idea here is we get what's called a pre-signed URL

and it's supported by most of these object stores

and this gives you a secure URL that allows you to send a request directly to the object store.

So after getting the pre-signed URL, our client is then going to send another request

which is going to upload the file directly to the object store

and this fixes a couple of big bottlenecks inside of our system.

So first of all the file upload API is no longer

a single point of failure as we can now upload

and download files from the object store using pre-signed URLs

which means we still have security as our users have to send the initial request to the upload API

which means they have to be authenticated in order to even get a pre-signed URL

and then we can get the benefits of object stores where they can work with large files.

They also have support for multi-part upload

which means we can break up a large file into multiple chunks and then stream those to the object store.

This also gives us the ability to pause and resume uploads.

We also get support for deduplication and we also get support for controlling file retention.

So instead of redesigning something like S3 from scratch, we can simply leverage it as another component inside of our file upload API.

Another component we need to support is the ability to have async processing.

So let's say we need to do a couple of operations behind the scenes after a file is uploaded.

So I'm going to add a couple of components here.

So let's say we need to perform virus scanning.

Then we could have things like thumbnail generation or more general preview generation.

We could perform things like optical character recognition or something more general like validation.

So all of these can be fairly long-running operations, especially when we need to support a large number of file uploads.

So how do we implement async processing for all of these operations?

Well, we need to introduce another component into our system.

So let's add some sort of pipeline and this is going to be our queue.

Let's add it here below the database and then we have a couple of ways how we could integrate this.

Either we could have our object storage enqueue a message after upload

and this could kick off the processing inside of our background services as simple message consumers

or we could have our object storage send some sort of callback to our file upload API

and then our file upload API would be responsible for sending a message to the queue on behalf of the object store.

I think the first approach here is probably more scalable as most object stores already have support for this built-in

and they can integrate with many popular messaging systems.

So let me actually move all of these components over here

as I'm going to need the bottom part of the screen for one more thing we want to add

and that is after mostly solving the file upload part we also have to solve how to reliably

and securely allow our users access to these files which means giving them the option to download the files.

So for this we can introduce what's called a content delivery

network where our clients can send a request to download the file and if it's not available, the content delivery network can reach out to our API to provide this file

and this is mostly good for public or static files.

So you can think of things like documentation that can be publicly accessible and things that don't change that often.

Of course, static assets for your website.

And why a content delivery network is good is

because it's usually globally distributed and it can route your user's request to the geographically closest component.

Now, what about requests for some specific files?

Those we can route directly to our API.

And to implement this securely, we're also going to utilize pre-signed URLs, except this is going to be some sort of download endpoint for example

and this is going to follow a similar idea as uploading to the object store

and is going to allow our clients to leverage the power of our object store to get access to their files.

So this final design should now check most of our boxes.

We can upload and download file, we can store the metadata either in the object store

or inside of our custom database and we can pull this value after a callback

or we can consume the message from a queue.

We can work with large files as most object stores support this.

We also have security built in as we have dedicated access

100

control for each file we can make them public we can make them private we can share them between users

101

and we've also got the ability to do async processing as

102

we can pick up messages from a queue from some background worker

103

and perform the work behind the scenes and then notify our file upload service

104

when the processing is done and lastly i just want to leave off with an idea

105

if you want to self-host an object store there are open source options out there like for example rustfs

106

which is fully s3 compatible and you could run it on your own system

107

if for some reason you don't want to use a cloud solution like AWS S3 or Azure Blob Storage.

108

And another option I ran into is Seaweed FS.

109

There are options out there like Mini.io, which was open source but now isn't, then Rust FS or Seaweed FS, and I'm going to leave some useful resources and links in the description of this video.

110

Let me know in the comments what you would change about this design

111

and what other considerations you would have made that I may have possibly missed.

112

Also, if you would like to see more system design discussions like this one, consider leaving a suggestion for what topic I should cover.

113

If you enjoyed this video, gently tap the like button to let me know.

114

Thanks a lot for watching, and until next time, stay awesome!

Uygulamayı İndir

Akıcı konuşmak için ihtiyacınız olan her şey

AI TelaffuzHer cümleyi puanla

IPA AlıştırmasıHer sesi ustalaştır

Kelime BilgisiKelime bankanızı oluşturun

Kelime OyunuOynarken öğrenin

QR Tara

Download on theApp Store Get it onGoogle Play

FOR YOU

Video Öner

67 cümleler

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Daily ConversationUniversity StudentsWorking Adults

375 cümleler

Configure storage accounts | AZ-104 | Episode 16

Daily ConversationUniversity StudentsWorking Adults

392 cümleler

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

Interview EnglishUniversity StudentsWorking Adults

127 cümleler

How to design a Deployment Pipeline (GitOps)

Daily ConversationUniversity StudentsWorking Adults

65 cümleler

What Is REST API? Examples And How To Use It: Crash Course System Design #3

Daily ConversationUniversity StudentsWorking Adults

279 cümleler

Microservices with Databases can be challenging...

Daily ConversationUniversity StudentsWorking Adults

Bu dersin kelimeleri ve konuşma notları

Bu C1 seviyesindeki konuşma dersi “Design a File Upload Service Like a Senior Engineer” videosuna dayanıyor. En çok tekrarlanan kelimeler: file, upload, store, object, Api. Bu videoda gölgeleme çalışması için 114 cümle ve 1716 kelime var. Konuşma bölümü 8:01 sürüyor. Konuşmacı hızlı konuşuyor, dakikada yaklaşık 214 kelime; bu yüzden birbirine bağlanan ve zayıflayan sesler duyacaksınız. Kelimelerin %83’i İngilizcede en sık kullanılan 3.000 kelime arasında; geri kalanına başlamadan önce bakmakta fayda var.

Bu videodaki önemli kelimeler

Videodaki en ileri düzey 15 kelime, telaffuzu ve anlamıyla:

Kelime	Telaffuz	Anlam
endpoint isim		bitim noktası, bitiş noktası
metadata isim	/ˈmɛt.əˌdeɪ.tə/	üstveri
queue isim	/kju/	kuyruk, sıra
callback isim		geri çağırma
securely zarf	/səˈkjʊɹ.li/	güvenle, güvenlice
leverage isim	/ˈlɛv.(ə.)ɹɪd͡ʒ/	kaldıraç kuvveti
enqueue fiil		sıralamak
relational sıfat		ilişkisel
query isim	/ˈkwɪɹ.i/	sorgulama
chunk isim	/t͡ʃʌŋk/	parça
notify fiil	/ˈnoʊtɪfaɪ/	bildirmek
distribute fiil	/dɪˈstɹɪbjuːt/	dağıtmak, paylaştırmak
primitive isim	/ˈpɹɪmɪtɪv/	ilkel insan
altar isim	/ˈɔl.tɚ/	sunak, mihrap
consume fiil	/kənˈsum/	kullanmak, harcamak

Dikkat edilecek telaffuzlar

Konuşmacı we're, I'm, don't gibi kısaltılmış biçimleri 22 kez kullanıyor. Bunları duyduğunuz gibi kısa söyleyin.

“th” sesleri: authenticate /ɔˈθɛn.tɪ.keɪt/, thumbnail /ˈθʌm.neɪl/
“sh” ve “zh” sesleri: specialize /ˈspɛʃəˌlaɪz/, deduplication /diːˌdʒuːplɪˈkeɪʃən/, validation /ˌvæl.əˈdeɪ.ʃən/, retention /ɹɪˈtɛnʃən/, documentation /ˌdɑkjəmɛnˈteɪʃən/
Uzun kelimeler — vurguyu doğru yere koyun: asynchronous /eɪˈsɪŋ.kɹə.nəs/, metadata /ˈmɛt.əˌdeɪ.tə/, authenticate /ɔˈθɛn.tɪ.keɪt/, deduplication /diːˌdʒuːplɪˈkeɪʃən/, validation /ˌvæl.əˈdeɪ.ʃən/

Türkçe konuşanların zorlandığı sesler:

/w/ — dudaklar yuvarlak, /v/ değil: query /ˈkwɪɹ.i/, questionable /ˈkwɛst͡ʃənəbl̩/
Kelime başındaki ünsüz kümesi — araya ünlü eklemeyin: queue /kju/, scalable /ˈskeɪ.lə.bl̩/, specialize /ˈspɛʃəˌlaɪz/, static /ˈstæt.ɪk/, query /ˈkwɪɹ.i/
/æ/ — “e”den daha açık: static /ˈstæt.ɪk/, validation /ˌvæl.əˈdeɪ.ʃən/, lastly /ˈlæstli/, compatible /kəmˈpæt.ə.bəl/, interact /ɪn.təˈɹækt/

Bu videoyla nasıl çalışılır

Videonun tamamını konuşmadan bir kez dinleyin ve bilmediğiniz kelimeleri not edin.
0,75× hızla başlayın, cümle cümle tekrar edin ve kolaylaşınca normal hıza dönün.
Kendinizi kaydedin ve orijinaliyle karşılaştırın; endpoint, metadata, queue gibi kelimelere özellikle dikkat edin.

Gölgeleme Tekniği Nedir?

Gölgeleme, başlangıçta profesyonel tercüman eğitimi için geliştirilen ve çok dilli Dr. Alexander Arguelles tarafından popüler hale getirilen, bilim destekli bir dil öğrenme tekniğidir. Yöntem basit ama güçlüdür: ana dili İngilizce olan bir sesi dinler ve hemen yüksek sesle tekrar edersiniz — konuşmacıyı 1-2 saniye gecikmeyle takip eden bir gölge gibi. Pasif dinleme veya dilbilgisi alıştırmalarının aksine, gölgeleme beyninizi ve ağız kaslarınızı gerçek konuşma kalıplarını eşzamanlı olarak işlemeye ve yeniden üretmeye zorlar. Araştırmalar, telaffuz doğruluğu, tonlama, ritim, bağlı konuşma, dinleme anlama ve konuşma akıcılığını önemli ölçüde geliştirdiğini göstermektedir — bu da onu IELTS Konuşma hazırlığı ve gerçek dünya İngilizce iletişimi için en etkili yöntemlerden biri yapar.

Shadowing tekniği: adım adım eksiksiz rehberi okuyun →

Shadowing Practice: Design a File Upload Service Like a Senior Engineer - Learn English Speaking with Video

Bir pratik modu seçin

Uygulamayı İndir

8 Most Important System Design Concepts You Should Know

How I deploy serverless containers for free

OWASP Top 10 Explained

SDLC Vs STLC: Software Development Life Cycle and Software Testing Life Cycle

Architectural Design Process Explained

Video

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Configure storage accounts | AZ-104 | Episode 16

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

How to design a Deployment Pipeline (GitOps)

What Is REST API? Examples And How To Use It: Crash Course System Design #3

Microservices with Databases can be challenging...

Bu dersin kelimeleri ve konuşma notları

Bu videodaki önemli kelimeler

Dikkat edilecek telaffuzlar

Bu videoyla nasıl çalışılır

Gölgeleme Tekniği Nedir?