Shadowing Practice: Design a File Upload Service Like a Senior Engineer - Learn English Speaking with Video

Kies een oefenmodus

Elke modus opent een volledig scherm oefenruimte.

114 zinnen

Als zinnen te kort of te lang zijn, klik op Bewerken om ze aan te passen.

How can you design a scalable file upload service?

In this video, I'm going to walk you through a system design discussion where

we are going to build a file upload service step by step, starting from a very simple scenario and then expanding it into a more reliable and scalable solution.

So let's start by listing out some requirements for our file upload service.

These are going to be our functional requirements

and we need to be able to upload

and download files, obviously we need to be able to work with large files we need to make our system secure

and we also have to allow for async processing of the files

so that our users don't have to wait too long

or any background work that we have to perform behind the scenes

so you may start simple where you have your clients

and they have some files that they want to upload to our API

so let's add a component here this is going to be our file upload API

and this is going to be their entry point for their initial request.

So they're going to send the request to our API

and we could have an endpoint like files and then slash upload for example.

We're also going to need some database and this is going to hold our files and metadata for our initial implementation.

It doesn't really matter which specific database we're using but let's say it's some sort of relational database.

So our API needs to be able to store the data inside of our database

and this could be our initial primitive implementation.

Now let's look at our functional requirements.

So we'll be able to upload and download files, store our metadata.

Now the ability to work with large files is somewhat questionable as all of the requests are flowing through our API.

So it's going to quickly become a bottleneck

if many users are attempting to upload a file at the

same time as we have to buffer the file in memory before we can store it in our database.

So I'm going to leave a question mark there even though we should probably fail this requirement.

When it comes to security we can still secure this just fine

and then async processing is somewhat possible with background services but if it's all still running on our single API instance, then we're going to quickly run into the same bottlenecks.

Another problem that we're going to run into is with the fact

that we are initially storing the files and the metadata in our database.

So most databases can store binary data just fine, but this is going to cause problems with page size explosion, as we can assume these files can be large,

and most databases have an upper limit on how large a single page can be.

So you can imagine a single file spanning multiple database pages, and this incurs more costs when writing to the database and also querying from the database.

So it's something to keep in mind.

And this is where we could reach out to a more specialized solution for working with files.

So we can introduce another component into our system, and we're going to call this the object store.

So these are specialized services.

So I'm going to give you a couple examples.

So let's say something like SV or Azure Blob Storage, and they are designed to work with files efficiently.

So with the object store now part of our system, we're going to slightly alter how clients are interacting with the file upload API.

So what's going to happen is instead of having an endpoint to directly upload the file to our API, we're going to introduce a different API endpoint.

So let's add it here and let's call it something like files pre-signed upload.

So the idea here is we get what's called a pre-signed URL

and it's supported by most of these object stores

and this gives you a secure URL that allows you to send a request directly to the object store.

So after getting the pre-signed URL, our client is then going to send another request

which is going to upload the file directly to the object store

and this fixes a couple of big bottlenecks inside of our system.

So first of all the file upload API is no longer

a single point of failure as we can now upload

and download files from the object store using pre-signed URLs

which means we still have security as our users have to send the initial request to the upload API

which means they have to be authenticated in order to even get a pre-signed URL

and then we can get the benefits of object stores where they can work with large files.

They also have support for multi-part upload

which means we can break up a large file into multiple chunks and then stream those to the object store.

This also gives us the ability to pause and resume uploads.

We also get support for deduplication and we also get support for controlling file retention.

So instead of redesigning something like S3 from scratch, we can simply leverage it as another component inside of our file upload API.

Another component we need to support is the ability to have async processing.

So let's say we need to do a couple of operations behind the scenes after a file is uploaded.

So I'm going to add a couple of components here.

So let's say we need to perform virus scanning.

Then we could have things like thumbnail generation or more general preview generation.

We could perform things like optical character recognition or something more general like validation.

So all of these can be fairly long-running operations, especially when we need to support a large number of file uploads.

So how do we implement async processing for all of these operations?

Well, we need to introduce another component into our system.

So let's add some sort of pipeline and this is going to be our queue.

Let's add it here below the database and then we have a couple of ways how we could integrate this.

Either we could have our object storage enqueue a message after upload

and this could kick off the processing inside of our background services as simple message consumers

or we could have our object storage send some sort of callback to our file upload API

and then our file upload API would be responsible for sending a message to the queue on behalf of the object store.

I think the first approach here is probably more scalable as most object stores already have support for this built-in

and they can integrate with many popular messaging systems.

So let me actually move all of these components over here

as I'm going to need the bottom part of the screen for one more thing we want to add

and that is after mostly solving the file upload part we also have to solve how to reliably

and securely allow our users access to these files which means giving them the option to download the files.

So for this we can introduce what's called a content delivery

network where our clients can send a request to download the file and if it's not available, the content delivery network can reach out to our API to provide this file

and this is mostly good for public or static files.

So you can think of things like documentation that can be publicly accessible and things that don't change that often.

Of course, static assets for your website.

And why a content delivery network is good is

because it's usually globally distributed and it can route your user's request to the geographically closest component.

Now, what about requests for some specific files?

Those we can route directly to our API.

And to implement this securely, we're also going to utilize pre-signed URLs, except this is going to be some sort of download endpoint for example

and this is going to follow a similar idea as uploading to the object store

and is going to allow our clients to leverage the power of our object store to get access to their files.

So this final design should now check most of our boxes.

We can upload and download file, we can store the metadata either in the object store

or inside of our custom database and we can pull this value after a callback

or we can consume the message from a queue.

We can work with large files as most object stores support this.

We also have security built in as we have dedicated access

100

control for each file we can make them public we can make them private we can share them between users

101

and we've also got the ability to do async processing as

102

we can pick up messages from a queue from some background worker

103

and perform the work behind the scenes and then notify our file upload service

104

when the processing is done and lastly i just want to leave off with an idea

105

if you want to self-host an object store there are open source options out there like for example rustfs

106

which is fully s3 compatible and you could run it on your own system

107

if for some reason you don't want to use a cloud solution like AWS S3 or Azure Blob Storage.

108

And another option I ran into is Seaweed FS.

109

There are options out there like Mini.io, which was open source but now isn't, then Rust FS or Seaweed FS, and I'm going to leave some useful resources and links in the description of this video.

110

Let me know in the comments what you would change about this design

111

and what other considerations you would have made that I may have possibly missed.

112

Also, if you would like to see more system design discussions like this one, consider leaving a suggestion for what topic I should cover.

113

If you enjoyed this video, gently tap the like button to let me know.

114

Thanks a lot for watching, and until next time, stay awesome!

App downloaden

Alles wat je nodig hebt om vloeiend te spreken

AI UitspraakScoor elke zin

IPA OefeningBeheers elk geluid

WoordenschatBouw je woordenschat op

WoordenspelLeren terwijl je speelt

Scan QR

Download on theApp Store Get it onGoogle Play

FOR YOU

Aanbevolen Video

67 zinnen

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Daily ConversationUniversity StudentsWorking Adults

375 zinnen

Configure storage accounts | AZ-104 | Episode 16

Daily ConversationUniversity StudentsWorking Adults

392 zinnen

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

Interview EnglishUniversity StudentsWorking Adults

127 zinnen

How to design a Deployment Pipeline (GitOps)

Daily ConversationUniversity StudentsWorking Adults

65 zinnen

What Is REST API? Examples And How To Use It: Crash Course System Design #3

Daily ConversationUniversity StudentsWorking Adults

279 zinnen

Microservices with Databases can be challenging...

Daily ConversationUniversity StudentsWorking Adults

Woordenschat en spreektips bij deze les

Deze spreekles op niveau C1 is gebaseerd op de video “Design a File Upload Service Like a Senior Engineer”. Deze woorden komen het vaakst terug: file, upload, store, object, Api. Deze video bevat 114 zinnen en 1716 woorden om na te spreken. Het gesproken deel duurt 8:01. De spreker praat snel, ongeveer 214 woorden per minuut, dus reken op verbonden en ingeslikte klanken. 83% van de woorden hoort bij de 3.000 meest gebruikte Engelse woorden; de rest kun je beter vooraf bekijken.

Belangrijke woorden in deze video

De 15 moeilijkste woorden uit de video, met uitspraak en betekenis:

Woord	Uitspraak	Betekenis
upload werkwoord	/ˈʌpˌloʊd/	uploaden
asynchronous bijvoeglijk naamwoord	/eɪˈsɪŋ.kɹə.nəs/	asynchroon
metadata zelfstandig naamwoord	/ˈmɛt.əˌdeɪ.tə/	metagegevens, metadata
queue zelfstandig naamwoord	/kju/	rij
bottleneck zelfstandig naamwoord		flessenhals
scalable bijvoeglijk naamwoord	/ˈskeɪ.lə.bl̩/	klimbaar, beklimbaar
leverage zelfstandig naamwoord	/ˈlɛv.(ə.)ɹɪd͡ʒ/	hefboom, hefboomkracht
authenticate werkwoord	/ɔˈθɛn.tɪ.keɪt/	authenticeren, authentificeren
geographically bijwoord		geografisch
thumbnail zelfstandig naamwoord	/ˈθʌm.neɪl/	duimnagel
dedicate werkwoord	/ˈdɛdɪˌkeɪt/	inwijden, wijden
query zelfstandig naamwoord	/ˈkwɪɹ.i/	vraag, verzoek
chunk zelfstandig naamwoord	/t͡ʃʌŋk/	stuk, brok
validation zelfstandig naamwoord	/ˌvæl.əˈdeɪ.ʃən/	validering
notify werkwoord	/ˈnoʊtɪfaɪ/	mededelen, waarschuwen

Phrasal verbs die je zult horen

Woord	Betekenis
pick up werkwoord	oppakken, optillen

Uitspraak om op te letten

De spreker gebruikt 22 samentrekkingen en verkorte vormen, zoals we're, I'm, don't. Spreek ze kort uit, zoals je ze hoort.

De “th”-klanken: authenticate /ɔˈθɛn.tɪ.keɪt/, thumbnail /ˈθʌm.neɪl/
De klanken “sh” en “zh”: specialize /ˈspɛʃəˌlaɪz/, deduplication /diːˌdʒuːplɪˈkeɪʃən/, validation /ˌvæl.əˈdeɪ.ʃən/, retention /ɹɪˈtɛnʃən/, documentation /ˌdɑkjəmɛnˈteɪʃən/
Lange woorden — let op de klemtoon: asynchronous /eɪˈsɪŋ.kɹə.nəs/, metadata /ˈmɛt.əˌdeɪ.tə/, authenticate /ɔˈθɛn.tɪ.keɪt/, deduplication /diːˌdʒuːplɪˈkeɪʃən/, validation /ˌvæl.əˈdeɪ.ʃən/

Klanken die Nederlandstaligen lastig vinden:

Stemhebbende eindmedeklinker — /b/, /d/, /g/, /z/, /v/ niet verscherpen: upload /ˈʌpˌloʊd/, specialize /ˈspɛʃəˌlaɪz/, leverage /ˈlɛv.(ə.)ɹɪd͡ʒ/, utilize /ˈjuː.tɪ.laɪz/, primitive /ˈpɹɪmɪtɪv/
/æ/ — opener dan de Nederlandse “e”: static /ˈstæt.ɪk/, validation /ˌvæl.əˈdeɪ.ʃən/, lastly /ˈlæstli/, compatible /kəmˈpæt.ə.bəl/, interact /ɪn.təˈɹækt/
/g/ — een harde plofklank, geen Nederlandse “g”: integrate /ˈɪn.tɪ.ɡɹeɪt/, globally /ˈɡloʊbəli/

Zo oefen je met deze video

Luister de hele video één keer zonder te spreken en noteer de woorden die je niet kent.
Begin op 0,75× snelheid, spreek zin voor zin na en ga terug naar normale snelheid zodra het makkelijk gaat.
Neem jezelf op en vergelijk met het origineel; let daarbij op woorden als upload, asynchronous, metadata.

Wat is de Shadowing-techniek?

Shadowing is een wetenschappelijk onderbouwde taalleermethode die oorspronkelijk is ontwikkeld voor professionele tolkentraining en gepopulariseerd door polyglot Dr. Alexander Arguelles. De methode is eenvoudig maar krachtig: je luistert naar native Engelse audio en herhaalt het onmiddellijk hardop — als een schaduw die de spreker volgt met slechts 1–2 seconden vertraging. In tegenstelling tot passief luisteren of grammaticadrills, dwingt shadowing je hersenen en mondspieren om echte spraakpatronen tegelijkertijd te verwerken en te reproduceren. Onderzoek toont aan dat het de uitspraaknauwkeurigheid, intonatie, ritme, verbonden spraak, luisterbegrip en spreekvaardigheid aanzienlijk verbetert — waardoor het een van de meest effectieve methoden is voor IELTS Speaking-voorbereiding en echte Engelse communicatie.

Shadowing-techniek: lees de volledige stap-voor-stap-gids →

Shadowing Practice: Design a File Upload Service Like a Senior Engineer - Learn English Speaking with Video

Kies een oefenmodus

App downloaden

8 Most Important System Design Concepts You Should Know

How I deploy serverless containers for free

OWASP Top 10 Explained

SDLC Vs STLC: Software Development Life Cycle and Software Testing Life Cycle

Architectural Design Process Explained

Video

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Configure storage accounts | AZ-104 | Episode 16

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

How to design a Deployment Pipeline (GitOps)

What Is REST API? Examples And How To Use It: Crash Course System Design #3

Microservices with Databases can be challenging...

Woordenschat en spreektips bij deze les

Belangrijke woorden in deze video

Phrasal verbs die je zult horen

Uitspraak om op te letten

Zo oefen je met deze video

Wat is de Shadowing-techniek?