Pratique du Shadowing: Design a File Upload Service Like a Senior Engineer - Apprendre l'anglais à l'oral avec la vidéo

How can you design a scalable file upload service?

In this video, I'm going to walk you through a system design discussion where

we are going to build a file upload service step by step, starting from a very simple scenario and then expanding it into a more reliable and scalable solution.

So let's start by listing out some requirements for our file upload service.

These are going to be our functional requirements

and we need to be able to upload

and download files, obviously we need to be able to work with large files we need to make our system secure

and we also have to allow for async processing of the files

so that our users don't have to wait too long

or any background work that we have to perform behind the scenes

so you may start simple where you have your clients

and they have some files that they want to upload to our API

so let's add a component here this is going to be our file upload API

and this is going to be their entry point for their initial request.

So they're going to send the request to our API

and we could have an endpoint like files and then slash upload for example.

We're also going to need some database and this is going to hold our files and metadata for our initial implementation.

It doesn't really matter which specific database we're using but let's say it's some sort of relational database.

So our API needs to be able to store the data inside of our database

and this could be our initial primitive implementation.

Now let's look at our functional requirements.

So we'll be able to upload and download files, store our metadata.

Now the ability to work with large files is somewhat questionable as all of the requests are flowing through our API.

So it's going to quickly become a bottleneck

if many users are attempting to upload a file at the

same time as we have to buffer the file in memory before we can store it in our database.

So I'm going to leave a question mark there even though we should probably fail this requirement.

When it comes to security we can still secure this just fine

and then async processing is somewhat possible with background services but if it's all still running on our single API instance, then we're going to quickly run into the same bottlenecks.

Another problem that we're going to run into is with the fact

that we are initially storing the files and the metadata in our database.

So most databases can store binary data just fine, but this is going to cause problems with page size explosion, as we can assume these files can be large,

and most databases have an upper limit on how large a single page can be.

So you can imagine a single file spanning multiple database pages, and this incurs more costs when writing to the database and also querying from the database.

So it's something to keep in mind.

And this is where we could reach out to a more specialized solution for working with files.

So we can introduce another component into our system, and we're going to call this the object store.

So these are specialized services.

So I'm going to give you a couple examples.

So let's say something like SV or Azure Blob Storage, and they are designed to work with files efficiently.

So with the object store now part of our system, we're going to slightly alter how clients are interacting with the file upload API.

So what's going to happen is instead of having an endpoint to directly upload the file to our API, we're going to introduce a different API endpoint.

So let's add it here and let's call it something like files pre-signed upload.

So the idea here is we get what's called a pre-signed URL

and it's supported by most of these object stores

and this gives you a secure URL that allows you to send a request directly to the object store.

So after getting the pre-signed URL, our client is then going to send another request

which is going to upload the file directly to the object store

and this fixes a couple of big bottlenecks inside of our system.

So first of all the file upload API is no longer

a single point of failure as we can now upload

and download files from the object store using pre-signed URLs

which means we still have security as our users have to send the initial request to the upload API

which means they have to be authenticated in order to even get a pre-signed URL

and then we can get the benefits of object stores where they can work with large files.

They also have support for multi-part upload

which means we can break up a large file into multiple chunks and then stream those to the object store.

This also gives us the ability to pause and resume uploads.

We also get support for deduplication and we also get support for controlling file retention.

So instead of redesigning something like S3 from scratch, we can simply leverage it as another component inside of our file upload API.

Another component we need to support is the ability to have async processing.

So let's say we need to do a couple of operations behind the scenes after a file is uploaded.

So I'm going to add a couple of components here.

So let's say we need to perform virus scanning.

Then we could have things like thumbnail generation or more general preview generation.

We could perform things like optical character recognition or something more general like validation.

So all of these can be fairly long-running operations, especially when we need to support a large number of file uploads.

So how do we implement async processing for all of these operations?

Well, we need to introduce another component into our system.

So let's add some sort of pipeline and this is going to be our queue.

Let's add it here below the database and then we have a couple of ways how we could integrate this.

Either we could have our object storage enqueue a message after upload

and this could kick off the processing inside of our background services as simple message consumers

or we could have our object storage send some sort of callback to our file upload API

and then our file upload API would be responsible for sending a message to the queue on behalf of the object store.

I think the first approach here is probably more scalable as most object stores already have support for this built-in

and they can integrate with many popular messaging systems.

So let me actually move all of these components over here

as I'm going to need the bottom part of the screen for one more thing we want to add

and that is after mostly solving the file upload part we also have to solve how to reliably

and securely allow our users access to these files which means giving them the option to download the files.

So for this we can introduce what's called a content delivery

network where our clients can send a request to download the file and if it's not available, the content delivery network can reach out to our API to provide this file

and this is mostly good for public or static files.

So you can think of things like documentation that can be publicly accessible and things that don't change that often.

Of course, static assets for your website.

And why a content delivery network is good is

because it's usually globally distributed and it can route your user's request to the geographically closest component.

Now, what about requests for some specific files?

Those we can route directly to our API.

And to implement this securely, we're also going to utilize pre-signed URLs, except this is going to be some sort of download endpoint for example

and this is going to follow a similar idea as uploading to the object store

and is going to allow our clients to leverage the power of our object store to get access to their files.

So this final design should now check most of our boxes.

We can upload and download file, we can store the metadata either in the object store

or inside of our custom database and we can pull this value after a callback

or we can consume the message from a queue.

We can work with large files as most object stores support this.

We also have security built in as we have dedicated access

100

control for each file we can make them public we can make them private we can share them between users

101

and we've also got the ability to do async processing as

102

we can pick up messages from a queue from some background worker

103

and perform the work behind the scenes and then notify our file upload service

104

when the processing is done and lastly i just want to leave off with an idea

105

if you want to self-host an object store there are open source options out there like for example rustfs

106

which is fully s3 compatible and you could run it on your own system

107

if for some reason you don't want to use a cloud solution like AWS S3 or Azure Blob Storage.

108

And another option I ran into is Seaweed FS.

109

There are options out there like Mini.io, which was open source but now isn't, then Rust FS or Seaweed FS, and I'm going to leave some useful resources and links in the description of this video.

110

Let me know in the comments what you would change about this design

111

and what other considerations you would have made that I may have possibly missed.

112

Also, if you would like to see more system design discussions like this one, consider leaving a suggestion for what topic I should cover.

113

If you enjoyed this video, gently tap the like button to let me know.

114

Thanks a lot for watching, and until next time, stay awesome!

Qu'est-ce que la technique du Shadowing ?

Le Shadowing est une technique d'apprentissage des langues fondée sur la science, développée à l'origine pour la formation des interprètes professionnels. Le principe est simple mais puissant : vous écoutez de l'anglais natif et le répétez immédiatement à voix haute — comme une ombre suivant le locuteur avec un décalage de 1 à 2 secondes. Les recherches montrent une amélioration significative de la précision de la prononciation, de l'intonation, du rythme, des liaisons, de la compréhension orale et de la fluidité.

Pratique du Shadowing: Design a File Upload Service Like a Senior Engineer - Apprendre l'anglais à l'oral avec la vidéo

Why practice speaking with this video?

Grammar & Expressions in Context

Common Pronunciation Traps

Qu'est-ce que la technique du Shadowing ?