Shadowing Practice: Design a File Upload Service Like a Senior Engineer - Learn English Speaking with Video

בחר מצב תרגול

כל מצב פותח חדר תרגול במסך מלא.

114 משפטים

אם המשפטים קצרים או ארוכים מדי, לחץ על ערוך כדי להתאים אותם.

How can you design a scalable file upload service?

In this video, I'm going to walk you through a system design discussion where

we are going to build a file upload service step by step, starting from a very simple scenario and then expanding it into a more reliable and scalable solution.

So let's start by listing out some requirements for our file upload service.

These are going to be our functional requirements

and we need to be able to upload

and download files, obviously we need to be able to work with large files we need to make our system secure

and we also have to allow for async processing of the files

so that our users don't have to wait too long

or any background work that we have to perform behind the scenes

so you may start simple where you have your clients

and they have some files that they want to upload to our API

so let's add a component here this is going to be our file upload API

and this is going to be their entry point for their initial request.

So they're going to send the request to our API

and we could have an endpoint like files and then slash upload for example.

We're also going to need some database and this is going to hold our files and metadata for our initial implementation.

It doesn't really matter which specific database we're using but let's say it's some sort of relational database.

So our API needs to be able to store the data inside of our database

and this could be our initial primitive implementation.

Now let's look at our functional requirements.

So we'll be able to upload and download files, store our metadata.

Now the ability to work with large files is somewhat questionable as all of the requests are flowing through our API.

So it's going to quickly become a bottleneck

if many users are attempting to upload a file at the

same time as we have to buffer the file in memory before we can store it in our database.

So I'm going to leave a question mark there even though we should probably fail this requirement.

When it comes to security we can still secure this just fine

and then async processing is somewhat possible with background services but if it's all still running on our single API instance, then we're going to quickly run into the same bottlenecks.

Another problem that we're going to run into is with the fact

that we are initially storing the files and the metadata in our database.

So most databases can store binary data just fine, but this is going to cause problems with page size explosion, as we can assume these files can be large,

and most databases have an upper limit on how large a single page can be.

So you can imagine a single file spanning multiple database pages, and this incurs more costs when writing to the database and also querying from the database.

So it's something to keep in mind.

And this is where we could reach out to a more specialized solution for working with files.

So we can introduce another component into our system, and we're going to call this the object store.

So these are specialized services.

So I'm going to give you a couple examples.

So let's say something like SV or Azure Blob Storage, and they are designed to work with files efficiently.

So with the object store now part of our system, we're going to slightly alter how clients are interacting with the file upload API.

So what's going to happen is instead of having an endpoint to directly upload the file to our API, we're going to introduce a different API endpoint.

So let's add it here and let's call it something like files pre-signed upload.

So the idea here is we get what's called a pre-signed URL

and it's supported by most of these object stores

and this gives you a secure URL that allows you to send a request directly to the object store.

So after getting the pre-signed URL, our client is then going to send another request

which is going to upload the file directly to the object store

and this fixes a couple of big bottlenecks inside of our system.

So first of all the file upload API is no longer

a single point of failure as we can now upload

and download files from the object store using pre-signed URLs

which means we still have security as our users have to send the initial request to the upload API

which means they have to be authenticated in order to even get a pre-signed URL

and then we can get the benefits of object stores where they can work with large files.

They also have support for multi-part upload

which means we can break up a large file into multiple chunks and then stream those to the object store.

This also gives us the ability to pause and resume uploads.

We also get support for deduplication and we also get support for controlling file retention.

So instead of redesigning something like S3 from scratch, we can simply leverage it as another component inside of our file upload API.

Another component we need to support is the ability to have async processing.

So let's say we need to do a couple of operations behind the scenes after a file is uploaded.

So I'm going to add a couple of components here.

So let's say we need to perform virus scanning.

Then we could have things like thumbnail generation or more general preview generation.

We could perform things like optical character recognition or something more general like validation.

So all of these can be fairly long-running operations, especially when we need to support a large number of file uploads.

So how do we implement async processing for all of these operations?

Well, we need to introduce another component into our system.

So let's add some sort of pipeline and this is going to be our queue.

Let's add it here below the database and then we have a couple of ways how we could integrate this.

Either we could have our object storage enqueue a message after upload

and this could kick off the processing inside of our background services as simple message consumers

or we could have our object storage send some sort of callback to our file upload API

and then our file upload API would be responsible for sending a message to the queue on behalf of the object store.

I think the first approach here is probably more scalable as most object stores already have support for this built-in

and they can integrate with many popular messaging systems.

So let me actually move all of these components over here

as I'm going to need the bottom part of the screen for one more thing we want to add

and that is after mostly solving the file upload part we also have to solve how to reliably

and securely allow our users access to these files which means giving them the option to download the files.

So for this we can introduce what's called a content delivery

network where our clients can send a request to download the file and if it's not available, the content delivery network can reach out to our API to provide this file

and this is mostly good for public or static files.

So you can think of things like documentation that can be publicly accessible and things that don't change that often.

Of course, static assets for your website.

And why a content delivery network is good is

because it's usually globally distributed and it can route your user's request to the geographically closest component.

Now, what about requests for some specific files?

Those we can route directly to our API.

And to implement this securely, we're also going to utilize pre-signed URLs, except this is going to be some sort of download endpoint for example

and this is going to follow a similar idea as uploading to the object store

and is going to allow our clients to leverage the power of our object store to get access to their files.

So this final design should now check most of our boxes.

We can upload and download file, we can store the metadata either in the object store

or inside of our custom database and we can pull this value after a callback

or we can consume the message from a queue.

We can work with large files as most object stores support this.

We also have security built in as we have dedicated access

100

control for each file we can make them public we can make them private we can share them between users

101

and we've also got the ability to do async processing as

102

we can pick up messages from a queue from some background worker

103

and perform the work behind the scenes and then notify our file upload service

104

when the processing is done and lastly i just want to leave off with an idea

105

if you want to self-host an object store there are open source options out there like for example rustfs

106

which is fully s3 compatible and you could run it on your own system

107

if for some reason you don't want to use a cloud solution like AWS S3 or Azure Blob Storage.

108

And another option I ran into is Seaweed FS.

109

There are options out there like Mini.io, which was open source but now isn't, then Rust FS or Seaweed FS, and I'm going to leave some useful resources and links in the description of this video.

110

Let me know in the comments what you would change about this design

111

and what other considerations you would have made that I may have possibly missed.

112

Also, if you would like to see more system design discussions like this one, consider leaving a suggestion for what topic I should cover.

113

If you enjoyed this video, gently tap the like button to let me know.

114

Thanks a lot for watching, and until next time, stay awesome!

הורד אפליקציה

כל מה שאתה צריך כדי לדבר בשטף

הגיית AIציון לכל משפט

תרגול IPAשלוט בכל צליל

אוצר מיליםבנה את מאגר המילים שלך

משחק אוצר מיליםלמד תוך כדי משחק

סרוק QR

Download on theApp Store Get it onGoogle Play

FOR YOU

המלץ על סרטון

67 משפטים

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Daily ConversationUniversity StudentsWorking Adults

375 משפטים

Configure storage accounts | AZ-104 | Episode 16

Daily ConversationUniversity StudentsWorking Adults

392 משפטים

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

Interview EnglishUniversity StudentsWorking Adults

68 משפטים

what is a database index?

Daily ConversationUniversity StudentsWorking Adults

689 משפטים

Database Sharding! Designing Data-Intensive Applications chapter 6

Daily ConversationUniversity StudentsWorking Adults

169 משפטים

Can This Vibe Coder Beat a Senior Developer?

Daily ConversationUniversity StudentsWorking Adults

אוצר מילים והערות הגייה לשיעור הזה

שיעור הדיבור הזה ברמה C1 מבוסס על הסרטון “Design a File Upload Service Like a Senior Engineer”. המילים שחוזרות הכי הרבה: file, upload, store, object, Api. בסרטון הזה יש 114 משפטים ו-1716 מילים לתרגול Shadowing. הדיבור נמשך 8:01. הדובר מדבר מהר, כ-214 מילים בדקה, אז צפו לצלילים מחוברים ומקוצרים. 83% מהמילים נמצאות בין 3,000 המילים הנפוצות ביותר באנגלית; כדאי ללמוד את השאר לפני שמתחילים.

אוצר מילים מרכזי בסרטון הזה

11 המילים המתקדמות ביותר בסרטון, עם ההגייה והמשמעות שלהן:

מילה	הגייה	משמעות
queue שם עצם	/kju/	תּוֹר
leverage שם עצם	/ˈlɛv.(ə.)ɹɪd͡ʒ/	הנפה
dedicate פועל	/ˈdɛdɪˌkeɪt/	הקדיש
distribute פועל	/dɪˈstɹɪbjuːt/	חילק
altar שם עצם	/ˈɔl.tɚ/	מִזְבֵּחַ, בָּמָה
consume פועל	/kənˈsum/	צרך
compatible שם תואר	/kəmˈpæt.ə.bəl/	תואם, מתאים
pause שם עצם	/pɔːz/	אתנח
span שם עצם	/spæn/	זֶרֶת
scan פועל	/skæn/	סָרַק
optical שם תואר	/ˈɑptɪkəl/	אופטי

Phrasal verbs שתשמעו

מילה	משמעות
pick up פועל	הרים, אסף

הגייה ששווה לשים לב אליה

הדובר משתמש ב-22 קיצורים וצורות מקוצרות, כמו we're, I'm, don't. אמרו אותם בצורה הקצרה, כפי שאתם שומעים אותם.

הצלילים של “th”: authenticate /ɔˈθɛn.tɪ.keɪt/, thumbnail /ˈθʌm.neɪl/
הצלילים של “sh” ו-“zh”: specialize /ˈspɛʃəˌlaɪz/, deduplication /diːˌdʒuːplɪˈkeɪʃən/, validation /ˌvæl.əˈdeɪ.ʃən/, retention /ɹɪˈtɛnʃən/, documentation /ˌdɑkjəmɛnˈteɪʃən/
מילים ארוכות — שימו לב להטעמה: asynchronous /eɪˈsɪŋ.kɹə.nəs/, metadata /ˈmɛt.əˌdeɪ.tə/, authenticate /ɔˈθɛn.tɪ.keɪt/, deduplication /diːˌdʒuːplɪˈkeɪʃən/, validation /ˌvæl.əˈdeɪ.ʃən/

צלילים שדוברי עברית מתקשים בהם:

/w/ — שפתיים מעוגלות, לא /v/: query /ˈkwɪɹ.i/, questionable /ˈkwɛst͡ʃənəbl̩/
/æ/ — פתוח יותר מה-“אֶ” העברית, לא כמו “a” ב-“cup”: static /ˈstæt.ɪk/, validation /ˌvæl.əˈdeɪ.ʃən/, lastly /ˈlæstli/, compatible /kəmˈpæt.ə.bəl/, interact /ɪn.təˈɹækt/
/r/ — הלשון לא נוגעת בשום מקום, לא ה-“ר” הגרונית: asynchronous /eɪˈsɪŋ.kɹə.nəs/, securely /səˈkjʊɹ.li/, integrate /ˈɪn.tɪ.ɡɹeɪt/, leverage /ˈlɛv.(ə.)ɹɪd͡ʒ/, redesign /ˌɹiː.dɪˈzaɪn/

איך לתרגל עם הסרטון הזה

הקשיבו לכל הסרטון פעם אחת בלי לדבר, ורשמו את המילים שאתם לא מכירים.
התחילו במהירות 0.75×, תרגלו Shadowing משפט אחר משפט, וחזרו למהירות רגילה כשזה מרגיש קל.
הקליטו את עצמכם והשוו למקור, תוך תשומת לב למילים כמו queue, leverage, dedicate.

מהי טכניקת השאדואינג?

שאדואינג היא טכניקת למידת שפה מבוססת מדע שפותחה במקור להכשרת מתורגמנים מקצועיים ופופולרית על ידי הפוליגלוט ד"ר אלכסנדר ארגואלס. השיטה פשוטה אך עוצמתית: אתה מקשיב לאודיו באנגלית ילידית וחוזר עליו בקול רם מיד — כמו צל העוקב אחרי הדובר עם עיכוב של 1–2 שניות בלבד. בניגוד להאזנה פסיבית או תרגילי דקדוק, שאדואינג מכריח את המוח והשרירים בפה לעבד ולשחזר דפוסי דיבור אמיתיים בו זמנית. מחקרים מראים שזה משפר משמעותית את דיוק ההגייה, האינטונציה, הקצב, הדיבור המחובר, הבנת השמיעה ושטף הדיבור — מה שהופך אותו לאחת השיטות היעילות ביותר להכנה לIELTS Speaking ולתקשורת באנגלית בעולם האמיתי.

טכניקת Shadowing: קרא את המדריך המלא שלב אחר שלב →

Shadowing Practice: Design a File Upload Service Like a Senior Engineer - Learn English Speaking with Video

בחר מצב תרגול

הורד אפליקציה

8 Most Important System Design Concepts You Should Know

How I deploy serverless containers for free

OWASP Top 10 Explained

SDLC Vs STLC: Software Development Life Cycle and Software Testing Life Cycle

Architectural Design Process Explained

Prompt Engineering - Guidelines

Rate Limiter System Design in Under 60 Seconds #techprep #programming #systemdesign

Configure storage accounts | AZ-104 | Episode 16

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

what is a database index?

Database Sharding! Designing Data-Intensive Applications chapter 6

Can This Vibe Coder Beat a Senior Developer?

אוצר מילים והערות הגייה לשיעור הזה

אוצר מילים מרכזי בסרטון הזה

Phrasal verbs שתשמעו

הגייה ששווה לשים לב אליה

איך לתרגל עם הסרטון הזה

מהי טכניקת השאדואינג?