Shadowing Practice: Data Ingestion Best Practices! How to Ingest Data Efficiently at Enterprise Scale! - Learn English Speaking with Video
בחר מצב תרגול
כל מצב פותח חדר תרגול במסך מלא.
133 משפטים
אם המשפטים קצרים או ארוכים מדי, לחץ על ערוך כדי להתאים אותם.
1
Hey y'all, Data Guy here and today I'm back with yet another viewer request video,
2
this time talking about best practices for data ingestion.
3
So really focusing on the first part of your data pipelines, really the first part of almost any data engineering workflow,
4
right, is bringing data into the platform that you want that data to be brought into.
5
And so that's what we're going to explore today, is really how you can design effective systems for data ingestion regardless of what tools you're using.
6
So this isn't really a video exploring, hey, what tools you use for data ingestion, but rather what high level design practices, which are really global to any kind of data ingestion use case,
7
should you apply
8
when you're designing your data ingestion workflows to make sure you're designing them as efficiently as possible
9
and you're designing them for enterprise scale because that's what everyone wants, right?
10
So if you like these videos, please like, subscribe, consider joining my Patreon, It helps me out a ton, but without further ado, let's get into it.
11
Now, when you're initially designing your data pipelines and data ingestion, you want to design for scale and elasticity above all else.
12
Especially in cloud architectures, it is a necessity because data volumes can spark unpredictably due to external events, user behavior, system interactions.
13
There's so many different ways that your data volumes can spike unexpectedly.
14
And so your best practice here is going to be using things that are serverless or auto-scaling.
15
So, you know, servers like AWS Lambda functions or Google Cloud functions can ingest bursts of data without provisioning infrastructure,
16
but really any kind of serverless or auto-scaling component that you might want to use to, or
17
that you're going to need to use to make sure you
18
can handle those bursts of data without having a bunch of extra compute sitting idly by.
19
And in tandem with this, managed integration services like AWS Kinesis
20
or Google PubSub also help provide built-in scalability where you only pay for what you use
21
and so they're designed for high throughput ingestion in that way
22
because they'll automatically scale to match the amount of data that you actually need to adjust.
23
And another thing to you know think about designing for even
24
at the earlier stages of your data ingestion pipelines are using time-based
25
or key-based partition strategies to in your object storage systems to prevent any performance bottlenecks downstream,
26
you know, by making your data partitioned effectively earlier on, then the downstream queries don't need to query and go over all of that data.
27
They only need to go toward through their relevant partition.
28
Now, the next thing you want to think about is ensuring data quality at your ingestion point.
29
It's a lot cheaper to catch errors early and trim bad data than processing it and trying to fix it downstream.
30
Bad data is going to create bad outputs.
31
And so by not putting garbage in, you're not gonna get garbage out.
32
And so to accomplish this, you want to do things like input data quality checks to validate schemas at ingestion, you know, using schema enforcement tools.
33
You know, there's glue schema registry, there's confluent schema registry that you can use to compare the ingested schema versus an existing schema.
34
And there's also open source tools as well to make sure that the data is of the expected shape.
35
And then also implement data contracts, which are basically strict expectations programmatically defined for incoming data from producers using tools like open API,
36
protocol buffers, or even just having JSON schemas that say this data has to match this particular format.
37
And then also, your biggest kind of guard against bad data is leveraging integrating tools like Soda,
38
like Great Expectations, to automatically verify and apply more complex freshness
39
or no value checks or to check distribution of data or type consistency.
40
Those tools will actually help you dig into the data
41
and understand it rather than just doing high level schema or shape validation.
42
So it's a combination of both of those tools using conjunction
43
to help raise any issues as early as possible in the data pipeline to stop that bad data getting processed any further.
44
Now, the next thing you're going to want to think about, especially in today's day and age, is being able to support both batch and streaming workloads.
45
It's highly unlikely in modern stacks that you're only going to be doing batch processing, only going to be doing stream processing,
46
and you're likely going to be doing some kind of hybrid pipeline where, you know, you might be processing historical data in bulk and then streaming any real-time updates after that fact, right?
47
And the best practices here for making sure you can support
48
both of these use cases are to use cloud-native hybrid ingestion frameworks, you know, Databricks structured streaming and,
49
you know, Spark structured streaming essentially is a great way to, hey, if I want to use Spark for both streaming and batch, you can do that.
50
You also have Flink, Apache Flink, which will run both batch and streaming workloads or Apache Beam as well.
51
And within here as well, you know, other than, you know, the kind of tools and platforms you want to use, you also are going to want to design your ingestion logic for item potency
52
and making sure that your ingestion logic can safely replay
53
or reprocess data and get the same result to avoid duplication and loss,
54
because that's really critical with streaming workloads and avoiding data from being processed twice and again gumming up your backend databases.
55
Another good practice here is to separate raw and processed stones.
56
You have a raw data ingestion zone in S3
57
or GCS bucket before anything actually gets processed to support reprocessing
58
and recovery if you ever need to go back and reprocess any raw data as well.
59
Those are all great ways to make sure you're able to accomplish batch and streaming use cases equally effectively.
60
Now the next topic I want to discuss is everyone's favorite and that is secured.
61
Data ingestion is most often the way you're going to be ingesting and opening your systems up to sensitive data,
62
and you're going to need to implement proper protections and observability on that to make sure it stays secure.
63
And the first thing, you're going to want to make sure your data is always encrypted in transit and at rest.
64
So use things like TLS for all network transmission and cloud-native encryption protocols for stored data.
65
and then also implement access controls using your IAM policies to restrict access by source, type, and role and implement least privileged based access
66
so everyone is only scoped to access the data that they are explicitly need for their job, right?
67
And then enable, you know, observability across your entire pipeline, you know, make sure your ingestion pipelines have structured logging
68
and metrics and tracing using tools like Prometheus and OpenTelemetry and Datadog
69
So you can also set up further gates against kind of bad data or, you know, any kind of unexpected or anomalous behavior, which might indicate, you know,
70
some type of, you know, leaking of data or, you know, some kind of degradation in your security.
71
Now, the next thing I'll talk about is designing for fault tolerance and resiliency.
72
A big piece of designing good, effective data investment pipelines is making sure they can gracefully handle failures to maintain SLAs and data integrity.
73
And here, you know, number one, you're going to want to think about ensuring a high availability, deploy ingestion services across multiple zones or regions when possible.
74
You know, things like multi-zone Kafka clusters or regional failover in Event Hub.
75
So if there's a failure or degradation service in one region, your jobs will start to get processed in another active region.
76
Another thing you might want to use is something like a dead letter queue.
77
And that will route any unprocessable records to separate storage instead of just deleting them
78
or throwing them out for later inspection remediation so
79
if you get bad data you can actually look at
80
that bad data down the line understand hey why did we get this
81
and fix the process or the issue that caused
82
that bad data in the first place
83
and then also retry logic making sure you're have retry logical exponential back off
84
so you don't flood systems during outages now
85
if you don't have exponential back off you know
86
and you just have retry logic then all of your pipelines
87
that are failing are going to start trying again and pinging
88
and just going to overload any back-end service that they're connecting to.
89
So you want to make sure that you have exponential backoffs
90
that gets spread out over a longer time period and you don't have those service overloads occur.
91
Now, the next thing you want to think about is metadata capture and lineage tracking.
92
Metadata and lineage are vital for any kind of debugging or governance or auditing use cases.
93
And it's really hard to get that down the line if you aren't capturing it during ingestion.
94
So capture attributes like source system, like the time stand, like the transformation status of every step of your data pipeline
95
so that you understand how data is processed and evolved.
96
And then also implement lineage tracking tools like OpenLineage with Marques
97
or Dataplex to record and track and visualize how data flows go from start to finish.
98
You have a visual representation.
99
You can see how data is changing and evolving
100
and going through your various systems so you can identify bottlenecks or areas of issue.
101
And then also adopt a data catalog.
102
You know, you're going to want to make sure you have a central
103
place to store all of this metadata to make sure it's easily queryable and understandable and has all the context necessary.
104
And here you have, you know, something like AWS Glue catalog or Azure Purview or Google Data Catalog for discoverability and compliance and storing all this metadata.
105
Now, finally, the last piece of advice I have for you is making sure you're choosing the right tool for the job.
106
different ingestion scenarios are gonna require different tools and
107
so it's you're gonna want to make sure you're matching the right technology to your needs
108
and not trying to cram everything in one
109
that is the biggest cause of pain I see out there
110
is hey my company said we're all going all in on Databricks
111
and Spark so everything is gonna happen through Spark
112
and there's a lot of things
113
that aren't really well designed for running with Spark right and
114
so making sure you're using the right tool for each job
115
and having a whole different set of different tools for each scenario you might need to encounter is crucial
116
And it doesn't and won't force you to just cram use cases into tools
117
that aren't designed for that and have them run in efficiently.
118
So some general guidelines, you can see some examples in here, is for any kind of real-time streaming tool,
119
use a dedicated real-time streaming tool like Kafka or Pulsar or Kinesis or Flink or Azure Event Hub or Cloud PubSub.
120
For tools like for change data capture, that's where tools like Debezium is probably the gold standard.
121
Then you also have 5Tran or Airbyte that can also stream changes from operational databases as well.
122
And then for any kind of batch data ingestion, orchestrators like Airflow or Mage
123
or Daxter are really great tools for managing all the different stages of your data pipeline
124
and managing all the different tools you might need to ingest from instead of learning mechanisms
125
and give you a single kind of framework to manipulate data and move data around your environments.
126
And then finally, for IoT or event heavy pipelines, then you're actually going to want to look at time series ingestion platforms like InfluxDB or TimescaleDB,
127
which are going to be really well custom designed for those very specific set of needs you use for those use cases.
128
So that is really all different things I wanted to talk about here.
129
Data ingestion in cloud architectures is no longer, hey, you can just have a one size fits all approach.
130
you need to have modularity, elasticity, flexibility to design effective data pipelines in the day, state, and age.
131
So I hope this video has helped you figure out how to do that.
132
I hope you have a great rest of your day.
133
Data Guy out.
הורד אפליקציה
כל מה שאתה צריך כדי לדבר בשטף
הגיית AIציון לכל משפט
תרגול IPAשלוט בכל צליל
אוצר מיליםבנה את מאגר המילים שלך
משחק אוצר מיליםלמד תוך כדי משחק
אותו ערוץ
C17
Data Modeling Best Practices! Best Practices for Designing Your Data Models!
Daily ConversationUniversity StudentsWorking Adults
C13
What is Master Data Management? MDM Explained for Beginners
Daily ConversationWorking Adults
C12
How to Create Gold, Silver, and Bronze Tables in Postgres for a Medallion Data Architecture!
Business EnglishUniversity StudentsWorking Adults
FOR YOU
המלץ על סרטון
C11
159 משפטים
Top 5 AI Gateway Use Cases | Solo.io
Daily ConversationWorking Adults
C11
114 משפטים
Design a File Upload Service Like a Senior Engineer
Daily ConversationUniversity StudentsWorking Adults
C12
4 משפטים
What is Event Driven Architecture (EDA)?
Daily ConversationUniversity StudentsWorking Adults
C1231 משפטים
Redis Crash Course - the What, Why and How to use Redis as your primary database
Daily ConversationUniversity StudentsWorking Adults
C1120
15 משפטים
Accelerating generative AI training at scale with SageMaker HyperPod | Amazon Web Services
Daily ConversationWorking Adults
C11
497 משפטים
Technical deep dive on Managed Instance for App Service | BRK102
Daily ConversationWorking Adults
C11
689 משפטים
Database Sharding! Designing Data-Intensive Applications chapter 6
Daily ConversationUniversity StudentsWorking Adults
C12
70 משפטים
A Practical Guide To Becoming An AI Engineer (2026)
Daily ConversationUniversity StudentsWorking Adults
C11
164 משפטים
I Bet Nobody Has Ever Explained the Backend Ecosystem Like This Before
Daily ConversationWorking Adults
C11
37 משפטים
How Anthropic uses Claude in Product Management
Daily ConversationUniversity StudentsWorking Adults
B23
121 משפטים
Speaking Like This Makes Your YouTube Videos Addictive
Daily ConversationWorking Adults
אוצר מילים והערות הגייה לשיעור הזה
שיעור הדיבור הזה ברמה C1 מבוסס על הסרטון “Data Ingestion Best Practices! How to Ingest Data Efficiently at Enterprise Scale!”. המילים שחוזרות הכי הרבה: ingestion, tool, pipeline, streaming. בסרטון הזה יש 133 משפטים ו-2130 מילים לתרגול Shadowing. הדיבור נמשך 11:27. הדובר מדבר מהר, כ-186 מילים בדקה, אז צפו לצלילים מחוברים ומקוצרים. רק 80% מהמילים נמצאות בין 3,000 המילים הנפוצות ביותר באנגלית, כך שאוצר המילים מאתגר.
אוצר מילים מרכזי בסרטון הזה
15 המילים המתקדמות ביותר בסרטון, עם ההגייה והמשמעות שלהן:
| מילה | הגייה | משמעות |
|---|---|---|
| spark שם עצם | /ˈspɑɹk/ | נִיצוֹץ, גֵּץ |
| exponential שם תואר | /ˌɛk.spəˈnɛn.t͡ʃəl/ | מעריכי, אקספוננציאלי |
| lineage שם עצם | /ˈlɪn.i.ɪd͡ʒ/ | שׁוֹשֶׁלֶת |
| catalogue שם עצם | /ˈkæt.əˌlɔɡ/ | קָטָלוֹג |
| elasticity שם עצם | /ɪ.læsˈtɪs.ɪ.ti/ | גְּמִישׁוּת |
| glue שם עצם | /ɡlu/ | דֶבֶק |
| automobile שם עצם | /ˈɔː.tə.məˌbil/ | מְכוֹנִית |
| accomplish פועל | /əˈkɑm.plɪʃ/ | בִּצֵּעַ |
| encrypt פועל | /ɪnˈkɹɪpt/ | הצפין |
| freshness שם עצם | /ˈfɹɛʃnəs/ | רעננות |
| potency שם עצם | /ˈpəʊtənsi/ | כוח |
| viewer שם עצם | /ˈvju.ɚ/ | צוֹפֶה |
| attribute שם עצם | /ˈæt.ɹɪˌbjut/ | לוואי, לְוַאי |
| queue שם עצם | /kju/ | תּוֹר |
| restrict פועל | /ɹɪˈstɹɪkt/ | הִגְבִּיל |
Phrasal verbs שתשמעו
| מילה | משמעות |
|---|---|
| figure out פועל | הֵבִין, מצא תשובה |
דקדוק בסרטון הזה
המבנים שהדובר משתמש בהם הכי הרבה, עם המילים המדויקות מהסרטון:
| מבנה | בסרטון |
|---|---|
| סביל (Passive voice) be + past participle — הדגש הוא על מה שקורה, לא על מי שעושה את הפעולה | be brought · being processed · is secured |
| משפטי זיקה (Relative clauses) who / which + clause — מידע נוסף על אדם או דבר | practices, which are · contracts, which are · Flink, which will |
| הווה מושלם (Present perfect) have/has + past participle — פעולה שקרתה בעבר ועדיין יש לה חשיבות או השפעה עכשיו | have structured · has helped |
הגייה ששווה לשים לב אליה
הדובר משתמש ב-43 קיצורים וצורות מקוצרות, כמו you're, don't, gonna. אמרו אותם בצורה הקצרה, כפי שאתם שומעים אותם.
- הצלילים של “sh” ו-“zh”: partition /pɑɹˈtɪ.ʃən/, degradation /ˌdɛɡɹəˈdeɪʃən/, expectation /ˌɛk.spɛkˈteɪ.ʃən/, accomplish /əˈkɑm.plɪʃ/, duplication /d(j)u.plɪˈkeɪ.ʃən/
- מילים ארוכות — שימו לב להטעמה: metadata /ˈmɛt.əˌdeɪ.tə/, exponential /ˌɛk.spəˈnɛn.t͡ʃəl/, elasticity /ɪ.læsˈtɪs.ɪ.ti/, degradation /ˌdɛɡɹəˈdeɪʃən/, automobile /ˈɔː.tə.məˌbil/
צלילים שדוברי עברית מתקשים בהם:
- /æ/ — פתוח יותר מה-“אֶ” העברית, לא כמו “a” ב-“cup”: batch /bæt͡ʃ/, catalogue /ˈkæt.əˌlɔɡ/, cram /kɹæm/, elasticity /ɪ.læsˈtɪs.ɪ.ti/, programmatically /pɹoʊɡɹəˈmætɪk(ə)li/
- /r/ — הלשון לא נוגעת בשום מקום, לא ה-“ר” הגרונית: spark /ˈspɑɹk/, retry /riˈtɹaɪ/, downstream /ˌdaʊnˈstɹiːm/, partition /pɑɹˈtɪ.ʃən/, cram /kɹæm/
- /ŋ/ בסוף מילה — צליל אפי בלי “ג” בסוף: ping /pɪŋ/, incoming /ˈɪnˌkʌmɪŋ/
איך לתרגל עם הסרטון הזה
- הקשיבו לכל הסרטון פעם אחת בלי לדבר, ורשמו את המילים שאתם לא מכירים.
- התחילו במהירות 0.75×, תרגלו Shadowing משפט אחר משפט, וחזרו למהירות רגילה כשזה מרגיש קל.
- הקליטו את עצמכם והשוו למקור, תוך תשומת לב למילים כמו spark, exponential, lineage.
מהי טכניקת השאדואינג?
שאדואינג היא טכניקת למידת שפה מבוססת מדע שפותחה במקור להכשרת מתורגמנים מקצועיים ופופולרית על ידי הפוליגלוט ד"ר אלכסנדר ארגואלס. השיטה פשוטה אך עוצמתית: אתה מקשיב לאודיו באנגלית ילידית וחוזר עליו בקול רם מיד — כמו צל העוקב אחרי הדובר עם עיכוב של 1–2 שניות בלבד. בניגוד להאזנה פסיבית או תרגילי דקדוק, שאדואינג מכריח את המוח והשרירים בפה לעבד ולשחזר דפוסי דיבור אמיתיים בו זמנית. מחקרים מראים שזה משפר משמעותית את דיוק ההגייה, האינטונציה, הקצב, הדיבור המחובר, הבנת השמיעה ושטף הדיבור — מה שהופך אותו לאחת השיטות היעילות ביותר להכנה לIELTS Speaking ולתקשורת באנגלית בעולם האמיתי.