Shadowing Practice: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - Learn English Speaking with Video

בחר מצב תרגול

כל מצב פותח חדר תרגול במסך מלא.

58 משפטים

אם המשפטים קצרים או ארוכים מדי, לחץ על ערוך כדי להתאים אותם.

Hi, and welcome to this series on Zero2Hero for Natural Language Processing using TensorFlow.

If you're not an expert on AI or ML, don't worry.

We're taking the concepts of NLP and teaching them from first principles.

In this first lesson, we'll talk about how to represent words in a way that a computer can process them, with a view to later training a neural network that can understand their meaning.

This process is called tokenization.

So let's take a look.

Consider the word listen, as you can see here.

It's made up of a sequence of letters.

These letters can be represented by numbers using an encoding scheme.

A popular one called ASCII has these letters represented by these numbers.

This bunch of numbers can then represent the word listen.

But the word silent has the same letters, and thus the same numbers, just in a different order.

So it makes it hard for us to understand sentiment of a word just by the letters in it.

So it might be easier, instead of encoding letters, to encode words.

Consider the sentence, I love my dog.

So what would happen if we start encoding the words in this sentence instead of the letters in each words?

So for example, the word I could be one.

And then the sentence, I love my dog, could be one, two, three, four.

Now, if I take another sentence, for example, I love my cat, how would we encode it?

Now we see I love my has already been given one, two, three.

So all I need to do is encode a cat.

I'll give that the number five.

And now if we look at the two sentences, they are one, two, three, four and one, two, three,

five, which already show some form of similarity between them.

And it's a similarity you'd expect because they're both about loving a pet.

Given this method of encoding sentences into numbers, now let's take a look at some code to achieve this for us.

This process, as I mentioned before, is called tokenization, and there's an API for that.

We'll look at how to use it with Python.

So here's your first look at some code to tokenize these sentences.

Let's go through it line by line.

First of all, we'll need the tokenizer APIs, and we can get these from TensorFlow Keras like this.

We can represent our sentences as a Python array of strings like this.

It's simply the I love my dog and I love my cat that we saw earlier.

Now the fun begins.

I can create an instance of a tokenizer object.

The numWords parameter is the maximum number of words to keep.

So instead of, for example, just these two sentences, imagine if we had hundreds of books to tokenize.

But we just want the most frequent 100 words in all of that.

This would automatically do that for us when we do the next step.

And that's to tell the tokenizer to go through all the text and then fit itself to them like this.

The full list of words is available as the tokenizer's word index property.

So we can take a look at it like this and then simply print it out.

The result will be this dictionary showing the key being the word and the value being the token for that word.

So, for example, my has a value of three.

The tokenizer is also smart enough to catch some exceptions.

So, for example, if we updated our sentences to this by adding a third sentence, noting that dog here is followed by an exclamation mark.

The nice thing is that the tokenizer is smart enough to spot this and not create a new token.

It's just dog.

And you can see the results here.

There's no token for dog exclamation, but there is one for dog.

And there's also a new token for the word you.

If you want to try this out for yourself, I've put the code in a colab here.

Take it for a spin and experiment.

You've now seen how words can be tokenized and the tools in TensorFlow that handle that tokenization for you.

Now that your words are represented by numbers like this, you'll next need to represent your sentences by sequences of numbers in the correct order.

You'll then have data ready for processing by a neural network to understand or maybe even generate new text.

You'll see the tools that you can use to manage this sequencing in the next episode.

So don't forget to hit that subscribe button.

הורד אפליקציה

כל מה שאתה צריך כדי לדבר בשטף

הגיית AIציון לכל משפט

תרגול IPAשלוט בכל צליל

אוצר מיליםבנה את מאגר המילים שלך

משחק אוצר מיליםלמד תוך כדי משחק

סרוק QR

Download on theApp Store Get it onGoogle Play

FOR YOU

המלץ על סרטון

34 משפטים

What You'll Practice in This Lesson

This lesson helps you learn English with videos by breaking down how computers understand language, focusing on tokenization. You'll practice listening to clear, explained concepts and repeating key terms, which is great for English speaking practice. The video's slow, deliberate pace makes it easy to follow, so you can focus on both the content and pronunciation.

Key Vocabulary & Phrases

Tokenization: The process of converting words into numbers for computers.
Encode: To change information into a different form (e.g., words to numbers).
Neural network: A computer system modeled after the human brain, used for learning patterns.
Sentiment: The feeling or attitude expressed in a word or sentence.
Frequent: Happening or appearing often (e.g., "most frequent words").
API: A set of tools for building software (used here for tokenization).

Practice Tips for Shadowing

Shadow speech is a great way to improve your speaking. The video's tone is friendly and instructional, so mimic that warmth as you repeat. Pause after each sentence, then say it aloud immediately—this trains your brain to keep up with natural speech. Focus on words like "tokenization" and "neural network" to get their rhythm right. For phrases like "encode words instead of letters," stress the key terms to match the speaker's emphasis. Repeat sections where the speaker explains code (e.g., "create an instance of a tokenizer object") to practice clear, slow pronunciation. This not only boosts your English but also helps you follow technical explanations—perfect for shadow speak practice.

דקדוק בסרטון הזה

המבנים שהדובר משתמש בהם הכי הרבה, עם המילים המדויקות מהסרטון:

מבנה	בסרטון
סביל (Passive voice) be + past participle — הדגש הוא על מה שקורה, לא על מי שעושה את הפעולה	is called · can be represented · been given
הווה מושלם (Present perfect) have/has + past participle — פעולה שקרתה בעבר ועדיין יש לה חשיבות או השפעה עכשיו	has already been given · I've put · You've now seen

מהי טכניקת השאדואינג?

שאדואינג היא טכניקת למידת שפה מבוססת מדע שפותחה במקור להכשרת מתורגמנים מקצועיים ופופולרית על ידי הפוליגלוט ד"ר אלכסנדר ארגואלס. השיטה פשוטה אך עוצמתית: אתה מקשיב לאודיו באנגלית ילידית וחוזר עליו בקול רם מיד — כמו צל העוקב אחרי הדובר עם עיכוב של 1–2 שניות בלבד. בניגוד להאזנה פסיבית או תרגילי דקדוק, שאדואינג מכריח את המוח והשרירים בפה לעבד ולשחזר דפוסי דיבור אמיתיים בו זמנית. מחקרים מראים שזה משפר משמעותית את דיוק ההגייה, האינטונציה, הקצב, הדיבור המחובר, הבנת השמיעה ושטף הדיבור — מה שהופך אותו לאחת השיטות היעילות ביותר להכנה לIELTS Speaking ולתקשורת באנגלית בעולם האמיתי.

טכניקת Shadowing: קרא את המדריך המלא שלב אחר שלב →

Shadowing Practice: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - Learn English Speaking with Video

בחר מצב תרגול

הורד אפליקציה

#1 Machine Learning Specialization [Course 1, Week 1, Lesson 1]

Why AI Tokens are so Expensive - Computerphile

Python3 For Pentesting - Introduction

ChatGPT + Podcasts = English Fluency? Try These 12 Prompts!

Dear Higgsfield, I reverse engineered you in 12 hours

Video

Give me 31 minutes, I will make PyTorch click forever

Crash Course Linguistics Preview

Master AI in 60 Minutes with Vector Databases!

Naive Bayes, Clearly Explained!!!

LangChain in 5 Minutes (Explained Clearly)

Translating Claude’s thoughts into language

What You'll Practice in This Lesson

Key Vocabulary & Phrases

Practice Tips for Shadowing

דקדוק בסרטון הזה

מהי טכניקת השאדואינג?