跟读练习: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - 通过视频学习英语口语

Hi, and welcome to this series on Zero2Hero for Natural Language Processing using TensorFlow.

If you're not an expert on AI or ML, don't worry.

We're taking the concepts of NLP and teaching them from first principles.

In this first lesson, we'll talk about how to represent words in a way that a computer can process them, with a view to later training a neural network that can understand their meaning.

This process is called tokenization.

So let's take a look.

Consider the word listen, as you can see here.

It's made up of a sequence of letters.

These letters can be represented by numbers using an encoding scheme.

A popular one called ASCII has these letters represented by these numbers.

This bunch of numbers can then represent the word listen.

But the word silent has the same letters, and thus the same numbers, just in a different order.

So it makes it hard for us to understand sentiment of a word just by the letters in it.

So it might be easier, instead of encoding letters, to encode words.

Consider the sentence, I love my dog.

So what would happen if we start encoding the words in this sentence instead of the letters in each words?

So for example, the word I could be one.

And then the sentence, I love my dog, could be one, two, three, four.

Now, if I take another sentence, for example, I love my cat, how would we encode it?

Now we see I love my has already been given one, two, three.

So all I need to do is encode a cat.

I'll give that the number five.

And now if we look at the two sentences, they are one, two, three, four and one, two, three,

five, which already show some form of similarity between them.

And it's a similarity you'd expect because they're both about loving a pet.

Given this method of encoding sentences into numbers, now let's take a look at some code to achieve this for us.

This process, as I mentioned before, is called tokenization, and there's an API for that.

We'll look at how to use it with Python.

So here's your first look at some code to tokenize these sentences.

Let's go through it line by line.

First of all, we'll need the tokenizer APIs, and we can get these from TensorFlow Keras like this.

We can represent our sentences as a Python array of strings like this.

It's simply the I love my dog and I love my cat that we saw earlier.

Now the fun begins.

I can create an instance of a tokenizer object.

The numWords parameter is the maximum number of words to keep.

So instead of, for example, just these two sentences, imagine if we had hundreds of books to tokenize.

But we just want the most frequent 100 words in all of that.

This would automatically do that for us when we do the next step.

And that's to tell the tokenizer to go through all the text and then fit itself to them like this.

The full list of words is available as the tokenizer's word index property.

So we can take a look at it like this and then simply print it out.

The result will be this dictionary showing the key being the word and the value being the token for that word.

So, for example, my has a value of three.

The tokenizer is also smart enough to catch some exceptions.

So, for example, if we updated our sentences to this by adding a third sentence, noting that dog here is followed by an exclamation mark.

The nice thing is that the tokenizer is smart enough to spot this and not create a new token.

It's just dog.

And you can see the results here.

There's no token for dog exclamation, but there is one for dog.

And there's also a new token for the word you.

If you want to try this out for yourself, I've put the code in a colab here.

Take it for a spin and experiment.

You've now seen how words can be tokenized and the tools in TensorFlow that handle that tokenization for you.

Now that your words are represented by numbers like this, you'll next need to represent your sentences by sequences of numbers in the correct order.

You'll then have data ready for processing by a neural network to understand or maybe even generate new text.

You'll see the tools that you can use to manage this sequencing in the next episode.

So don't forget to hit that subscribe button.

✨ 推荐视频

145 句

为什么用这个视频练习口语？

这个讲解自然语言处理的视频不仅能帮你了解AI知识，更是英语口语练习的好素材。视频中 speaker 用清晰的逻辑和简单的词汇讲解“标记化”，语速适中，适合影子跟读（shadow speak）。通过模仿他的发音、语调和节奏，你能同时提升听力理解和口语流畅度。尤其是他对专业术语的解释方式，能让你学会用简单英语表达复杂概念，这对日常交流和职场表达都很有帮助。

语境中的语法与表达

视频里有不少实用表达值得关注：

“If you're not an expert on... don't worry.” 这是典型的安慰句式，在口语中用来缓解对方压力，可模仿用于日常对话。
“Consider the word... as you can see here.” 讲解时常用的引导语，能让表达更有条理，适合在分享观点或解释事物时使用。
“The tokenizer is smart enough to... ” 用“be smart enough to”强调某物的功能，比直接说“can”更生动，可用于描述工具或技术的优势。

常见发音陷阱

练习时要注意这些易出错的地方：

“tokenization” 这个专业术语的发音，重音在第二个音节，不要读成“to-KEN-i-za-tion”。可通过影子跟读反复模仿，纠正发音。
“sentiment” 中的“ti”发 /ʃ/ 音，而非 /tɪ/，容易被误读。视频中 speaker 发音清晰，可仔细听并重复。
“exclamation” 一词较长，注意每个音节的连贯性，避免吞音。通过提高英语发音的练习，可增强对长单词的掌控力。

视频中的语法

说话人最常用的结构，并附上视频中的原话：

结构	视频中的用法
被动语态 be + 过去分词 — 强调发生了什么，而不是谁做的	is called · can be represented · been given
现在完成时 have/has + 过去分词 — 过去发生但与现在仍有关联的事	has already been given · I've put · You've now seen

什么是跟读法？

跟读法 (Shadowing) 是一种有科学依据的语言学习技巧，最初开发用于专业口译员的培训，并由多语言者Alexander Arguelles博士普及。这个方法简单而强大：您在听英语母语原声的同时立即大声重复——就像是一个延迟1-2秒紧跟说话者的影子。与被动听力或语法练习不同，跟读法强迫您的大脑和口腔肌肉同时处理并模仿真实的讲话模式。研究表明它能显着提高发音准确性，语调，节奏，连读，听力理解和口语流利度——使其成为雅思口语备考和真实英语交流最有效的方法之一。

影子跟读法: 阅读完整分步指南 →

跟读练习: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - 通过视频学习英语口语

Most devs don't understand how LLM tokens work

Python3 For Pentesting - Introduction

MIT Introduction to Deep Learning | 6.S191

Why Chatbots Understand You (And Why They Don't) | NLP Explained

Tokens vs Variables in Tokens Studio for Figma | Variables Series

C1 English Listening Practice Podcast

I think AI slop is about to trigger a renaissance, actually

LangChain in 5 Minutes (Explained Clearly)

Introduction to statistical prediction models

Computational Linguistics: Crash Course Linguistics #15

Course Introduction | AZ-104 | Episode 1

We Let $200 GPT-Pro Think for 7 Days on Unsolved Math

为什么用这个视频练习口语？

语境中的语法与表达

常见发音陷阱

视频中的语法

什么是跟读法？