跟讀練習: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - 通過影片學習英語口語

選擇練習模式

每種模式都會開啟全屏練習室。

58 句

如果句子過短或過長，請點選 Edit 進行調整。

Hi, and welcome to this series on Zero2Hero for Natural Language Processing using TensorFlow.

If you're not an expert on AI or ML, don't worry.

We're taking the concepts of NLP and teaching them from first principles.

In this first lesson, we'll talk about how to represent words in a way that a computer can process them, with a view to later training a neural network that can understand their meaning.

This process is called tokenization.

So let's take a look.

Consider the word listen, as you can see here.

It's made up of a sequence of letters.

These letters can be represented by numbers using an encoding scheme.

A popular one called ASCII has these letters represented by these numbers.

This bunch of numbers can then represent the word listen.

But the word silent has the same letters, and thus the same numbers, just in a different order.

So it makes it hard for us to understand sentiment of a word just by the letters in it.

So it might be easier, instead of encoding letters, to encode words.

Consider the sentence, I love my dog.

So what would happen if we start encoding the words in this sentence instead of the letters in each words?

So for example, the word I could be one.

And then the sentence, I love my dog, could be one, two, three, four.

Now, if I take another sentence, for example, I love my cat, how would we encode it?

Now we see I love my has already been given one, two, three.

So all I need to do is encode a cat.

I'll give that the number five.

And now if we look at the two sentences, they are one, two, three, four and one, two, three,

five, which already show some form of similarity between them.

And it's a similarity you'd expect because they're both about loving a pet.

Given this method of encoding sentences into numbers, now let's take a look at some code to achieve this for us.

This process, as I mentioned before, is called tokenization, and there's an API for that.

We'll look at how to use it with Python.

So here's your first look at some code to tokenize these sentences.

Let's go through it line by line.

First of all, we'll need the tokenizer APIs, and we can get these from TensorFlow Keras like this.

We can represent our sentences as a Python array of strings like this.

It's simply the I love my dog and I love my cat that we saw earlier.

Now the fun begins.

I can create an instance of a tokenizer object.

The numWords parameter is the maximum number of words to keep.

So instead of, for example, just these two sentences, imagine if we had hundreds of books to tokenize.

But we just want the most frequent 100 words in all of that.

This would automatically do that for us when we do the next step.

And that's to tell the tokenizer to go through all the text and then fit itself to them like this.

The full list of words is available as the tokenizer's word index property.

So we can take a look at it like this and then simply print it out.

The result will be this dictionary showing the key being the word and the value being the token for that word.

So, for example, my has a value of three.

The tokenizer is also smart enough to catch some exceptions.

So, for example, if we updated our sentences to this by adding a third sentence, noting that dog here is followed by an exclamation mark.

The nice thing is that the tokenizer is smart enough to spot this and not create a new token.

It's just dog.

And you can see the results here.

There's no token for dog exclamation, but there is one for dog.

And there's also a new token for the word you.

If you want to try this out for yourself, I've put the code in a colab here.

Take it for a spin and experiment.

You've now seen how words can be tokenized and the tools in TensorFlow that handle that tokenization for you.

Now that your words are represented by numbers like this, you'll next need to represent your sentences by sequences of numbers in the correct order.

You'll then have data ready for processing by a neural network to understand or maybe even generate new text.

You'll see the tools that you can use to manage this sequencing in the next episode.

So don't forget to hit that subscribe button.

下載應用

流利口語所需的一切

AI 發音評分逐句打分

音標練習掌握每個音

詞彙學習建立你的詞庫

詞彙遊戲邊玩邊學

掃碼

Download on theApp Store Get it onGoogle Play

FOR YOU

為什麼用這個影片練習口語？

這個講解自然語言處理的影片不僅能幫你瞭解AI知識，更是英語口語練習的好素材。影片中 speaker 用清晰的邏輯和簡單的詞彙講解“標記化”，語速適中，適合影子跟讀（shadow speak）。通過模仿他的發音、語調和節奏，你能同時提升聽力理解和口語流暢度。尤其是他對專業術語的解釋方式，能讓你學會用簡單英語表達複雜概念，這對日常交流和職場表達都很有幫助。

語境中的語法與表達

影片裡有不少實用表達值得關注：

“If you're not an expert on... don't worry.” 這是典型的安慰句式，在口語中用來緩解對方壓力，可模仿用於日常對話。
“Consider the word... as you can see here.” 講解時常用的引導語，能讓表達更有條理，適合在分享觀點或解釋事物時使用。
“The tokenizer is smart enough to... ” 用“be smart enough to”強調某物的功能，比直接說“can”更生動，可用於描述工具或技術的優勢。

常見發音陷阱

練習時要注意這些易出錯的地方：

“tokenization” 這個專業術語的發音，重音在第二個音節，不要讀成“to-KEN-i-za-tion”。可通過影子跟讀反覆模仿，糾正發音。
“sentiment” 中的“ti”發 /ʃ/ 音，而非 /tɪ/，容易被誤讀。影片中 speaker 發音清晰，可仔細聽並重復。
“exclamation” 一詞較長，注意每個音節的連貫性，避免吞音。通過提高英語發音的練習，可增強對長單詞的掌控力。

影片中的語法

說話人最常用的結構，並附上影片中的原話：

結構	影片中的用法
被動語態 be + 過去分詞 — 強調發生了什麼，而不是誰做的	is called · can be represented · been given
現在完成時 have/has + 過去分詞 — 過去發生但與現在仍有關聯的事	has already been given · I've put · You've now seen

什麼是跟讀法？

跟讀法 (Shadowing) 是一種有科學依據的語言學習技巧，最初開發用於專業口譯員的培訓，並由多語言者Alexander Arguelles博士普及。這個方法簡單而強大：您在聽英語母語原聲的同時立即大聲重複——就像是一個延遲1-2秒緊跟說話者的影子。與被動聽力或語法練習不同，跟讀法強迫您的大腦和口腔肌肉同時處理並模模擬實的講話模式。研究表明它能顯著提高發音準確性，語調，節奏，連讀，聽力理解和口語流利度——使其成為雅思口語備考和真實英語交流最有效的方法之一。

影子跟讀法: 閱讀完整分步指南 →

跟讀練習: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - 通過影片學習英語口語

選擇練習模式

下載應用

推薦影片

#1 Machine Learning Specialization [Course 1, Week 1, Lesson 1]

Why AI Tokens are so Expensive - Computerphile

Python3 For Pentesting - Introduction

English Podcast 1: Learning Languages

Dear Higgsfield, I reverse engineered you in 12 hours

Video

Give me 31 minutes, I will make PyTorch click forever

Master AI in 60 Minutes with Vector Databases!

LangChain in 5 Minutes (Explained Clearly)

Translating Claude’s thoughts into language

Mixture of Experts Explained Visually: How Trillion-Parameter Models Actually Work

Crash Course Linguistics Preview

為什麼用這個影片練習口語？

語境中的語法與表達

常見發音陷阱

影片中的語法

什麼是跟讀法？