Pratique du Shadowing: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - Apprendre l'anglais à l'oral avec la vidéo

Hi, and welcome to this series on Zero2Hero for Natural Language Processing using TensorFlow.

If you're not an expert on AI or ML, don't worry.

We're taking the concepts of NLP and teaching them from first principles.

In this first lesson, we'll talk about how to represent words in a way that a computer can process them, with a view to later training a neural network that can understand their meaning.

This process is called tokenization.

So let's take a look.

Consider the word listen, as you can see here.

It's made up of a sequence of letters.

These letters can be represented by numbers using an encoding scheme.

A popular one called ASCII has these letters represented by these numbers.

This bunch of numbers can then represent the word listen.

But the word silent has the same letters, and thus the same numbers, just in a different order.

So it makes it hard for us to understand sentiment of a word just by the letters in it.

So it might be easier, instead of encoding letters, to encode words.

Consider the sentence, I love my dog.

So what would happen if we start encoding the words in this sentence instead of the letters in each words?

So for example, the word I could be one.

And then the sentence, I love my dog, could be one, two, three, four.

Now, if I take another sentence, for example, I love my cat, how would we encode it?

Now we see I love my has already been given one, two, three.

So all I need to do is encode a cat.

I'll give that the number five.

And now if we look at the two sentences, they are one, two, three, four and one, two, three,

five, which already show some form of similarity between them.

And it's a similarity you'd expect because they're both about loving a pet.

Given this method of encoding sentences into numbers, now let's take a look at some code to achieve this for us.

This process, as I mentioned before, is called tokenization, and there's an API for that.

We'll look at how to use it with Python.

So here's your first look at some code to tokenize these sentences.

Let's go through it line by line.

First of all, we'll need the tokenizer APIs, and we can get these from TensorFlow Keras like this.

We can represent our sentences as a Python array of strings like this.

It's simply the I love my dog and I love my cat that we saw earlier.

Now the fun begins.

I can create an instance of a tokenizer object.

The numWords parameter is the maximum number of words to keep.

So instead of, for example, just these two sentences, imagine if we had hundreds of books to tokenize.

But we just want the most frequent 100 words in all of that.

This would automatically do that for us when we do the next step.

And that's to tell the tokenizer to go through all the text and then fit itself to them like this.

The full list of words is available as the tokenizer's word index property.

So we can take a look at it like this and then simply print it out.

The result will be this dictionary showing the key being the word and the value being the token for that word.

So, for example, my has a value of three.

The tokenizer is also smart enough to catch some exceptions.

So, for example, if we updated our sentences to this by adding a third sentence, noting that dog here is followed by an exclamation mark.

The nice thing is that the tokenizer is smart enough to spot this and not create a new token.

It's just dog.

And you can see the results here.

There's no token for dog exclamation, but there is one for dog.

And there's also a new token for the word you.

If you want to try this out for yourself, I've put the code in a colab here.

Take it for a spin and experiment.

You've now seen how words can be tokenized and the tools in TensorFlow that handle that tokenization for you.

Now that your words are represented by numbers like this, you'll next need to represent your sentences by sequences of numbers in the correct order.

You'll then have data ready for processing by a neural network to understand or maybe even generate new text.

You'll see the tools that you can use to manage this sequencing in the next episode.

So don't forget to hit that subscribe button.

✨ Vidéo recommandée

145 phrases

Vocabulaire et conseils d’expression pour cette leçon

Cette vidéo contient 58 phrases et 773 mots à répéter en shadowing. La partie parlée dure 4:25. Le locuteur parle à un rythme naturel d’environ 175 mots par minute, proche d’une conversation courante. 86 % des mots font partie des 3 000 mots les plus courants en anglais ; le reste mérite d’être vu avant de commencer.

Vocabulaire clé de cette vidéo

15 mots de la vidéo à retenir, avec leur prononciation et leur sens :

Mot	Prononciation	Sens
token nom	/ˈtoʊkən/	symbole
sequence nom	/ˈsiː.kwəns/	suite, séquence
neural adjectif	/ˈnʊɹəl/	neural
similarity nom	/ˌsɪm.ɪˈlæɹ.ɪ.ti/	similarité, similitude
lesson nom	/ˈlɛs.ən/	leçon
silent adjectif	/ˈsaɪ̯lənt/	silencieux, se taire
string nom	/stɹɪŋ/	corde
experiment nom	/ɪkˈspɛɹ.ə.mənt/	expérience
spin verbe	/spɪn/	tournoyer, pivoter rapidement
exception nom	/əkˈsɛpʃən/	exception
principle nom	/ˈpɹɪn.sɪ.pəl/	principe
frequent adjectif	/ˈfɹiː.kwənt/	fréquent
generate verbe	/ˈd͡ʒɛn.ə.ɹeɪt/	générer
array nom	/əˈɹeɪ/	éventail
sentiment nom	/ˈsɛn.tɪ.mənt/	sentiment

Les verbes à particule que vous entendrez

Mot	Sens
go through verbe	traverser

La grammaire de cette vidéo

Les structures que le locuteur utilise le plus, avec les mots exacts de la vidéo :

Structure	Dans la vidéo
Voix passive be + participe passé — l’accent est mis sur ce qui arrive, pas sur qui le fait	is called · can be represented · been given
Present perfect have/has + participe passé — une action passée qui compte encore maintenant	has already been given · I've put · You've now seen

Prononciation à surveiller

Le locuteur utilise 15 contractions et formes réduites, comme we'll, you'll, don't. Prononcez-les sous leur forme courte, telles que vous les entendez.

Les sons « sh » et « zh »: exception /əkˈsɛpʃən/, dictionary /ˈdɪk.ʃəˌnɛ.ɹi/, exclamation /ˌɛkskləˈmeɪʃn̩/
Mots longs — placez bien l’accent: similarity /ˌsɪm.ɪˈlæɹ.ɪ.ti/, experiment /ɪkˈspɛɹ.ə.mənt/, dictionary /ˈdɪk.ʃəˌnɛ.ɹi/, parameter /pəˈɹæm.ə.tɚ/

Comment s’entraîner avec cette vidéo

Écoutez la vidéo en entier une fois sans parler et notez les mots que vous ne connaissez pas.
Commencez à la vitesse 0,75×, répétez phrase par phrase, puis revenez à la vitesse normale quand cela devient facile.
Enregistrez-vous et comparez avec l’original, en faisant attention à des mots comme token, sequence, neural.

Qu'est-ce que la technique du Shadowing ?

Le Shadowing est une technique d'apprentissage des langues fondée sur la science, développée à l'origine pour la formation des interprètes professionnels. Le principe est simple mais puissant : vous écoutez de l'anglais natif et le répétez immédiatement à voix haute — comme une ombre suivant le locuteur avec un décalage de 1 à 2 secondes. Les recherches montrent une amélioration significative de la précision de la prononciation, de l'intonation, du rythme, des liaisons, de la compréhension orale et de la fluidité.

Technique du shadowing : lire le guide complet étape par étape →

Pratique du Shadowing: Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1) - Apprendre l'anglais à l'oral avec la vidéo

Most devs don't understand how LLM tokens work

Python3 For Pentesting - Introduction

MIT Introduction to Deep Learning | 6.S191

Why Chatbots Understand You (And Why They Don't) | NLP Explained

Tokens vs Variables in Tokens Studio for Figma | Variables Series

C1 English Listening Practice Podcast

I think AI slop is about to trigger a renaissance, actually

LangChain in 5 Minutes (Explained Clearly)

Introduction to statistical prediction models

Computational Linguistics: Crash Course Linguistics #15

Dear Higgsfield, I reverse engineered you in 12 hours

Course Introduction | AZ-104 | Episode 1