Name: NMTの概念とパラメータ - ディープラーニング、Python、TensorFlowを使ったチャットボットの作成 p.8 (NMT Concepts and Parameters - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.8)
Uploaded: 2021-01-14T12:27:52.000Z
Duration: 28 min 17 s
Description: VoiceTubeの動画で発音を聞きながら英語表現を覚えよう！学べる英語：

Everybody, We're gonna part eight of our check pot with python in tensorflow tutorial.

What I'd like to do is talk about some of the more high level concepts and parameters of our chat bond with the neural machine translation code that we're using.

And I hope thio least give you an idea of a better idea of what's actually going on here because there's actually a lot going on here.

My initial intention was to start with the basic secrets sequence, the old English to French translation model that was done with tensorflow.

But tensorflow deprecate id it, and you really can't run it without running the tensorflow one point.

Oh, it leaves the time on my filming this we're already at 1.4.

And so I just was like, Now I'll just start with with the current anti model, but it's gonna be a lot of information that I'm about to throw at you.

So So, first of all, any time you've got a translation, whether it's, you know, neural machine translation translation from one language to another language or in our case, even though it's English, English, it's still a form of translation that's gonna be going on there.

It's just input to an output of language.

Any time you have that, obviously words are not numbers.

So the first thing that we're gonna we need to do is we need to token eyes, inputs and an easy way you could Token eyes inputs is just a split by space and probably punctuation.

There's a special token for I a token for a space.

And generally that's what the encoder is gonna do.

It's just gonna be token izing your input.

And then once it's token ized Thea other thing an encoder is likely to do is take those tokens.

And one thing we could do is just a sign it like an arbitrary I d.

We'd like those ideas to be somewhat meaningful, So we generally are gonna create a word vector that is gonna put similar words and give them similar I.

And this is gonna help both in actually translating.

But also, it's gonna help us in evaluating how good our translations were because a lot of times you might be really close.

Um, you say you've got airplane in Japanese and you translate it Thio car.

You translated to car versus airplane and Japanese and you translated Thio, Um, I don't know, shoe or cat or something like that, right?

At least car is closer to plane than shoe or cat is.

So we would know that as we get closer, that correct translation we want to reward for that we don't wanna have necessarily equally wrong just because we missed the word right.

So generally we're gonna have word vectors as well.

Once you have those ideas, though, then you're gonna feed it through some sort of a neural network with language information.

This is generally a recurrent neural network that just makes sense.

So we have that sort of non static temporal sense going forward.

Um, and then generally, it's, ah of a recurring role network re usually using long short term memory.

Then once you've done that, you feed it again through a decoder, and then you've got your output.

And that's your basic sequences sequence on language data Translation model.

Now, let's start talking about some of the problems that we might have while doing that.

So, first of all your input and your output, first of all, these don't even match right.

A student is four tokens, not including spaces.

And then the output is three tokens again, not including spaces.

Is your input always gonna be those four tokens or whatever many tokens you you're gonna have?

Like initially that we saw for that was with padding.

So we might say, Okay, what's our longest sense we've ever had?

Well, what we're gonna say is the input layer is always gonna be 100 tokens or 100 nodes and each nodes value.

And then let's say we only have five tokens.

Well, we put in those 1st 5 and then we have a special token called the Pad token, and then we're just gonna use that pad took on every single node after.

So we just do a bunch of padding, and that's one way we can solve it.

But of course, that's not a good idea, most likely, and it doesn't train well and it doesn't perform well because what ends up happening is because we use padding so much longer your sentence, the less impact those later words are gonna wind up having because the neuron hours gonna learn like Okay, these pads don't mean anything.

And then we get a long sentence once in a while in those last few words, just don't mean anything.

So then again, still on that first sequence sequence, translation.

And again, that was also this is Charles.

Like if you guys have been following either the twitch stream or you follow Charles the Aye Aye, on Twitter.

Um, that was This is what he has been running.

So he's been on that kind of V one initial sequence of sequence model from TENSORFLOW until very recently until this serious started.

Basically, um, So So the other idea was that we could use bucket ing so that the idea here was like, Okay, well, what we could do is we could have input layers that are buckets so we'll take our tokens, Won't have, like, four buckets will have a bucket for the stuff that's like 5 to 10 long.

We'll have a bucket for the stuff that's 11 toe, 25 long a bucket for 25 35 then a bucket for 35 50 or something like that we would do bucket ing.

And then whichever bucket held the longest version of that string, that's the bucket that we would use.

And we were trained, and this did okay, you know, it did all right, but not ideal, because we still are going to be using padding.

It's still so that brings us to today with Tensorflow.

We have what are called dynamic, recurrent neural networks, and we can have a dynamic input coming in now, moving back in time a little bit and also getting into now again.

So what kind of talk about up to this point Right before dynamic was that first secrets of sequence stuff from tensorflow.

Um, And now what I'm gonna be talking about is actually the empty the current age and empty that were using.

So another problem that we have besides the bucket ing, which is solved.

But besides the budgeting and padding, which is basically solved by dynamic, recurrent all networks, we have the issue of just language in general.

So, for example, let's let's consider a translation chap up a translation task like English to French.

In general, if you want to translate English or French or English or German or English or Spanish, there's there might be slight variance in the syntax of like, a noun phrase.

But in general there's a algorithmic solving of the translation of English thio, French, German, Spanish, whatever.

But then you have a language like Japanese, so English and Japanese are very, very different languages, and they don't follow even remotely similar rules and in Japanese, Sometimes that last character changes the meaning of all the other characters.

And things were just totally different, right?

And then also with the chat bought, the same thing is true.

And in a lot of translations, same thing is true that it's like with an L S t m generally and l s t m is really pretty good at remembering in a sequence about, like, 10 to maybe 20 tokens.

So in our case, we are token izing toe word.

NMTの概念とパラメータ - ディープラーニング、Python、TensorFlowを使ったチャットボットの作成 p.8 (NMT Concepts and Parameters - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.8)

字幕リスト動画再生

NMTの概念とパラメータ - ディープラーニング、Python、TensorFlowを使ったチャットボットの作成 p.8 (NMT Concepts and Parameters - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.8)

stuff

basically

negative

character