Name: データベースからトレーニングデータへ-ディープラーニング、Python、TensorFlowでチャットボットを作る p.6 (Database to Training Data - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.6)
Uploaded: 2021-01-14T12:27:46.000Z
Duration: 12 min 29 s
Description: VoiceTubeの動画で発音を聞きながら英語表現を覚えよう！学べる英語：

Well, everybody And welcome to part six of the Champ box of groceries with python intense or flow.

過去形

In the last tutorial, we built our database, we inserted all the roads and at this point, I'm assuming you guys have created a relatively large database of pears on.

to 不定詞

And then in this tutorial, what I'm gonna be showing you guys is how you can create training data from that database when it's done, if you don't have, if you have less than, like 100,000 pairs, I wouldn't suggest that you continue falling along unless you're just kind of curious to see how things work.

So with that, let's go out and get started.

So what we want to do here is basically to create the actual training data that we're gonna use for our models we're gonna be using.

We're kind of poke around with a few models in the serious, but pretty much it's always gonna be the same kind of format.

And the idea is that generally, what you're gonna have is a, um a from file to A to file.

So again, this is all kind of this basically a running man.

Sure, if this is again, but basically what we're doing is tensorflow sequence to sequence.

OK, so whether it's a chap pot, which is a comment in a reply for it, it's a language translation, which is what a lot of the sequence sequence tutorials are doing.

Everything in life pretty much boils down to a sequence to a sequence.

It's not really so much a ah fixed input to a fixed output.

It's variable length input, variable length output.

And that's what's really intrigued me about sequence to sequence and especially some of the later implementations of sequences sequence from tensorflow.

That's what's pretty exciting about it, anyways, back to Planet Earth.

What we want to do is create basically a parent comment file and then and then a reply file where each row or each line number corresponds to the other file.

Okay, so Line 15 and the parent would be the initial comment and then line 15 in the sewer line.

15 In the from file is the parent comment and then line 15 in the to file.

Is the child the reply to that parent kind?

Okay, so in order to do this we're blinded.

Uh, then we're gonna have time frames and basically built this with it in mind that I might have many different databases with different times 2055.

But really, I think that, you know, you might you might you combine them most likely, if that's what you're gonna do.

And then there's a four time frame in time.

So what we're gonna do is we're gonna build this connection Time time frames were going to do is build this connection and then use read SQL from pandas to read it.

No, you actually don't need to use Pan is to do this.

I'm just gonna use pandas because there might be times when I want to add a little bit more functionality, a little bit more logic Thio to the SQL kind of pull here and or even just data manipulation or whatever.

I'm for that reason I'm using pandas here, but for what we're gonna do in this tutorial, Siri's at least for what I know, I have planned out.

Uh, I guess you wouldn't mean to use band aids.

So anyway, uh, connection will be sq light three dot con act, and we will connect to that date of it, So dot TV Why did it again?

The connection dot curse or, um And then what we're gonna say here is first, let's have a limit equals 5000 Will say last units equal zero.

Her length equals limit counter equal zero and test done.

So limit will be how much we're gonna pull at a time to throw into our pandas data friend.

Last UNIX will help us to basically buffer through our database.

So we'll pull Will grab the last UNIX time stamp of that pole.

And then well, and then from there, we know.

Okay, in our next pole, that's a UNIX must be greater than last UNIX.

And so we just keep doing that with each pole has a limit of whatever this number is.

Eventually, we could raise that most I want 5000 because test is not done yet.

Yeah, you're gonna have a two or a from and then a to file.

But you also wanna have testing files, something out of sample, just to see how the models doing.

So we're going to use a test file, and that test file will be the first 5000 rows of data.

You could make this anything you could do.

But, yeah, you can You can do something else if you want.

So then we're gonna go ahead and do is we're gonna ask the question.

While cur length is equal to basically whatever the limit is, that means we were able to make a poll that that completely exhausted whatever a limit was.

So chances are, either there's zero Rose left, but we'll find that out a moment or they're still rose left.

So as long as we're able to get our limits worth from the database, we probably have more poles to make, So we'll keep making pools.

So then what we're gonna say is DEA for data frame eagles, pandas A p d dot reed s Q l And what we're gonna read boat, we'd ask you.

What we're going to say is, um, the SQL statement.

We'll just do all from parent and reply where UNIX is greater than something.

Um, And on this should be all caps and parent, not Knoll.

And school is greater than zero, but it sure as heck better be order.

And basically, what we need to do is, uh, unit seems to be greater than last UNIX.

Yeah, I guess that's the only for we just did UNIX and then the limit.

And then finally the other thing when you d'oh p d don't read SQL you pass first the SQL statement and then you pass the connection.

We're gonna say last underscore UNIX equals D f dot tale one.

So the last thing UNIX you next dot values zero with Boom.

So now we've updated that last UNIX her length.

Let's see, what's the length of the data from?

Now we're gonna ask, um, if not test done.

We're gonna with open, uh, Louise is called his test up from with the attention to upend.

And we're to specify the encoding as utf eight.

Um, as f what we want to do is four content in D f.

データベースからトレーニングデータへ-ディープラーニング、Python、TensorFlowでチャットボットを作る p.6 (Database to Training Data - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.6)

字幕リスト動画再生

データベースからトレーニングデータへ-ディープラーニング、Python、TensorFlowでチャットボットを作る p.6 (Database to Training Data - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.6)

assume

content

basically

priority