Everybody hope everybody is being safe, smart and socially distant.

I want to today to draw your attention to the following cattle challenge, obviously highly relevant to today.

But also I find this to appear to be much more like what the the types of tasks, at least for me, that I have found businesses to need when people come to me for, like contracting and consulting, the data tends to look more like this than your average cattle competition.

Now nothing against karaoke competitions.

There's definitely true competition there.

Ah, but generally cattle comes down to, um, a competition of optimization rather than a competition of finding insights in highly highly unstructured data.

Now, sometimes that's wrong.

But for the most part, that's kind of what you find on casual, because cattle has to be objective in enforcing that objective objectivity.

There's people that will optimize for those objective results.

So anyways, enough on that, this is very interesting data set.

I've downloaded it and kind of peeked at it on.

That's kind of why I decided not.

This would be really good to do a video on it just barely peaked at the data and then we're gonna dive in by a code and you will see kind of firsthand the process that I at least begin with looking through a data set.

This kind of data set, though I mean, you will spend, you know, 22 hundreds of hours, you know, going through here, trying to get insight.

So probably here at best, will see, like the first hour.

So don't expect anything too crazy, but this is just kind of my process at least.

So Thio get the data, Just go ahead, make an account.

You could download the data.

Also, they have various tasks here.

We'll talk about this in a little bit, but as you've likely found, like the problem here is, there's so much information like if we look at data says, there's 29,000 articles and then 13,000 are full text and these air scholarly articles Justo, make that clear, which is important.

And there's so much information in so much new information that's just being pushed out very fast.

There's historical information and the new information, and it's very difficult.

So as you've probably found yourself like trying to learn about this is the variability in the information and fax that you might hear is really high.

And then not only that, the variability, but there's also just there's so many little details that just really difficult.

And so, in this case with the, um, you know, there's the critical in acute nature of what we're going through here.

It's making it very hard.

Normally, it really would be like grad students that would sit and go through all this stuff.

We just do not have time for that, you know?

And then businesses, for example, like they it's either you or they're gonna hire some interns or something to do to try to answer these questions.

But it turns out, programming can often, uh, do a better job at it.

So anyway, uh, let's dig in.

We'll talk about these tasks in a moment, but first start to download of data, and then we're going to start going through, um, the actual data.

Once it's downloaded, you can either pause or just Lazar download is going.

Let's look at my my file so you'll get it.

Uh, you know, a zip, you'll unzip it, and then you'll get this basically.

So the last update was March 13th.

It is March 19th at the time of my recording.

This but eh, so hopefully at some point, maybe they'll make an update.

But we'll see.

Um, so coming in here, this is what we get.

We get these four directories initially, I kind of expected this to be the full thing, right?

And then this was like, the commercial subset, noncommercial subset and then some sort of custom license subset.

But then I click on here.

I see 803 items on, but I click on this one and I see 9000 items.

This one has almost 2000 items on the custom license has 1400.

So that tells me right away.

No, we were gonna have to go through all of these directories and then inside the directory, we have another directory.

First of all, just keep this in mind with the exact same name as the parent directory.

So who knows what that's going on?

But then we have these Jason files.

So now let's just look at the jays on file, so we know what we're looking at.

So already the Jason I'm like, Well, that's not what I expected there's got to be in Jason.

I mean, maybe it's coming from sort of Jake some sort of Jason based database.

I don't know anyway, s So what we have here is clearly just your typical, you know, keys and values scrolling down.

Okay, We've the first, like, body of text that we found at all.

It's called abstract, so we can grab the abstract.

Okay.

Noted, uh, then we truly have body text, and then we can already see quite quickly.

It's in chunks of text.

So you've got body text and then text, and then maybe some information.

In this case, there's really no information in their sights, bands, reference bands and section, I guess I don't know.

Um, so yeah, clearly we get chunks of text, and then here we definitely do have sight span stuff.

I'm not really sure what this is.

This is that, like, you know, character placement.

I've no idea.

No clue, but clearly, we're looking for the chunks of text inside of body text.

Okay.

Okay.

So I think we have a general idea.

Now that's about as much as I want to pour over the actual Jason files.

I think we're ready to begin coding.

So if you haven't fully downloaded the data yet, now would be a good time to pause.

Um, so the first thing I'm gonna do is, uh, make a file.

So, you know, rod up high, and we'll just a start, and, um, you go open with sublime.

I need to set Sublime is my default later.

So, uh, right away.

Let's just start.

It will take time to go through like each of these directories.

So let's just start with this top directory, so I'm just gonna grab the name for now.

Later.

We'll go through all four, probably so we can get all of the data possible.

But we're gonna say ders equals and we'll just make this a list.

There we go.

And then immediately, what we want to do is for d endures.

Um, we want to.

Let's go ahead import s way.

Let's just make sure we get all the files.

So the first thing we wanna do is like, you know your way through all the files.

So Ferdie Anders and then basically way want to do for file in os dot list dir.

And when you use that strings here, apparently because, um, you know, some people have said that it won't like for its last one work on windows, But from my history of working on windows for many years, I have found that even if strings on windows did work with a forward slash So someone police feel free to correct me if I'm wrong.

Like if you're on Windows and that's not working.

And you're on, like, python 37 or whatever.

Um, yeah, I guess.

Let me know.

But it should work.

Oh.

Oh, that's not getting a boon to is having a problem.

Uh, hopefully we're still recording.

Looks good.

We'll find out.

Cool.

So we know that it was the directory slash directory.

So then we just want to go through the files.

So the first thing I'm gonna dio is Let's just go to print file and let's just see.

I also don't have everything correctly set up yet on this machine is a relatively new machine for me, so I'm gonna run everything from the terminal.

We should make sure python gets a python for me.

Points to 3.7 Pipe on file.

Cool.

Look at all those.

Okay.

So we are pointing were able to find our Jason files.

Very good.

The next order of business is to open one.

Um, So let's go ahead and in court, Jase.

And what we want to do is a final path.

Will be, um I guess it would just be this We're here, so just copy.

Paste slash, Um, file cool s.

So then what would say j equals jayson dot loaded.

And we wanna load open file path with the intention to or a B.

Cool.

Now, let's prove J let us issue a break so we don't get too crazy, But we just want to make sure things were working as expected up to this point.

And it they are.

So, uh, so we got Jason.

Now, let's do a quick for key in J uh, Frankie, let's just get a general idea of the keys in this documents.

So we got paper I d meta data, abstract body text, and so on.

So let us now, uh, what's going on in print?

A couple of things.

Let's prance.

Uh, j meta data.

We already kind of I think I know what body text is already gonna look like that kind of looks pretty clear to me.

Metadata appears to be quite a bit of things, so you'll get the title.

This apparently has no title authors.

Um, gosh, I almost want Okay, let's do four K in J metadata.

It's almost would be nicer.

And a notebook format.

I didn't really plan for that.

Uh, okay, so really, just title authors really title, But that looks that's a lot of stuff for there to just be titling authors, but Okay, sure.

Okay, so we're gonna say is title equals J meta data title on then?

We also have abstract was equal to J abstract.

Uh, and then the next thing that we really want to do is, um, let's go before, let's print abstract real quick print abstract.

I need to leave this up.

I'm gonna close these two because that's gonna be annoying to keep doing that every time.

Oh, interesting.

So apt.

Abstract appears to be in a list format, which is odd.

I'm not really sure what to think about abstract being a list, but would there ever be too abs?

I mean, I'm not a professional lent papers, but, uh um, having written all of zero papers in my life.

But even in college, I'm pretty sure there was only one abstract, but okay, so, interestingly, abstract zero.

So the abstract needs to be abstract zero, I guess, Um, let's stop breaking and stop printing and let's run through everything to just see No.

Okay, so sometimes we don't have, um, Nash.

Rex.

Let's try abstract.

Except we're just trying.

I'm trying to see.

Is there ever a time where it's not a list?

Because it does feel weird, But shortly like my expectations.

This came from a Jason based database.

So my expectation is that everything will have the same format, but I could be totally wrong.

J abstract.

So my thought here is that this will be an empty list when we hit that exception.

Um, that's the only thing we're gonna print.

So hopefully we'll see a bunch of empty lists.

Yeah.

Cool.

Okay, So accept.

So in the case that there isn't an abstraction, We're just gonna say it are abstract.

Where's the abstract?

Is empty.

Just empty strings.

Okay, so all right.

So we've got the abstract, the title, and now we need to grab our body text.

Um So we're going to say full text equals and empty string for now.

And then four text in.

Uh, would it J uh body text, body text Four techs in J text.

Let's continue our break, because now we're going to just just try to see you, like, are we getting what we expected here?

Print for text.

So we need to say text text?

I guess so.

We call text protects and J by text text texting cake trip up.

Shit.

What happened there?

I said shoot, By the way.

Um okay, Yeah, that looks more like what we expect.

So, um text.

So now what we want to do is we're gonna upend this toe Full text, just wants a full text plus equals text text, plus new line.

New line.

Beautiful.

At the very end.

Let's print full text and let's stop ringing there and let me see now.

Well, we look at okay.

Yeah.

Okay.

So now we Finally after what?

Has it been 13 minutes?

14 minutes.

We've got the data in the format that we expected to get the data.

But again, this is more likely to be the case of how even text data if you're going to get it from like, for example, I've had, um, even retailers and then certain news companies and try to think of what else?

I've seen something similar to this, but, uh, it is data typically in a format that you did not expect.

It's either CSB and or Jason and not just like text documents.