Name: すべての変異体を殺せ!(突然変異検査入門) - デイブ・アロンソン｜JSConf Hawaii 2020 (Kill All Mutants! (Introduction to Mutation Testing) - Dave Aronson | JSConf Hawaii 2020)
Uploaded: 2021-01-14T10:22:21.000Z
Duration: 22 min 18 s
Description: VoiceTubeの動画で発音を聞きながら英語表現を覚えよう！学べる英語：

I'm Dave Aronson, the T rex of Code O Saurus LLC, And I flew over here on my pterodactyl that teach you howto kill mutants.

That of software development, not comic books.

There's something out of mutation testing.

So what on infinite Earths is mutation testing?

You might look at the name and think it's a way to test the mutations used in genetic algorithms or something.

But no, it's actually way to test both our code and our unit test suite by using mutations.

It's most unusual benefit is to help ensure that our tests are strict by finding the gaps that let our code get away with unintended behavior.

Lack of this strictness usually comes from lack of tests, of course, or poorly written or tests that were poorly maintained and just didn't keep pace with changes in the code.

Speaking of the code, mutation testing can also help insure that our code is meaningful, by which I mean that any tiny little change to it have a noticeable effect on its behavior.

Lack of meaning usually comes from code being redundant or unreachable or otherwise not.

Mutation testing puts these two together by checking first, that some tiny little change did indeed have a noticeable effect.

And then, at our unit test sweet spots that change and fails.

Not all the tests have to fail, but each change should make at least one unit test fail.

But as Fred Brookes told us back in 86 multi reminded us the other day, there's no silver bullet besides which there for killing where wolves, not mutants.

The first drawback is that it's rather hard labor on the CPU and therefore usually rather slow.

We don't want a mutation test our entire code based on every save, maybe over a lunch break for a small system or overnight or over a weekend for a larger one.

Fortunately, most of the tools do include an incremental mode, so we can test just what we changed since last time that maybe we can do on every safe if we save early save often like we know we're supposed to.

Another drawback is that it's not at all the beginner friendly technique.

It tells us that some particular change to the code made no difference to our test results.

You'll find that interpreting the results of mutation testing involves a lot of asking.

It takes a lot of interpretation to figure out what a mutant is trying to tell us.

They're almost as incoherent a zombies, but with a much larger vocabulary.

They're usually trying to tell us that our code is meaningless or our tests are lax or, of course, both.

But it can take a lot of time and effort to figure out exactly how and sometimes it's a false alarm.

For instance, the mutant might not have made any test fail, but it might not have made any actual difference in the codes behavior, either, at least not in anything we actually care about.

But what does mutation testing actually do?

Hence the name copies of our code to create test failures, otherwise known as faults.

So mutation testing is a fault based software testing technique, and this means it's sort of related to something you might already be familiar with chaos.

Just like chaos, Monkey uses faults to help Netflix discover flaws in their ever recovery process.

Mutation testing uses false who help us discover flaws in our code and our unit test suite.

But the way this works is kind of upside down from what chaos monkey does.

Chaos Monkey is best known for injecting faults into Netflix's production network.

If the customers don't notice and the metrics still look fine, then Netflix knows that there ever recovery is working great.

But mutation testing injects instead not faults but changes.

It doesn't know whether these changes will produce.

But that's actually up to the unit test suite.

It injects them not into our actual network but copies of our code.

And it does this in our best environment, not production.

Oh, and if everything's still passes, that doesn't mean everything is okay.

Remember, each change should make a least one unit test fail.

Let's peel back one layer of the onion and look at it from a high level view first our chosen tool breaks our code apart into pieces that test.

Usually these are going to be our functions for each function.

It then tries to find the tests because if there aren't any tests, well would be kind of redundant, too impossible to run its tests.

Assuming it finds any, then the tool makes mutants out of that particular function.

And to do that, it looks closely at the function to see how it can be changed and for each way that function can be changed.

The tool makes one mutant with just that one change in it after the tools done creating all the mutants it can from a given function and it it a rates over the list.

And now we get to the heart of the matter for each mutant made from a given function, the tool will run the unit tests from the original function, but using a mutant instead.

And if one of those unit tests fails, this is called killing the mutant quick side note.

Some people object to the violent nature of this metaphor, especially since in the comic books, mutants are often used as a metaphor for marginalized people.

I'm trying to come up with a better term, like maybe cover or rescue or some such.

I couldn't come up with something good in time to change this talk well anyway.

Killing the mutant means that the tiny little change that the tool made in order to create the mutant it indeed have a noticeable effect on the behavior.

And our test suite was strict enough to notice that change and fail.

At least one test after one test has failed.

The tool will stop running a test against that mutant and move on to the next one.

We don't care how many Maur unit tests that mutant might make.

Like so much else in computer science, we only care about ones and zeros.

But if a mutant let's all the functions unit tests pass, then it has the superpower of mimic re skilled enough to fool our tests.

This usually means that our code is meaningless or our tests are lax or both, and now it's up to us to figure out exactly how.

Now let's peel back another layer of the onion and look at some technical details of how this works.

First, our tool parses our code usually into an abstract syntax tree or a S T.

I think those were mentioned on Wednesday.

I know these boxes are a little small to read, but we don't need to actually understand this.

A s t in detail after our tool makes an a S t out of our code and it traverse.

Is that tree looking for sub trees that represent our functions after finding them?

It handles them pretty much like I described earlier, starting with looking for the tests.

This usually relies on us developers either annotating our code or following some kind of naming convention.

This is sometimes supplemented or even replaced by the tool.

Looking at what functions each of the tests call next.

The tool makes the mutants and to make them from an A s T sub tree it then traverse is the sub tree, just like it did to the whole thing.

But this time it's not looking for even smaller sub trees, twigs or something to extract, but nodes where it can change.

Something may be substituted different kind of node or whatever.

For instance, suppose our tool was traversing the S T.

I showed earlier, and it's gotten down to this not equals comparison, following those red lines each way it can change that.

Knowed it will make a fresh copy of that entire sub tree with just that.

No changed in that one way after it's done, making as many copies, which are the mutants.

It'll continue traversing this sub tree and do likewise to the rest of the nodes.

I've been talking about it making changes.

So what kind of changes am I talking about?

It can change a mathematical, logical orbit wise operator from one to another.

In cases where you can do one of the totally different category, it can even do that.