and Jason Arbon October 27, 2011

>>James Whittaker: I actually think we have two speakers for this next one, a pair of

Googlers. Ibrahim El Far and Jason Arbon actually both

work for me. There were no special privileges getting them in this event. They were all

still voted for. But these are the two people that if they do have a bad presentation, I

can actually do something about it. So just saying. No pressure. We're continuing our

theme of browser automation. And I think this is going to be completely different from a

lot of the other discussions. But I heard the sauce guy, Adam, say something about BITE.

That's part of what we're going to demo here. And maybe Sauce will pick it up and do something

with it, because we have open sourced this. So I think Ibrahim is going to start.

Ibrahim actually kind of followed me. We were in Florida for a little while. And then we

both went to Microsoft. And then -- >>Ibrahim El Far: You followed me into Microsoft.

>>James Whittaker: I followed you. That's true. You went to Microsoft first and then

I followed you. And then I went to Google and you followed me.

And eventually we're going to have to stop this stuff. People will talk.

[ Laughter ] >>James Whittaker: And Jason Arbon is also

another former Microsofty-become-Googler. And, I don't know, maybe I should stop saying

that. I said former Microsofter. When your badge

-- the joke around Google is you don't know you're fired until you come to work and your

badge doesn't work. That's when you know. Okay. Let's welcome our new speakers.

[ Applause ] >>Jason Arbon: He's got -- Welcome to the

cloud; right? >>Ibrahim El Far: Excellent. Fantastic.

So hi. My name is Ibrahim. I've been at Google for about a year and a half. I'm a manager

of a team of about -- until recently, of seven or eight people who basically just do test

tools. We're going to talk about two of these tools today.

And I work closely with Jason. Jason is the visionary and the man behind most of these

tools. So, basically, if you dislike any of the ideas,

just blame him -- >>Jason Arbon: In implementation, it's him.

>>Ibrahim El Far: Yes, indeed. So I figured, though, I will want to just

start with a silent presentation reflecting on some of the things that we saw yesterday.

[ Laughter ] >>Ibrahim El Far: So, anyway, --

[ Applause ] >>Ibrahim El Far: So, just to give -- I want

just to gauge a little bit what kind of audience we have here. So how many of you do UI testing?

By -- oh, cool. How many of you do JavaScript, like write

JavaScript on a daily basis? Oh, excellent. Interesting.

I always found it interesting, by the way, that a lot of the people who deal with Web

testing on a daily basis just don't really deal directly with JavaScript.

How many of you have -- deal with manual testers, like, work with manual testers on a daily

basis? Cool.

How many of you have filed bugs? [ Laughter ]

>>Ibrahim El Far: How many of you have spent a whole lot of time talking to developers

about convincing them you that need to fix those bugs?

How many of you had the opposite experience of just, you know, developers just fixed it

on a regular basis. Oh, fantastic. Of course, Tony has it right.

So there's always a place for testers. Can you help me with the clicking, please.

>>Jason Arbon: Yes, sir. >>Ibrahim El Far: That's why I brought in

Jason. He's basically my clicker. >>Jason Arbon: Not funny.

>>Ibrahim El Far: Yeah, pretty much. [ Laughter ]

>>Ibrahim El Far: So crowd sourcing was mentioned yesterday. And -- as a powerful sort of way

that actually scales in terms of bringing the crowd to test, sort of like hiring a dedicated

set of testers. And that is especially true for consumer applications.

So I'm not going to pretend that this works for enterprise applications. I'm not going

to pretend this works for operating systems, or even back-end stuff. But for consumers

with a lot of UI stuff, this works pretty well.

So there's always a place for highly skilled bug hunters who are actually working for organizations

whose core business is testing. Actually, even if companies, large companies, like Microsoft

or Google or et cetera, decide one day that we're -- James is right and test is dead and

we're not going to hire any more testers, I think there's always going to be a place

for testers in organizations whose core business model is about testing. We'll get into that

a little bit later in the presentation. So you can keep arguing about who needs to

be doing it or you can argue about where we need to do it, in the United States, or in

India and China, or in your company versus outside of your company. But that's really

not the point. >>Jason Arbon: We didn't practice any of this,

by the way. >>Ibrahim El Far: This is not very rehearsed.

So, really, the work is not going away. If -- you know, basically, what we need is,

we still need to test your software. We need to do it in an exploratory way or a scripted

way, doesn't matter, but it still needs to be done. Bugs still need to be filed. And

-- why are you not clicking? >>Jason Arbon: Because you don't make the

click noise. We talked about the click noise. (Clicking).

>>Ibrahim El Far: But, I mean, how are we going to go through this? You're just not

doing your job. >>Jason Arbon: I'm doing it. I'm just doing

a review level 3.1. >>Ibrahim El Far: I'm already talking about,

you know, the work that's not going away. And -- yeah, okay. Thank you.

[ Laughter ] >>Ibrahim El Far: So I'm already done with

this bullet, so it's done. >>Jason Arbon: I can quit.

>>Ibrahim El Far: So the work is not going away.

And -- I'm just going to do it myself. >>Jason Arbon: Please.

[ Laughter ] >>Jason Arbon: Management.

>>Ibrahim El Far: But we don't really -- here's the thing. I don't know if you ever hung out

with a manual tester or if you've ever been one. When I first joined Microsoft a long

time ago, I was hired as a test engineer, which technically meant that I clicked buttons.

And life was kind of miserable, actually. I had to, you know, set up my machine, install

a whole lot of software. And even in today's world, with, you know, everything being -- going

into the cloud, you still have to install a lot of tools and spend a lot of time learning

about these tools. So -- Click.

[ Laughter ] >>Ibrahim El Far: -- you also tend to spend

a lot of time away from the application. The thing about exploratory testers and manual

testers is, really, they need to spend as much time as possible just playing with the

app. And they don't. They keep switching context a lot. They keep switching and saying, "Oh,

I need to actually create a test case in the test case manager." Really? I mean, seriously?

I need to enter data, like who's the author of this test and what's the title of this

test, and here are the steps of this test. And, by the way, here's how you execute this

test. Really? I mean, come on. We're in 2011 already?

Something like that? Yeah. So that's the other job that Jason is there

for. Click on my slides, and shake his head. >>Jason Arbon: I'm ahead of you.

>>Ibrahim El Far: That's right. Thanks. So also, another really terrible thing, another

terrible experience about this is, you know, we go ahead and we know that some test cases

are, you know, going to fail or pass. But we give them to test -- because we haven't

got a chance, especially in the UI testing world, we haven't got a chance to automate

it, so what we do is we give them a whole bunch of spreadsheets and say, "Go ahead and

just run through -- run with them." And that's really unfortunate, because what

that means, instead of them exploring new functionality, they're stuck, essentially,

repeating the same thing over and over again. And then often we hire an army of them just

so we can cover things like all sorts of machine and browser combinations.

You know, they -- manual testers these days also still spend a whole lot of time testing

a whole lot of stuff that can be found by a machine. And so, really, test case managers,

bug filing systems, and record playback software, which is really central to the life of a manual

tester, I was going to say it just needs to die. But what I really mean is it just needs

to be invisible. It needs to be invisible so that the user of the software, who can

be a tester, or the developer of the software, or the manual tester, who is a dedicated manual

tester, anyone can actually -- [ Laughter ]

>>Ibrahim El Far: You know, we only have 30 minutes. So....

So, for us, when we talk about -- you know, at least on our team, when we talk about the

crowd, we actually don't really mean just people. We mean a crowd of people and a crowd

of machines. And it's really just the best of both worlds.

So this is the standard, oh, testing is labor-intensive. Ooh.

So let's go on to the next slide, actually. So I'm going to go through -- so I'm going

to go through just a bunch of the tools -- tool work that we've been doing. BITE stands for

browser integrated test environment. It's -- the client portions are already open source.

You can go and take a look at them. And I'm -- we're going to first take a look

at the record and playback feature. >>Jason Arbon: Your internal Google email

is popping up. >>Ibrahim El Far: Oh, yeah. Hopefully, -- oh,

well, just try to ignore that. Will that show up on video?

>>Jason Arbon: Yeah. >>Ibrahim El Far: Oh, crap. All right.

But I quit all notifiers, then. It's only the job that pays the bills.

So keep in mind the first thing that we wanted to do is to make sure that testers just don't

really -- especially exploratory testers don't have to do a lot of work to sort of enter

metadata about their test cases, and enough information for these test cases to be repeated.

So one of those core components is really a traditional kind of traditionally boring

piece of testing called record playback, except we do it in a pretty cool way.

So what you do is, all you need is to install an extension. BITE is really just a Chrome

extension, at least today. And what you do is just you start -- go ahead

and play with your -- with your application, in this case, you know, just open up the Google

page. And then as you go through this application, essentially, what BITE is doing is recording

everything that you're doing. And without you necessarily writing any of the code, what's

being recorded is both -- I don't know if you can see this very well from where you're

sitting -- but you can see both, essentially, JavaScriptish kind of code that's being generated,

but also kind of an -- a plain English sort of presentation of the test case. Depending

on who's using this tool, they might want to take a look at something a little bit more

friendly than JavaScript as a representation of your test case.

You can do things like also add validation. So as, for example, you're going ahead and

playing with your app, you can say, for example, that these are your expectations about what

to find on a certain page, like some things are a must, some things are to be ignored,

some things are optional. And then also, as you are exploring your software,

we are taking, essentially, a snapshot or a screen shot of the application step by step.

So this way, basically, now, you have three ways of looking at your test case later. You

can either look at the JavaScript code and run that. You can look at the English, plain

English representation of the test. Or you can look at a sequence of screen shots.

So -- But here's the thing: So, again, how many of you on a daily basis do UI testing

and think that UI testing is kind of easy to break? Or brittle?

Yeah. It's like -- so I've never really worked too

much on UI testing until recently. But I've -- you know, I work very closely with people

who did. And it's amazing. I mean, a lot of the time that people spend in UI testing is

about, essentially -- well, reimplementing a whole lot of these test cases.

And what we are finding is, that if you have the right set of tools, it's actually far

less expensive, far cheaper, if you have the right set of tools to simply rerecord all

the test cases than to actually develop -- you know, fix test cases or even develop them

from scratch. Just our experience. We'll talk about that some more.

So, for example, in this particular tool, what we do is, if your test case fails, what

we allow you to do is, by a simple click -- this dialogue says this step failed finding element

blah, and it actually -- it actually highlights that element.

And what you do is, once you click on that element, then you -- the new element in the

page that's equivalent, it kind of fixes it automatically and finds all the other instances

of that element in the same test case and sort of fixes that as well.

So what that means is, if someone messes with your devs or with elements on the page, it's

not the end of the world. You can simply just say, oh, here's the search -- here's the new

search button, as opposed to -- and then it'll find all the instances of the old search button

and basically replace them with the new search button, as an example.

So that's basically the record playback portion, it's basically record playback plus the ability

to fix broken tests reasonably easy. >>Jason Arbon: Can I add one thing real quick?

Thank you. The cool thing that we'll get to a little bit later is we're generating JavaScript;

right. We're not generating Java that need to be compiled local and deployed to machines.

You join these test cases, we save them automatically in the cloud. It's just saved to a server,

so you never have to worry about file management or local store or anything. And the core thing

is this stuff actually runs on Chrome OS. So it runs entirely in the browser, which

is one of the main motivating factors for this whole project. That's kind of a cool