>> Sarah Zampardo: Great. So my short talk today is going to focus on testing to learn.

So I'm going to start, as a Wisconsin native, and I'm sorry Tim, I should have given you

warning about this, I don't know where you went, I'm a big Packers fan. I'm really looking

forward to the beginning of the season in just two weeks against the Cardinals. And

this year I have a new strategy for Coach McCarthy. Instead of playing all of your top

players in every game, instead I recommend that you bench them and play your second string.

See, I did a little bit of analysis, and I see that when you play the top players, you

actually have a lower winning percentage than when your second-string players have more

playing time.

Not so fast, right?

In fact, it's the case that—it could also be that you're only putting in your second-string

players when you're far ahead in the game, so the more playing time that your second-string

players have, the more likely you are to win, but it's not because the second-string players

are actually creating the winning for the team in the end.

And so one way it could actually take a look at this would be to test actually putting

in the second-string players earlier in the game, and seeing if the Packers are more likely

to win if we had the guy that you're probably don't even know his name at the bottom, versus

Aaron Rogers who is actually playing in the game.

So, put more simply, it's not that correlation establishes causation, rather that experimentation

enables you to establish causation.

So Clara and the Heron Foundation have asked us to consider what is a good deal, and how

do we identify good deals? And that's defined as understanding in social and financial sectors,

how we can find different enterprises that create social and financial performance and

impact together. So identifying a good deal is pretensed upon the ability to actually

understand the impact of any given program, intervention or initiative.

And the gold standard of doing that is experimentation. The whole methodology is centuries old, the

scientific method, and it's been applied selectively and successfully in the social sector over

many years. However, the potential is much greater. Why? And what can we do about it?

I'm going to leave the topic of data infrastructure to other people who have far more experience

to talk about that, and instead focus on using K-12 education as an example of a social enterprise,

understanding the key and fundamental challenges that I hear often, against experimentation

in the education space. And then I'll share some of the ways, through examples and suggestions,

that we have actually overcome those challenges, using examples from the private sector.

So, a little bit of background about myself first. I'm with Applied Predictive Technologies,

as Hope was explaining, and we're the largest software cloud-based predictive analytics

firm in the world. We work with a variety of consumer-facing firms across industries,

including hospitality, telecoms, banking, CPG, including firms such as the ones you

see here , and we help them answer questions about the incremental impact of their programs,

which surround issues such as pricing and promotions, marketing effectiveness, brand

management, capital investment, remodels, among many others.

In addition to my client-facing work, I also head our social sector practice, where we

primarily partner with school districts and other educational institutions, to similarly

apply the experimental method, to understand the incremental impact of their programs.

As some additional background, all my academic background is in experimental economics, specifically

focused on the development space, and all my prior professional experience is in the

nonprofit and public sector spaces. I'm really excited because I've seen that the tools that

I've learned in the private sector that are much accelerated beyond what I've seen in

the social sector can actually be applied successfully in the social sector. And I hope

to share some of those meaningful examples with you.

So the first place that I'll start is actually with an example to help more explain the experimental

methodology. So this is the good deal: a different sort of good deal, a good sandwich deal. So

in 2004, there was a franchiser for Subway in Miami, who saw that sales were declining

on the weekends, so he wanted to offer a very simple and easy-to-understand promotion. So

he offered any sandwich for $5.

The $5 footlong! And on average that was $2 cheaper than your average footlong sandwich.

He saw wildly successful sales, so other franchisers in the area also opted to try the promotion,

and they also saw similar results.

Before long, franchisers across the country were clamoring for the corporate headquarters

to actually introduce the promotion nationally and put some big marketing support across

it. However, there was still a lot of skepticism within national.

So you can imagine the types of confounding events that could be creating the positive

sales and profit results that those franchisers saw in Miami. One possibility is that Miami

overall was just seeing an upward trend in the market, so people were more interested

in Subway, for whatever reason, what was going effectively in that area. Another thing that

the executives pointed out was that other areas of the country were also seeing positive

change, so sales were up in Denver, or in Pittsburgh, and they had no $5 footlong, so

how do we know that the $5 footlong was actually the idea that was creating the positive growth?

So at that time they actually partnered with APT to test the idea. In order to get rid

of all of the noise that's going on in the system, we selected a set of test franchisers

across the country to actually introduce the $5 footlong, and compared their performance

before the promotion was offered, and during the promotion was offered, on sales and margin,

to see how the $5 footlong actually affected those particular restaurants. And then, given

the results of that, in 2008 it was actually launched nationally.

So how does this all relate to K-12 education? In a similar fashion, there are huge decisions,

and great decisions, very expensive ones that are made every day across our schools. And

they happen on many different levels of government, also in combination with foundations and tech

startups and other entrepreneurs.

Also, the challenges are great that these interventions are trying to solve. You don't

need me to tell you about what the challenges are in education today in the country. The

question though, is how do we understand what works? If there are a whole slew of new software

platforms for teaching math more effectively to second graders, how do we know which one

is the one that we should actually get our teachers to learn, take them out of the schools

for a day to have an education session, and then bring them back to the classroom and

take time away from what the teachers would normally be doing with the students, to instead

be learning math on this new software program? How do we know which ones are more effective,

and which ones are actually driving the results that we're looking for?

The answer in the education space is the same as it is in the private sector. We need to

select a group of students or schools and call them "test". Introduce the intervention

into that special selected set, and then compare them to a group of control students or schools

so we can measure the effect of the intervention, above and beyond the other factors that are

actually separating and aggregating student performance.

So if this is such a great idea, as the economist in me asks, why aren't all school districts

doing this?

There are four fundamental challenges I hear in conversations with school districts that

I would like to share with you, in order to show what are those challenges to experimentation,

and then also suggest some ways that we have actually overcome them, through the examples

of what we've done in the education space.

So the first fundamental challenge that I hear is one about equity. We were actually

talking about that this morning. "Experimentation is unethical and is unfair", and this is a

quote. "How can you justify offering an intervention to only some students, when you know it could

help the students that are held out as control?"

And I think this is true in a lot of different areas when you're thinking about using experimentation

to measure social impact.

So the reality is that there are always a lot of different effects that are creating

different behavior, and impact, but be it for students, schools, or for stores. If you

think about a store for example, what's impacting the traffic that's coming into the store is

a function of some uncontrolled factors, such as the local economy, or maybe there's a store

down the floor in the mall that has a really successful national promotion going on, so

you have more traffic coming in, or maybe it's something like weather.

And it's also some controlled factors, like how strong the store manager is, or what types

of new products that store has introduced.

The same thing when it comes to schools. It could be the fact that there's a really effective

administrator or principal. It could also be that a school was selected by the superintendent

to receive a special grant, and there are only six grade schools in the entire district

who received that grant. And it's not that there's some reason that the other schools

in the district didn't receive it, or there could be some that are on the margin, but

some students are receiving additional resources just on the basis of limited resources that

exist, and no matter whatever sector that you're in.

So the way to approach this, instead of saying that it isn't fair, is to actually use that

variability and let it work for you, and use it as a way to actually do natural experiment

analysis. So an example is we partnered with a major school district, and we were taking

a look at turnaround schools. I'm not sure if you guys are familiar with the program

or the intervention, but basically the district has identified schools that are severely underperforming,

and they identify that the principal and about half the staff have to be turned around, and

then they invest additional resources within the school to bring the standardized test

scores up to proficiency within three years.

So no student chooses to attend a failing school, right? No one wants to attend a school

that has to be turned around the year after they enter the school, but yet the experience

of that student in that school that's turned around gives us an opportunity to compare

them to other students who are not in schools that are turned around. And that analysis

showed what are the different characteristics that lead to more successful turnarounds.

So that's just one example of how natural experiment analysis is actually taking areas

of inequity—and inequity is inevitable—and using it to still learn from it in very valuable

ways.

The second fundamental challenge is experimental efficacy. So to quote: "I'm concerned about

measuring the impact of a new technology in classrooms, because I don't have confidence

that all teachers will employ it correctly, or consistently. In this case the analysis

isn't measuring anything useful."

So this is one that I hear most often from interventionists who are actually coming from

outside of the education space, such as technology entrepreneurs. The reality is that you want

experimentation, you want any impact analysis, to measure not just how something should perform

in a clinical and perfect setting, but also how it performs in combination with the implementation

of that idea.

Let's stick with technology. If you have a new software program like the one I was talking

about earlier, teaching math to second-graders, if you have that particular software program,

you could, the coders who created it could go into schools, and work directly with the

students on using it, and say that it was wildly successful at increasing the scores

or math-readiness of students. However, a better test of how that program would actually

perform if launched across the district would be putting it in the hands of teachers, and

then measuring how those teachers both use the idea, and also implemented it consistently

in the classroom. So, the efficacy of how an idea is implemented is just as important

as the idea itself.

You can also measure it explicitly, when that's possible, so one example is we were taking

a look at the impact of a tutoring program on reading-readiness for smaller students,

and on average there was no impact they were able to measure. But then we could also look

at the number of hours of after-school tutoring that students experienced, and we saw that

as the number of hours increased, as expected, there was more of an impact of the program.

So in cases where there is a range of efficacy, you can actually measure that in order to

include that in the analysis that you're doing, and only further the idea of what's the specific

incremental impact of every hour of tutoring that was available to these students.

The third one is data, which is good given this session. "We don't have enough of the

right data. Our data is not very high quality, and it's very infrequent. Our schools report

standardized test scores only once every year per student and at the end of the school year."

So, this is absolutely right. The data that's available in the education space pales in

comparison to what we work with in the private sector. To illustrate this very concretely,

the particular school district I was talking about, if you take all the data we have for

them, so that's ten years, it's hundreds of schools, and tens of thousands of students,

if you take that data, it is less than the data of one store for one year in the same

city. Not even six months. It doesn't compare.

While that's true, we shouldn't let the perfect be the enemy of the good. There's actually

incredible work and analysis and experimentation that can be done just using the data that's

available today. So an example of this is, for about three years now we've been partnering

with a very innovative and leading charter school, and helping them take a look at the

incremental impact in academic performance of students who are entering their school.

And the nice thing with this particular charter school is that it's a very natural experiment

because some students are able to enter the charter school and others are naturally lotteried