製品の作り方 I, マイケル・セイベル, スティーブ・ハフマン, エメット・シアー - CS-183F (How to Build a Product I, Michael Seibel, Steve Huffman, Emmett Shear - CS-183F)

字幕表動画を再生する

So this is going to be the first of four lectures on how to build a great product.
I apologize because none of the people here today can speak to that topic.
>> [LAUGH] >> [LAUGH] My name is Michael.
I'm the CEO of YC and help run the batch and
program as most of you understand YC to be.
This here is Steve who's the co-founder and CEO of Reddit.
And there's Emmet who's cofounder and CEO of Twitch.
Can you guys introduce yourselves in a couple of seconds?
>> You just did but I can do it again.
>> [LAUGH] >> And Steve, yeah co-founder and
CEO of Reddit.
I study computer science.
I graduated from UVA in 2005.
I was at Reddit for five years, I left for five years.
I've been back for about a year and a half.
>> My name is Emmet I graduated from Yale computer science in 2005.
I started my first company right out of college.
We made a bad copy of Google calendar before Google calendar existed.
And- >> Google calendar
>> Yeah, that's right, and
then we started Justin TV which is the company that one turn and
so I've been doing, basically continuously employed
working on that since 2006 >> And
then can you just, each of you guys say whatever the KPI you guys track for
the business and approximately what it is?
Just to give folks a sense of why you guys might be
experts at building great products.
>> [COUGH] One of our more shareable ones is monthly active users and
we hit 300 million last month.
>> Nice. >> [LAUGH]
>> That's a good number.
>> Our primary metric that we care about is a metric called minutes watched,
the number of minutes of video that are consumed every month on Twitch.
And we just crossed 30 billion minutes.
This year for the first timers.
This is very exciting.
>> Total? >> No, per month.
>> Jesus.
>> [INAUDIBLE] >> So
that's the main thing we look at rather than uniques.
>> So, what we're going to do for
you guys today is I have about 12 questions I'm going to ask these guys.
We're going to see how many of them we get through.
And then we're going to leave the last ten minutes for
questions about start up stuff, YC, or whatever else you'd like to talk about.
My goal is to kind of focus this advice on early, early stage.
So, these guys do a lot of stuff now that really
is completely irrelevant to you >> But hopefully they'll be talking to you
about the things they did, bad or good, when they were starting.
So, I want to start with one of the major tenants of
YC is how you talking to your users.
And we repeat that a bunch of times, and then we don't really go into any detail.
So, can you guys talk a little bit about how you talk to talk to your users?
Maybe the good and maybe some of the mistakes you've made as well.
>> Sure, so, I, most of the advice I give today will have two perspectives,
one at Reddit and one at Hipmunk.
because the companies are completely different and had different paths and
I think, kind of the ground rules for this discussion,
the first piece of advice I would give is every company is really different,
and there are lots of different paths you can take.
And what works in one place, very likely won't work in another.
So at Reddit, I mean the product is talking to users, right?
That was built in.
In the early days we didn't have comments and we didn't have communities.
So actually, we got a lot of emails, was our,
for the first probably six months was the way we communicated to our users.
But when we added the comment feature,
there was basically no option but to talk to our users.
And the challenge then becomes which users do you want to listen to?
And what are they actually saying?
And what do they actually mean?
Because they're often saying one thing and they mean something else.
At Hipmunk, because it was selling plane tickets and hotel reservations.
It was completely different.
We didn't really have a good forum for connecting with our users.
We used a product called Olark which was basically this imbedded Java Script
chat thing.
And that was indispensable, we still use it.
It was users, if they got stuck, they could ask us for help.
If they wanted to give us compliments or criticisms, they could do that.
And we can often, both over email at Reddit and over Olark at Hipmunk,
turn Like emotional users, angry users will often become your most loyal
users if you can flip them around, there's like this absolute value of emotion.
And so, if you can find those angry users that are having a bad time and
treat them well they will likely.
Quickly turn into some of your most loyal supporters.
So, we found that whole cycle to be very valuable.
>> The interesting thing for me, between Twitch and Justin TV,
which are two very similar product right there.
With live video products with chat.
So for Justin TV, we were our own users.
We were building a product for us to build a TV show.
And after that,
we really sort of declined to actually speak to users in any significant way.
And, unsurprisingly, all of the good ideas we had were things we came up with
when we were building it for ourselves.
Which sort of brings me to my first point, of talking to users, is it's possible,
especially very early, to build a product yourself.
And, if you truly are building something that is for
you to use, that can be super effective.
You don't necessarily need to talk to talk to anyone else.
Eventually, you'll need to, because you'll learn about people other than yourself but
for the first three months it's to develop something, just something that you want.
But you have to actually really want it.
It can't be, you can't be like lying to yourself about that,
it can't be something you think is cool.
It has to be something you, yourself actually desperately want to use everyday.
And then we had went through a long period in the desert where we were
wondering about.
Not talking to users, not really using our own product, and
not building anything of value.
And the thing we did with Twitch that I thought was really good for
production users is we talked to them in person.
I had done some amount of surveys and talking to users over email.
And I found that getting people on, at least on Skype, or if not,
literally in person changed my depth of understanding of who they were.
The other thing I would say about about it is that we focused down really hard on
which of our users we cared about.
And we decided it was the broadcasters and
not only was it the broadcasters, it was the successful broadcasters.
We didn't want to talk to people who had two viewers.
We wanted to talk to people who had 200 views.
And so, we went and talked to a bunch of broadcasters on the platform who
decided they were the important ones.
The other thing we did that was I think really valuable is we decided it wasn't
just people who were currently using our service it was people who we wanted to use
our service.
And so, we talked to a bunch of people who were streaming on other platforms rather
than just already on Twitch.
Because the people who have chosen to not use your product,
especially if they have deliberately looked at your product,
decided it sucked and went somewhere else, are like,
some of the best people to talk to, because they know what's wrong with it.
And so, I actually think the most valuable things I learned, especially early on,
were from users who had tried Justin TV, they had tried UStream.
They tried own 3D, they tried YouTube, and
they had a coherent opinion as to why ours was not the one they chose.
And that was super valuable,
because they really pointed at the details of where we were wrong.
>> And often those details are smaller than you think.
>> It's like memorably one of the biggest issues was we were
not listed on the Team Liquid Starcraft fan website.
We lost maybe 3,000 broadcasters over that.
We had to email them and ask them to re list us,
because they just didn't have us there.
Tiny issue, but like really mattered to our users.
>> It's easy to run a company for years and then be like,
I should have fixed this stupid one day fix if you don't talk to your users.
>> So a lot of people talk about MVPs, minimum viable products.
How many of you have heard that term, MVPs?
So one of the challenges I have is when folks like you apply to YC and
get into YC even though you know the idea that you should be releasing something
that is crappy, you don't do it.
And as a result, users slow down.
And oftentimes, that prevents you from getting into YC or
prevents you from building and talking to customers.
And so, I think that what's interesting is that just TV and
Reddit are pretty good examples of crappy MVP's.
So, can you guys describe how the products worked in the beginning or
maybe how they only kind of worked?
>> Yeah, we started, I think I wrote my first line of code on June 3rd or
4th, 2005 for Reddit and we launched on June 22nd.
I didn't even launch it.
Paul Graham linked to us from his blog without telling me.
>> [LAUGH] >> And from then,
it was on and the good thing about that for
us in particular is we didn't actually have a vision for Reddit for
better or for worse which would affect us in a lot of different ways going forward.
But as soon as we launched, that's when we started to find the path and we just would
kind of follow the users everyday as what do we think is best for the users?
What do we think is best for us?
And just build towards that and
we've built a lot of loyalty through those actions and
that would not have happened, if we were sitting in our apartment not launching.
Because if we launched and then for the next six months, we just added a ton of
new features, only about 25% of which lasted more than a day or two, but
there was stuff we though was a great idea.
Like we had these categories, I remember in July at some point.
I made Alexis my co-founder stay up all night one night categorizing every link
that'd ever been submitted to Reddit.
And the next morning, I was like we lost.
It's not working turn it off, but we could've probably, if we had launched,
we could have had that feature until November and then launched and
then wondered why it didn't work.
>> I think that Amazon actually has a really good saying about mineral and
viral products.
They like to call them, what they're aiming for
a minimum remarkable product which I think gets more at what you're trying to
accomplish which is a viral product bring into mind this whole ecosystem of support
like a viable organism that's got all the things it needs to live and
you don't actually want that.
Like if you're launching something that's actually self-sufficient, that's viable in
a this thing could live on its own sense, you definitely waited too long.
This thing is going to be on live support when you launch it and
you are the life support system.
What you actually want is you want something about
the product when you launch it to be actually good.
You want to get to the part where you've built something that maybe at least one
person in the world finds remarkably useful or remarkably good and you don't
know really if you've gotten there until you put it in front of someone,
which is why you need to launch it so quickly.
I think it's really important to remember Dropbox, for example,
one of the big YC successes did not launch an immediate product, because they were
building a backup system and backup systems aren't allowed to lose your data.
[LAUGH] And so, the last thing you want to do if you're building backup software is
go rush something out that's going to lose your customer's data,
because you will never get that trust back.
And so, the minimum remarkable product for Dropbox was a video.
Basically, saying, look at this awesome thing we were going to build and
getting all the senate for a newsletter and that worked, because the idea like
the visual idea of what Dropbox is going to do was super compelling.
That would not have worked for just in TV.
We really need to get something out in front of customers Immediately.
Most importantly, because we all need to push the show in this idea we are going to
build the show and we basically did the minimum possible work
may be a little less than that to get our show on the internet.
And when we launch it, it like could only have one channel.
The way that broadcasting worked, the only way we could
actually get it to work consistently is we had a Mac Mini
running Parallels which is a virtualization environment for
Windows XP where we had this piece of a broadcasting
software called, it had got bought by Adobe later.
Onflicks?
OnLot?
I can't remember.
Anyway, that we were automating with Windows shell scripting
to pick up the right camera input and then auto reboot if it went down,
because it broke all the time because it was really buggy software and
that extremely jury-rigged thing worked.
It was like put a video stream on the internet and it was not a scale.
We could not scale that up to thousands of streams.
It was impossible, but we got something that at least was compelling to one person
onto the internet, that was compelling to us and that was a critical step for us.
because before, we did that.
We basically weren't learning anything.
>> At HitMonk, we again, started in June 2010 and
I'd given my co-founder, basically yhis ultimatum of what our MVP was.
We have to have legit data, because we could scrape anybody.
We needed legit data, real data.
And when a customer buys a flight, we get revenue.
And I'll say, if we can do that in three months,
we will continue with this business.
And if we can't launch that in three months, we're calling it.
And that for me, that MVP, that was that minimum useful thing.
Like if we had a customer who was giving us money,
that meant we did something right.
And unfortunately, we actually achieved that in three months, so
I worked on it for five years.
>> [LAUGH] >> But
I think it's good, sometimes to set yourself deadlines and specific
milestones to keep you focused and keep you pointed in the right direction.
>> So another thing that comes up a lot and this is actually one of
the things that hits the, when a company gets into YC is we will often ask them,
what are they doing for analytics?
What are they tracking?
And I think that this is another one of those MVP like things, everyone knows they
should be doing it and it's the first thing that gets caught off of any list.
So, can you guys talk about how an early stage startup
should think about measuring?
What analytics tools should they be using?
And generally, like how you incorporate tracking into building a product?
>> I have two very different answers, so reality is somewhere in between.
At Reddit, we just didn't.
We didn't know our traffic for years.
We didn't measure anything.
We did the product development by intuition for a very long time.
The company ended up working out.
I'm still now quite sure how many users we have and it's a huge,
huge pain in the **** now.
>> How many users you have?
When you get to 300 million, it's like- >> [LAUGH]
>> No, in all seriousness,
we have no idea how many unique people use the site every month.
>> It's probably somewhere between 250 and 320.
Yeah.
>> Anyway, at Hipmunk we were actually really good about it from day one.
And we were really because we didn't have that kind of instant feedback with
the users we actually got really disciplined about testing, and analytics,
and measuring, and watching.
And that company lasted honestly,
quite a bit longer than I think it would have without doing that.
And so if I can do it again,
there are a lot of corners you can cut when you're building your MVP.
You can create all kinds of tech sins and corporate sins and I don't know,
be completely dysfunctional, your users don't care.
But data debt can haunt you.
because you can't make up missing data, and
eventually your intuition's going to fail you.
So I would think very carefully about what's the minimum viable data.
You don't even have to look at it.
Just log it somewhere so it's there when you need it.
>> To reinforce that point, having historical baselines for
important user actions, you will thank yourself later.
Every time I've ever been like, right, how many people are using that feature?
And we don't know, and we can't find out.
And now even if we find out how much it is now, we don't know if it's up a lot, or
down a lot.
And we have no idea what the trends are.
It's awful and you can't make a decision for
another three to six months as you build that baseline.
Actually I got the best advice about this from Sue Hale, who runs Mixpanel,
when he was convincing us to run Mixpanel for the first time.
Which, by the way, didn't work, because Mixpanel at the time was only six month
old, and it broke when we tried to everything.
But, we went back to it- >> Thanks for
convincing me to use Mixpanel before you came to that conclusion.
[LAUGH] >> You're welcome.
>> It works better now.
>> It works a lot better now, we went back to it, we're still on Mixpanel today.
We use a bunch of other stuff to, because we're a little too big for just Mixpanel.
But Mixpanel, the advice Sue Hale gave me that was,
I think, more important that the software we were using.
Mixpanel in specific, even though I do think it's good software was,
pick your top five to seven most important user actions, whatever those are, and
just log those.
You don't need to log everything, you really don't.
Most of the things people are going to do with your product aren't important.
There's maybe five things people can do.
Like for Reddit, Reddit might be vote on something, comment on something,
submit a story, load a page, I don't know some short list of things.
And those are important and you can kind of just ignore everything else.
If they're not instrumented who cares.
And so we instrumented, I watched a minute of video,
I sent a chat message, I followed a stream,
and I bought a subscription.
And that turned out to just give, those four actions gave us a huge amount of
insight into what was going on with our customers, that we hadn't had before.
>> This is one of those areas where, if I could go back and give myself just advice
for 30 minutes, that would dramatically affect the trajectory of the company,
it would be around this topic.
And so if I were in your shoes, I would just take half a day, and
just read best practices for collecting events and storing them.
>> But and also, I know another thing I'd recommend is use a third party service.
Use, collect logs, by all means, put them into your own system.
because I guarantee you,
you'll want to do something a third party service doesn't allow at some point and
you'll be super annoyed if your data's only in the third party service.
So collect logs or whatever, but just in the beginning,
you don't have time to mess around with building analytics tools.
So just use Mixpanel or use Google Analytics or use whatever.
Anything, they're all better than what you need right now.
Later on you'll need something more complicated.
You don't need it today.
>> So one of the challenges that happens usually after a company,
in the early stages, starts talking to their customers.
Is they realize maybe there's a big disconnect between what they thought their
customers wanted and what they actually want.
And at YC this usually comes to us in office hours in terms of,
we call the big redesign.
So everything about our site's incorrect.
We just basically need to rebuild it from scratch.
Or rebuild huge components of it from scratch in order to start serving our
customers.
Needless to say, this is a very, how do I phrase this?
Distracting and oftentimes not fruitful exercise.
But how do you see that type of challenge of, man, we got this feedback and
we think we need to be going in a different direction now?
But how do you deal with that kind of challenge?
>> That's a big one.
And I think you need to be clear whether you're talking about tech or the product.
I'll repeat what I said before, is your users don't care about your tech.
So if you can move forward, leave that for another day.
When you're talking about redesigning your products.
You're probably in trouble,
because whatever habits led you to build a product your customers don't like.
If you haven't addressed those habits,
you're probably going to build another product your customers don't like.
So I would make sure that, because I've been through lots of rewrites.
At Hipmunk we rewrote out flight and hotels probably half a dozen times each.
At Reddit, we re-wrote the tech a bunch of times.
And we're redoing the UI and
UX now with better habits and better data.
And so you want to make sure that you know something new.
But honestly, if you're in that situation, I would also go through
another mental exercise of okay, what's the new MVP, right?
It comes up in tech a lot, right?
The second system syndrome.
The same thing happens in the product where you're like, all right,
I've got this vision.
I'm going to fix all of the products, all of the mistakes at once, and
it's going to be perfect.
That product never ships.
It never ships.
So can you iterate your way there, can you just start from scratch with an MVP and
build it back up?
Those are much more likely to actually get you somewhere.
But they're also harder to do, and requires a little bit more discipline.
>> So we've also gone through a bunch of redesigns.
Big redesigns where you have to rebuild the whole product.
They've been mixed actually.
We've had some that were super successful.
Twitch did a big redesign of our mobile app, maybe three years ago,
from the first version that had basically been a cloned version of
the Justin.tv app to a completely new version.
It's basically the version you can use today.
And that increased engagement by 35, 40% per customer when we launched it.
It's huge, changes that make that big of an impact are actually really rare.
>> Usually data errors.
>> Yeah, yeah, [LAUGH] yeah, they're usually data errors.
And in this case it wasn't.
In this case, it actually was a huge increase in usage.
And so absolutely, redesigns can make a big difference.
But we weren't doing the big Twitch, we didn't do that redesign because we'd
discovered we couldn't move forward on our existing product.
Since we got some insights is the what what things should
be presented at the top level of a nav.
And we re-did the navigation to present those things at the top level instead of
burying them three clicks deep.
Three touches deep, I should say.
Most of the time when I see people doing a big redesign it's because they fall into
this trap that engineers and project managers often fall into.
Which is, we have to fix this eight things.
Let's just fix them all at once it'll be easier.
That's wrong, you should just fix them each one at a time individually.
If you have eight things that are wrong with your product that's awesome,
you have this huge great list of things.
Just fix them one at a time, fix everything one at a time.
Crank them out quickly,
you'll get there way faster one change at a time than you will in a big rewrite.
That'd be great for
a really early stage startup is almost equivalent to pivoting the company.
And we generally advise against it.
>> It's this weird sort of programmer product procrastination.
If I don't know what to do,
therefore I'm going to invent this big project to work on.
>> [LAUGH] >> We don't need to use that library.
I'm going to build it myself.
>> [LAUGH] >> Very common, use the damn library.
>> [LAUGH] >> So I'd like to talk
a little bit about decision making.
So maybe after you talk to your users but before you decide what to build, usually
there has to be some type of consensus building process either function or
dysfunctional between the co-founders.
And what are some tips or tricks on how to come to a consensus?
And then maybe more importantly,
how do you deal with the contrarian, how do you deal with
having to make a decision where you know there's going to be a disagreement?
>> I think the first thing you said is actually super important.
It's something I want to reinforce.
You talk to your users first, and then you have the ideas about the product.
Almost everyone does it in the opposite order.
It's called product validation.
If you find yourself thinking, I'm going to go talk to my users to validate my
product idea, you've gone horribly off the rails.
You do it in the other order, you don't talk to users to validate your product
ideas, you talk to your users to have your product ideas.
And so I would just have a rule in general for
people who want to have a voice in product design,
is that they better talk to the users and look at the data too.
If you haven't talked to users and
you haven't looked at data, you don't get to have an opinion about the product.
I'm sorry, the person who's actually done the work gets to have the opinion.
You can have ideas but they get to make the call.
But in case you wind up where both people really have talked to the users.
And both people really have looked at the data.
And people still disagree as to what we should build.
Which by the way is actually much more rare.
Once both people have actually talked to users and
actually looked at the data, it's usually relatively easy to get to consensus.
But it will come up where people still disagree.
I've found it's better when you just have someone who's in charge.
And at the end of the day, you have someone who's the CEO, and
everyone agrees this person's judgement will be trusted.
And you let them make the call.
Don't avoid conflict.
Talk about it. Argue about it,
but then you have someone who just gets to make the call, because anything else
leads to just decision making that's far too slow for a startup.
>> Yeah, maybe it's because I'm usually the one in charge,
I just feel like I don't have a lot of disagreements.
>> [LAUGH] >> [LAUGH]
>> There are-
>> Meet the CEO.
>> Yeah.
Largely, I find the situation I'm in, I say one of two things a lot,
which is I don't want to argue about it if we can just test it,
and I'm very willing to be surprised on this.
I'm probably right, but
this isn't getting into my top three important things to worry about today.
So let's just see.
It's easy to do that when you have more resources, to let's just see.
But I think about, like Ciss and I over the years.
And Adam and I over the years at Hipmunk, I think as it panned out,
we've very rarely had these philosophical differences over with the product,
sometimes it was timing.
Should we build this thing now?
What's the prioritization of these things?
But we always agreed on general mechanics of what we were doing and
why we were doing it.
And so the strategy might change but the details, we're very rarely not emotional.
So those are getting emotional,
I think there's probably something else structurally wrong.
You might be swimming upstream.
That happens a lot.
The arguments have gotten that rid over the last year like when we try to promote,
we do cross promotion from mobile web to the app, it just doesn't feel great.
It works.
It works and we like the numbers going up but
it's not really with the user's best intention in mind all the time.
Sometimes it drifts into this like it's good for the company but
maybe not for the users.
And when you're in those situations,
that's usually when the disagreements come out.
And you're not really arguing about the future anymore.
You're arguing about short term or long term, or best for the business or best for
the users.
And then when you beat at it, and beat at it, and
beat at it, eventually you come to a solution that actually feels right, right.
That meets your intuition, you don't feel like you're fighting the technology,
you're not fighting the users anymore.
Those are usually situations I look out for.
>> Sir, I want to get to you guys telling product story but
I want to cover one topic first which is that lets take people passed lunch.
Well now you need to have some type of process around talking to users,
looking at data, making a decision, building a product, releasing a product.
Checking your success and repeating, product development cycle.
What are some advice you can give to early stage startups around how to set that up,
what that cadence should be, maybe some pitfalls to avoid?
>> The first, just to take one little step back is, I see a lot
of startups, they get too emotionally high when the numbers are going up.
And they're not investing in this sort of process.
Because growth masks all problems.
And then at some point that kind of stops and
you don't have any of these mannerisms.
I also see startups get a little too low when things not going right at first.
And they kind of psych themselves out.
So you want to live in that in between, right?
If the data's really good, well, check the data.
Right, if the data's really bad, check the data.
[LAUGH] >> [LAUGH]
>> Okay, so then in terms
of the actual process, first of all it's going to be different at every company and
it's going to change, as you scale.
I think it's just most important to have a way of working,
to have if you just want to start out really, really specific of.
The way I like to think about it is where do we want to be in a year?
Okay, let's draw a straight line from that to here,
what do we need to be doing today to get there in a year?
So that's why I like our hypothesis.
Do we need to validate anything about that?
Probably a few things here or there.
Do that, go to check out.
Yes, and then it's just like straight line.
And I like to work on two week cycles.
Check in every week on basically that kind of thing.
Where do we want to be in a year?
Are we still on track, are we still doing the right thing?
Does it still make sense?
Are our assumptions still holding to be true?
If not, pop it up a stack.
And then you give people whatever it is.
If you're early startup, you give them 13 days to work.
And if you're a more mature startup, you give them nine days to work.
[LAUGH] >> [LAUGH]
>> And then keep going on at that.
I found that that's worked really well for me.
>> One thing I found actually is that if your goal of talking to users
shouldn't be course corrections on a week to week basis.
I did a whole crap on a talking users at the very start of
trying to figure out what the users wanted, how to think about them.
But then actually, we didn't because we had other reasons to go talk to users,
but we could have gone six months without talking to a user,
because actually we knew what users wanted.
They wanted to make money on the service.
Streamers wanted to be able to make money, they wanted to reach more viewers and
they wanted positive social feedback for what they were doing.
And you kind of just check anything you were doing,
is this going to deliver this to our streamers, yes or no?
And if so how much?
Great, you don't have to talk to anybody.
You already know what they want.
And you need to go back and revisit it and sort of talk to users
periodically because you need to understand more of the nuances.
The things about your service that are bugging them [INAUDIBLE] doing something
really cool.
But the goal of talking to users is not to get them to tell you what features to
build because users are really bad at that.
They actually have no idea what features to build.
[INAUDIBLE] users is to get to understand them really well.
And users don't change that fast.
So I personally prefer more in-depth time With my customers,
talking to them for a while.
And then, depending on the size of the thing we need to do or we're trying to
move, we might go just work and build and look at metrics for the next six months.
And not actually talk to users that much for a while.
Because if you know that what people want is to make more money,
you can spend six months optimizing the sell through rate on an object or
on how much advertising you have.
And you just can have faith that it's going to be good and you will be rewarded.
>> I think the classic advice that's very difficult to follow is,
you need to have the courage of your convictions.
So, you decide what you're going to build, it might take six months to build it.
And at Reddit we just kind of do it in front of everybody and
get flamed by users the whole time.
Which is fine, I'm used to it, but I have to kind of tell our product people like,
just keep pushing, keep pushing.
Just ship a little bit every week, and then in six months, they'll get it.
And at the same time,
the completely contradictory advice is you have to know when you're wrong.
You have to be able to identify when you're wrong.
You have to be able to say, all of you people are wrong, and I'm right.
And then at some point you have to decide, wait no, you're right.
>> [LAUGH] >> And if you can walk that line, you need
to have this like good combination of high ego and high humility and balance those.
But I always try to just kind of run this process in my head.
Wait, am I wrong?
Nope, they're still wrong.
>> [LAUGH] >> As much as I said don't talk to users,
you don't have to talk to users every week, it's totally unnecessary.
You do need to look at your numbers every week, you really need to know.
And you don't need to look at them every day, it's actually like pointless and
almost just ego gratification to look at your numbers every day.
But you should look at your numbers every single week, and
you should confront, are they up or down and ask yourself why.
And the holy grail of product design is for your numbers to move and
for you to truly understand why they moved.
We still struggle with that.
Like, there have been points where I feel like I understood it, and
then it slips away again as the business gets more complicated.
But you're always trying to be able to explain fully down to like atomic ground
truth, we are up 20% this week because we got this many more new devices in.
And our retention rate is this on those devices, and
this is the conversation rates.
And if you do the math and you can explain it all the way down to this new feature
moved this conversation rate on this page, by this amount, and therefore we are up.
And if you can really do that, it's super powerful.
It's also really hard.
But you'll never get there if you don't build with the intention of looking at it
every week.
>> So in the last five minutes before we open for questions,
can you guys walk through a feature that is visible on your sites now?
And I'll let you choose, that's either bad or good.
And talk about the process of how it came to be.
>> I have a lot.
>> [LAUGH] >> I'll talk about
one of my favorite ones.
This is from the early days of Reddit.
So the early days of Reddit, it was just links.
Everything was external.
And eventually we added comments, probably about six months in.
The first comment was, this is the end of Reddit.
>> [LAUGH] >> The first of,
that would be the first comment on every product we've ever released.
And users started doing this curious thing where they would, so
on any post you could go, you could click on the link or
you could click on the comments page to go to the comments for that page.
Our IDs for posts are just a counter that's in the URL.
So users would submit the URL that would represent the comment page for
the URL they're submitting, right?
So that would just increment, they would take the newest post and
increment by one and submit that URL.
So that you had a post that would link to its own comments page.
And it was really cool, so we called those self posts.
And users were doing this, and I remember watching them do it thinking,
this is really cool.
I would have never thought to do this.
Sometimes they'd guess wrong, right?
And so they'd submit a post to somebody else's comment page.
>> [LAUGH] >> Which was funny.
And so because our user base at the time was fairly technical, the way we made this
work is when you're submitting a link, you would just type in in the URL area, self.
And it would automatically do that little circular thing for you.
And that was the beginning of self posts on Reddit.
Self posts are now 60% of our content.
>> [LAUGH] >> I don't know, a few million a day.
And we would've never built that feature if we hadn't been watching our users.
I don't think I, I mean I'm explaining it now, but if I explained
that feature without it existing I don't know if it would make a ton of sense.
And users, most of them didn't get it at first either.
They were just like, comments are destroying the site and
there's not even a link here, it's just comments.
And it just kind of goes to show, I think the best product ideas are the ones
where your users doing it anyway, and you just grease it a little bit.
That's when I talk about before where you're either swimming upstream, or
just everything just feels right.
That was a classic example of it just felt right.
And it completely changed the site.
>> I think my favorite story here is an advertising product actually,
which is funny.
So all of our users told us really really clearly, we want to
show advertising and make money off of our streams.
It was a clear desire, everybody wanted it.
And they also told us their least favorite thing about
our competitors' products was that they would show these ads over the stream, and
it would be disruptive to their user's experience, and they hated it.
And it was very interesting getting these two pieces of feedback because
they both really wanted us to put ads on.
And sometimes, literally, sometimes different people,
sometimes it was literally the same people.
Their biggest complaint about the opposing service was,
competitive service is annoying my viewers with all these ads.
And this was like, this is a hard nut to crack.
Because you kind of need to solve for both,
you're not allowed to just give up as a product manager and pick one.
And I'm really proud of our solution, which was, we gave them a button.
We inverted it, and
we gave the streamer the ability to hit a button to run ads on their own channel.
And, like it's funny because normally you'd think
of that as something that will, if you're uploading content to like Facebook,
you would never press a button on that to show more ads to your friends.
But because we were cutting them in on the revenue,
they were actually heavily incentivized to hit this button.
And we basically got this paid workforce
of people placing advertising in the downtime of the stream.
And I'm really proud of that clever solution and
I think it points at something that I think is a really important principle of
product design in general.
Which is, that you want to just try inverting all of your assumptions.
We got stuck on that advertising problem for
a long time because we didn't want to make it disruptive to the customer experience.
And we couldn't figure out how to automate putting it on the stream in a good way.
And at the same time we really wanted to show ads on the video stream.
And we had
this implicit assumption in our heads that we had to figure out where to put the ad.
And once we gave that assumption up,
it was much more clear what to do going forward.
And I think that's a, and it's one of the most, I don't know,
I'm just really proud of that product today because it was one of the things
that made our broadcasters super happy.
And no one, like you can't complain, you hit the button.
>> How many times is that button clicked every day now?
You know, that's a really good question, I don't know the numbers off hand, a lot,
it's clicked very often.
People really like that button because it's a,
click the button make $15 button and like, you know.
>> [LAUGH] >> But
they're also aware of the trade off.
Which is that their viewers don't like it when they hit the button too much.
And so by putting that in their court, they get to make the optimal trade-off for
their stream and their situation.
>> All right, those are my questions, I want to hook it up to you guys.
Happy to talk about product, happy to talk about YC stuff,
whatever you'd like to chat about, go ahead.
>> What about the process of discovering you've had product market fit.
Getting past that launch pathway and discovering there's something sustainable,
beyond this real core group of users that are really committed to you?
>> Just to repeat the question, how did you know when you had product market fit?
So at Reddit, this is the nice scenario, we were growing.
Despite not knowing anything about our business,
knowing little about our users, not really knowing how to run a business,
not having a ton of product sense, being down a lot.
We were growing.
So that's the nice scenario, that's a good one.
At Hipmunk, our natural state was zero users.
I'm not sure we ever got product market fit.
The users who came to Hipmunk liked it, but
we had to fight for every single one of them.
It's possible to do that, but we were, again, I think that entire company,
we were swimming upstream.
And I remember the day we kind of realized that.
We ran the survey of like, why do you use Hipmunk?
And we put a lot of effort into the product,
we thought it was a really good product.
Legitimately you can find a flight a lot faster on us than our competitors.
35% of people said it's because of the logo, and
50% of the people said because we had lower prices, which we didn't.
>> [LAUGH] >> So, [LAUGH] I mean it's like,
we made it.
The company ran for five years.
And I ended up selling, which is fine, but yeah.
I think we were really fighting that one.
>> I think this is a huge misconception, by the way.
Most of the companies that apply to YC don't have product market fit.
Most of the companies on demo day don't have product market fit.
Most companies die before product market fit.
A lot of people talk about product market fit as this organic step in the process.
You raise angel, then you get an office, and
you hire some people, and you have product market fit.
And it's just like, you would be surprised at how few companies.
Product market fit is typically described as, you are overwhelmed with demand for
your product.
>> It's really obvious when you've hit product market fit.
So your product is bad, you can point at seven things that are broken about it.
And you have all these features you know you need to build.
And weirdly, it's growing really fast anyways.
It's really obvious when you have product market fit,
your product is growing like crazy.
And you are spending, you don't get to fix any of the things that are broken or
bad about your product,
because you're spending all of your time fixing it as it scales.
And you are, so the metaphor I use for startups is kind of Sisyphean,
you're pushing this boulder up a hill.
And you're pushing the boulder and
you're pushing the boulder, you're pushing the boulder.
Product market fit is when you crest the hill.
And your job doesn't get easier, but it does change a lot, because now you
are chasing the boulder as it tries to roll away from you down the other side.
And now you're springing flat out, trying to catch up, which is hard in its own way.
But it feels totally different, because as whereas before you have this, God,
where every inch is like, as Steve was describing with Hipmunk,
every inch is you're pushing this boulder and you're fighting for every inch.
Now it's like, crap, if we don't keep up with this, it's going to run away from us.
And that's a totally different feeling, and you'll know.
>> And here's the problem.
Eventually you'll catch the boulder on the way down, and then you're like ****,
I'm actually pushing it uphill again.
>> [LAUGH] Yep, absolutely.
>> [LAUGH] >> That's what happens.
>> [LAUGH] >> And I guess it's like asking,
how do you know if you're in love?
Trust me, you know.
>> It's funny, because I think one of the major problems that YC companies have is,
how do they stay small and lean and don't spend a lot of money.
Don't have a management overhead,
don't have a lot of bills, until they reach product market.fit.
One of the typical mistakes is confusing an angel round or
a couple million dollars in the bank for product market fit.
It's extremely common.
All right, other questions?
>> How do you recognize your first group of user?
>> How do you recognize your first groups of users?
>> Reddit was easy, they were just like Alexis and I.
Emmet was kind of talking about the intuition phase of building products.
We were in that for a long time.
Our users, they read the same Paul Graham blog that we did.
They were programmers like I was, so it was really, really easy.
Hipmunk, we were actually completely wrong about who our users were.
And I don't know if we ever quite solved this problem.
They solved it through acquisition.
Which is, we built a product for ourselves, like tech savvy
people who travel a fair amount and buy their own travel.
But in reality, the only way to make money in travel is to sell to business
travelers, because they're the only one that travels multiple times a year.
And business travelers don't get use things like Hipmunk or Kayak, right?
They use American Express and Concur.
So it took us actually a couple years to synthesize that, and
we were just like, ****.
>> [LAUGH] >> I think there's two kinds of startups.
There are startups that are building things for themselves,
where your customer is easy to identify.
It's you and people like you.
And there are startups building things for third party group users.
Where you should arrive at which customers are your customers through
an analytic mode, right?
Through like thinking about the problem.
So Twitch was an analytic start up,
because I was a huge fan of watching game streaming.
But actually I wasn't much of a streamer, I've never been much of a streamer.
And I basically dug in on Justin TV and realized that 80% of
the minutes watched were coming from our top 200 streamers.
And I was like they're the most important people.
We lose them, we lose everything, and if we can get 200 more of them,
we'll be twice as big.
And so it was sort of identified analytically that
that was the right group of users, and those are the right customers.
And I think that you should just know which one you are.
And the most important thing there is, if it's not for you, if you're not
legitimately building something for you and for you to use, and you aren't a big
group of people in this case, you should absolutely use the analytic mode.
I don't know, it's different for every single company.
But think hard about it, I don't know.
>> But if it's not for you, your intuition is lying to you.
>> Yeah, yes, yes.
>> Every time, and you have to supplement with data or talking to users or whatever.
>> So shockingly, we were 100% on time.
If everyone can thank these guys for coming out today, appreciate it.
>> [APPLAUSE]