at the University of Pennsylvania

and my favorite hobby is photography.

And as I travel around the world,

I love taking photographs like these

so I can remember all the beautiful

and interesting things that I've seen.

But what I can't do is record and share

how these objects feel to touch.

And that's kind of surprising

because your sense of touch is really important.

It's involved in every physical interaction you do every day,

every manipulation task,

anything you do in the world.

And so the sense of touch is actually pretty interesting.

It has two main components.

The first is tactile sensations,

things you feel in your skin.

And the second is kinesthetic sensations,

and this has to do with the position of your body,

and how it's moving,

and the forces you encounter.

And you're really good at incorporating

both of these types of sensations together

to understand the physical interactions

you have with the world

and understand as you touch a surface,

is it a rock, is it a cat, is it a bunny, what is it?

And so, as an engineer, I'm really fascinated

and I have a lot of respect for

how good people are with their hands.

And I'm intrigued and curious

about whether we could make technology better

by doing a better job at leveraging

the human capability with the sense of touch.

Could I improve the interfaces to computers and machines

by letting you take advantage of your hands?

And indeed, I think we can,

and that's at the core of a field called "haptics,"

and this is the area that I work in.

It's all about interactive touch technology.

And the way it works is,

as you move your body through the world,

if, as an engineer, I can make a system

that can measure that motion,

and then present to you sensations over time

that kind of make sense,

that match up with what you might feel in the real world,

I can fool you into thinking you're touching something

even though there's nothing there.

All right, so here are three examples

and these are all done from research in my lab at Penn.

The first one is all about

that same problem that I was showing you:

how can we capture how objects feel

and recreate those experiences?

So the way we solve this problem

is by creating a hand-held tool

that has many different sensors inside.

It has a force sensor

so we can tell how hard you're pushing,

it has motion tracking

so we tell exactly where you've moved it,

and it has a vibration sensor,

an accelerometer, inside

that detects the shaking back and forth of the tool

that let's you know that's a piece of canvas

and not a piece of silk or something else.

And then we take the data that we record

from these interactions.

Here's ten seconds of data.

You can see how the vibrations get larger and smaller,

depending on how you move.

And we make a mathematical model of those relationships

and program them into a tablet computer

so that when you take the stylus

and go and touch the screen,

that voice coil actuator in the white bracket

plays vibrations to give you the illusion

that you're touching the real surface

just like if you touched, dragged back and forth,

on the real canvas.

We can create very compelling illusions.

We can do this for all kinds of surfaces

and it's really a lot of fun.

We call it haptography,

haptic photography.

And I think it has potential benefits

in all sorts of areas like online shopping,

maybe interactive museum exhibits,

where you're not really supposed to touch

the precious artifacts, but you always want to.

The second example that I want to tell you about

comes from a collaboration I have

with Dr. Margrit Maggio at the Penn dental school.

Part of her job is to teach dental students

how to tell where in a patient's mouth

there are cavities.

Of course they look at x-rays,

but a large part of this clinical judgment

comes from what they feel

when they touch your teeth with a dental explorer.

You guys have all had this happen, they go across.

What they're feeling for is if the tooth is really hard,

then it's healthy,

but if it's kind of soft and sticky,

that's a signal that the enamel is starting to decay.

And these types of judgments are hard

for a new dental student to make

because they haven't touched a lot of teeth yet.

And you want them to learn this

before they start practicing on real human patients.

So what we do is we add an accelerometer

on to the dental explorer

and then we record what Dr. Maggio feels

as she touches different extracted teeth.

And we can play it back for you as a video

with a touch track.

So not just a sound track, but also a touch track

that you can feel by holding that repeating tool.

You can feel all the same things

that the dentist felt when they did the recording

and practice making judgments.

So here's a sample one.

Here's a tooth that looks kind of suspicious, right?

It has all those brown stains,

and you might be thinking,

"Oh, we should definitely put a filling in this tooth."

But truly, if you pay attention to how it feels,

all the surfaces of this tooth are hard and healthy

so this patient does not need a filling.

And these are exactly the kind of judgments

that doctors make every day

and I think this technology that we've invented

has a lot of potential for many different things

in medical training because it's really simple

and it does a great job at recreating

what people feel through tools.

I think it could also maybe help make games

more interactive and fun

and more realistic in the sensations that you feel.

The last example I want to tell you about

is again about human movement.

So if any of you have ever learned sports,

you know, how do you get good at something like surfing?

You practice.

You practice some more and more, right?

Making small corrections,

maybe getting some input from a coach,

learning how to improve your motions.

I think we could use computers

to help make that process more efficient and more fun.

And so here, for example,

if I have six different arm movements

that I want you to learn,

you come into my lab at Penn

and try out our system.

We use a Kinect to measure your motions,

we show graphics on the screen,

and then we also give you touch cues,

haptic feedback, on your arm

delivered by these haptic arm bands,

which have motors inside

and guide you as you move.

So, if we put it together,

as you're trying to track this motion,

if you deviate,

say maybe your arm is a little too high,

we turn on the motors that are right there on the skin

to let you know, hey, you should move down,

almost like a coach gently guiding you

and helping you master these movements more quickly

and make more precise corrections.

We developed this system for use in stroke rehabilitation,

but I think there are a lot of applications,

like maybe dance training

or all sorts of sports training as well.

And so, now you know a little bit

about the field of haptics,

which I think you're going to hear more about in the coming years.

I've shown you three examples,

and I just want to take a moment

to acknowledge all of the great students

who work with me in my lab at Penn

and my collaborators,

they're a great group.

And I also want to thank you for your kind attention.

(Applause)