everyone.

My name is David Malan,

and I'm so sorry I couldn't be there in person with you this week,

but I'm here to present on behalf of myself,

Brian Yu and Doug Lloyd from CS50 at Harvard.

If at any point during today's talk you have any questions or after the talk,

please feel free to drop me a note.

I'll keep an eye on my email throughout the day.

And if you'd like to get a copy of the paper in question,

head over now or later,

to cs50.ly/sigcse20-paper or for

these very slides,

you can go to cs50.ly/sigcse20-slides

So,

without further ado,

CS50 is our int roductory course in computer science for majors and non-majors at Harvard.

It's a one semester amalgam,

the course is generally known elsewhere,

as CS1 and CS2

We teach it primarily in C,

followed by Python.

And as of the past few years CS50 happens to be Harvard's largest course,

with over 800 or so students on campus as well as online through Harvard's Extension

School, 2/3 of whom have never taken a computer science course before.

The workload itself is nontrivial,

with most students spending 12+ hours per week on the course's

problem sets, or programming projects.

And in terms of the course's support structure:

We have lectures once per week,

followed by sections or recitations led by the course's teaching fellows,

or TFs,

as well as tutorials and office hours held by all of the staff,

which are one-on-one opportunities for help with those same problem sets.

Now, within the course itself,

we have different tracks,

so to speak,

for those less comfortable,

those more comfortable, and those somewhere in between,

whereby different demographics of students convened for different sections so that they are

among like minded and similarly experienced or inexperienced classmates.

And even within the courses problem sets do we tend to have different versions of problems for those less comfortable

or more comfortable that students can choose between week by week,

depending on the ceiling or floor that they would like to have ahead of them.

So the course,

in terms of academic honesty,

has a long standing and very detailed policy on academic honesty in the syllabus that

defines what is reasonable and what is not.

In fact,

if you'd like to look at the full detail feel free to forge ahead to

cs50.ly/sigcse20-syllabus

But in essence,

the syllabus allows and encourages students to collaborate,

particularly in pseudocode and when it comes to the thought before their programs.

But it does prescribe guard rails on what collaboration is ultimately allowed,

insofar as ultimately we prescribed

for all of the course's

problem sets that students implement their programs largely on their own and

write their own code.

An exception to this is the course's final project at term's end, where they're free to collaborate with

one or two classmates.

But in essence,

within the syllabus,

we provide this heuristic that generally guides these guard rails for students.

"When asking for help.

You may show your code to others,

but you may not view.

theirs."

In other words,

if two students, A and B happened to be sitting next to each other at office hours or in their dorms or the like,

and one of those students is struggling,

it's fine for the other student who isn't struggling to look at the struggling student's screen and perhaps

point out rhetorically or,

more explicitly,

what lines of code, or bugs,

or syntax they might want to focus on.

But it would be crossing the line if the student who's struggling outright looks at the code of another

student,

but again see the syllabus for more precise guard rails as well.

Even so,

despite these constraints,

in the course of syllabus,

we have the unfortunate distinction in CS50 of referring more students most every year to the

university's honor council or disciplinary process as a result of students having

crossed some line prescribed in the syllabus.

How we know this is that we, like a lot of universities or intro courses,

do run our student submissions through software be it e-tector or MOSS or some

other such tool with which you might be familiar.

And it outputs as part of that process any number of similarities that we then apply human

eyes to.

So just for context in fall 2019 most recently among our 800 or so

students,

and our 10 or so problem sets did we have some 6 million+ pairwise comparisons

conducted by this automated software,

out of which might come 1200 seemingly worrisome matches side by side,

two submissions that look unduly the same,

at which point we have a gauntlet of humans,

either two or three pairs of eyes that then iteratively review those matches and whittle it down to a more

manageable list.

For instance,

we ended up with 116 matches after initial human review,

and then myself and the course's more senior staff

each week sit down and review those matches and decide

collectively whether or not to send a case forward to Harvard's Honor Council or not,

in this most recent fall semester did we refer 38 students to the Honor Council,

at which point we document the case,

we write up the similarities,

provides the samples of those submissions,

and then it's an independent body,

this Honor Council that ultimately adjudicates the outcome.

Now,

in terms of what these cases involve,

it generally involves students having based their code on someone else's,

whether that's someone else's on campus or perhaps some online solutions.

In terms of the numbers each semester,

it's highly variable over the years

but it generally ranges between 0 and 10% of the class that we ultimately

refer to Harvard's disciplinary body.

From conversations with peer institutions.

we gather,

unfortunately,

that the 0 to 10% is within range of other courses, as well.

But it's worth noting in our case it seems to be trending upwards,

and that's in part because of some of the interventions we'll present here today.

But even so,

it's been of interest to us,

certainly over the years and to the university to reduce the frequency of these acts

of dishonesty themselves,

students copying unduly,

some online or some on-campus source.

So this paper and in turn,

today's talk is really a look at everything we tried over the years

and the spoiler,

unfortunately,

is that nothing really has worked,

at least more precisely,

and nothing has worked in our case that has put a consistent downward pressure on the

total number of cases.

But even so,

we do think a number of the interventions we've tried have been successful educationally,

ultimately transforming what has historically been purely punitive processes into more

teachable moments.

So what have we considered and

what have we tried? Well,

it's often suggested here on campus.

And perhaps beyond, that

we simply change.

The course is problem sets each year the programming assignments.

Unfortunately,

at least in our case,

the homework assignments tend not to be,

tend not to be as simple as problems that allow you to change the inputs and

expect different outputs.

They do tend to be programming projects unto themselves,

so changing the problem sets would involve rewriting and coming up with brand new problem sets.

And at least in my case,

I actually find that over time our problem sets get better over time.

As I work out the kinks,

fix bugs address FAQs within the specifications themselves.

And so we feel it would actually be a net negative for the sake of that smaller demographic of students,

simply to throw out problems that seem to be challenging a majority of students and working well overall.

So we've generally shied away from changing problem sets for changing problem set's sake.

Though they certainly do evolve by choice on our part.

over time.

It's often been suggested too why don't we just re-weight the problem sets? Make them worth much less

in terms of a student's final grade to put downward pressure.

Therefore,

on the

the pressure they might be putting on themselves and also the effect that plagiarized work might

have on a student's overall quantitative performance.

Here,

too,

we feel that that would be a disservice to the sheer amount of time that students are spending on these

programming assignments.

Indeed,

it is largely the problem sets that characterized students learning and hands on experience throughout

the class.

And so I've never been comfortable.

For instance,

with the idea of instituting proctored

exams or weighting the exams in the class much more heavily than that which the students are spending most of

their time on in the first place.

But we have tried communication each year.

Have we spend more and more,

sometimes less,

but in general,

more and more time on communicating to students exactly what our expectations are in terms of

academic honesty

Reviewing past cases that have gotten some of their prior classmates in trouble

albeit anonymously,

and sharing outright examples of code that we adjudicated to be

too similar or indeed identical in some cases,

and we even go so far as to point out examples of the kinds of things we via software are looking for.

Variables whose names are somewhat transposed but are still in essence serving the same function.

And certainly for multiple lines of code as well do we point out similarities there as well.

We've also focused on raising awareness.

Some years ago,

did we have the,

the university's dean of undergraduate education even come to one of our orientation meetings

at term's start to speak to students not only about academic honesty more generally,

but also about the course's

own policies and his own experience.

And his colleagues' own experience with,

unfortunately,

the disciplinary process that all too often results at the end of the term.

Also,

by nature of our having referred so many students over the years to this Honor Council

has there generally been,

in some years,

more than others,

just more awareness on campus,

of classmates who have been disciplined in some way.

And so we find that in parts, are numbers trending up and down

depending on the year as a reaction to the past year's numbers of cases and students awareness

thereof.

But we've also taken more software based approach is so built into,

for instance,

the courses software via which students submit

their work nowadays, is an explicit prompt,

to which they must answer yes or conversely no,

We asked them explicitly:

"Keeping in mind the course's policy on academic honesty,

are you sure you want to submit these files?"

The presumption being that even if a student has unfortunately crossed some line or made some poor

judgment,

they have this final chance before they submit their work and put their name on that work to say "No,

this is not,

in fact,

my own."

Unfortunately,

all too often have students still is typed in "yes",

and submitted their work despite this final prompt here.

But we've also added other prompts over the years to the courses

homework assignments via forms that the students submit a la

Google Forms whereby we have an opportunity to ask them not only short answer

code-unrelated questions,

but also checkboxes and the like that we require that they check we,