Original subtitles

♪ ♪

SARAH BRAYNE: We live in this era where

we leave digital traces throughout the course

of our everyday lives.

ANDY CLARNO: What is this data,

how is it collected, how is it being used?

NARRATOR: One way it's being used is to make predictions

about who might commit a crime...

Hey, give me all your money, man!

NARRATOR: ...and who should get bail.

JUDGE: On count one, you're charged with felony intimidation...

ANDREW FERGUSON: The idea is that if you look at past crimes,

you might be able to predict the future.

WILLIAM ISAAC: We want safer communities,

we want societies that are less incarcerated.

NARRATOR: But is that what we're getting?

Are the predictions reliable?

CATHY O'NEIL: I think algorithms can,

in many cases, be better than people.

But, of course, algorithms don't have consciousness.

The algorithm only knows what it's been fed.

RUHA BENJAMIN: Because it's technology,

we don't question them as much as we might

a racist judge or a racist officer.

They're behind this veneer of neutrality.

ISAAC: We need to know who's accountable

when systems harm the communities

that they're designed to serve.

NARRATOR: Can we trust the justice of predictive algorithms?

And should we?

"Computers Vs. Crime,"

right now, on "NOVA."

(computers booting up)

♪ ♪

NARRATOR: We live in a world of big data,

where computers look for patterns

in vast collections of information

in order to predict the future.

And we depend on their accuracy.

Is it a good morning for jogging?

Will this become cancer?

What movie should I choose?

The best way to beat traffic?

Your computer can tell you.

Similar computer programs, called predictive algorithms,

are mining big data to make predictions

about crime and punishment--

reinventing how our criminal legal system works.

Policing agencies have used these computer algorithms

in an effort to predict where the next crime will occur

and even who the perpetrator will be.

ASSISTANT DISTRICT ATTORNEY: Here, the state is recommending...

NARRATOR: Judges use them

to determine who should get bail

and who shouldn't.

JUDGE: If you fail to appear next time, you get no bond.

NARRATOR: It may sound like the police of the future

in the movie "Minority Report." I'm placing you under arrest

for the future murder of Sarah Marks.

NARRATOR: But fiction it's not.

How do these predictions actually work?

Can computer algorithms

make our criminal legal system more equitable?

Are these algorithms truly fair and free of human bias?

ANDREW PAPACHRISTOS: I grew up in Chicago

in the 1980s and early 1990s.

♪ ♪

My dad was an immigrant from Greece,

we worked in my family's restaurant,

called KaMar's.

NARRATOR: Andrew Papachristos was a 16-year-old kid

in the North Side of Chicago in the 1990s.

I spent a lot of my formative years busing tables,

serving people hamburgers and gyros.

It kind of was a whole family affair.

NARRATOR: Young Papachristos was aware the streets could be dangerous,

but never imagined the violence would touch him or his family.

REPORTER: Two more gang-related murders Monday night.

PAPACHRISTOS: And of course, you know, the '80s and '90s in Chicago

was some of the historically most violent periods in Chicago.

Street corner drug markets, street organizations.

And then like a lot of other businesses on our, on our block

and in our neighborhood,

local gangs tried to extort my family and the business.

And my dad had been running KaMar's for 30 years

and kind of just said no.

♪ ♪

(sirens blaring)

NARRATOR: Then, one night, the family restaurant burned to the ground.

Police suspected arson.

PAPACHRISTOS: It was quite a shock to our family,

'cause everybody in the neighborhood worked

in the restaurant at one point in their life.

And my parents lost 30 years of their lives.

That was really one of the events that made me want to

understand violence.

Like, how could this happen?

♪ ♪

NARRATOR: About a decade later,

Papachristos was a graduate student searching for answers.

PAPACHRISTOS: In graduate school,

I was working on a violence prevention program

that brought together community members,

including street outreach workers.

And we were sitting at a table,

and one of these outreach workers asked me,

the university student,

"Who's next?

Who's going to get shot next?"

And where that led was me sitting down

with stacks of shooting and, and homicide files

with a red pen and a legal pad,

by hand creating these network images

of, this person shot this person,

and this person was involved with this group and this event,

and creating a web of these relationships.

And then I learned that

there's this whole science about networks.

I didn't have to invent anything.

♪ ♪

NARRATOR: Social network analysis was already influencing

popular culture.

"Six Degrees of Separation" was a play on Broadway.

And then, there was Six Degrees of Kevin Bacon.

PAPACHRISTOS: The idea was, you would play this game,

and whoever got the shortest distance to Kevin Bacon

would win.

So Robert De Niro was in a movie with so-and-so,

who was in a movie with Kevin Bacon.

It was creating, essentially, a series of ties

among movies and actors.

And in fact, there's a mathematics

behind that principle.

It's actually old mathematical graph theory, right?

That goes back to 1900s mathematics.

And lots of scientists started seeing that there were

mathematical principles,

and computational resources-- computers, data--

were at a point that you could test those things.

So it was in a very exciting time.

We looked at arrest records, at police stops,

and we looked at victimization records.

Who was the victim of a homicide

or a non-fatal shooting?

♪ ♪

The statistical model starts by creating the social networks of,

say, everybody who may have been arrested

in a, in a particular neighborhood.

So Person A and Person B were in a robbery together,

they have a tie, and then Person B and Person C

were, were stopped by the police in another instance.

And it creates networks of thousands of people.

Understanding that events are connected,

places are connected.

That there are old things, like disputes between crews,

which actually drive behavior for generations.

What we saw was striking.

(snaps): And you could see it immediately,

and you could see it a mile away.

Which was, gunshot victims clumped together.

You, you very rarely see one victim.

You see two, three, four.

Sometimes they string across time and space.

And then the model predicts, what's the probability

that this is going to lead to a shooting

on the same pathway in the future?

(gun firing, people shouting)

REPORTER: Another young man lies dead.

NARRATOR: In Boston, Papachristos found

that 85% of all gunshot injuries

occurred within a single social network.

Individuals in this network

were less than five handshakes away

from the victim of a gun homicide

or non-fatal shooting.

The closer a person was

connected to a gunshot victim, he found,

the greater the probability that that person would be shot.

Around 2011, when Papachristos was presenting

his groundbreaking work on social networks

and gang violence,

the Chicago Police Department wanted to know more.

PAPACHRISTOS: We were at a conference.

The then-superintendent of the police department,

he was asking me a bunch of questions.

He had clearly read the paper.

NARRATOR: The Chicago Police Department

was working on its own predictive policing program

to fight crime.

They were convinced that Papachristos's model

could make their new policing model even more effective.

could make their new policing model even more effective.

LOGAN KOEPKE: Predictive policing involves

looking to historical crime data to predict future events,

either where police believe crime may occur

or who might be involved in certain crimes.

♪ ♪

So it's the use of historical data to forecast a future event.

NARRATOR: At the core of these programs is software,

which, like all computer programs,

is built around an algorithm.

So, think of an algorithm like a recipe.

♪ ♪

You have inputs,

which are your ingredients, you have the algorithm,

which is the steps.

♪ ♪

And then there's the output,

which is hopefully the delicious cake you're making.

GROUP: Happy birthday!

So one way to think about algorithms is to think about

the hiring process.

In fact, recruiters have been studied for a hundred years.

And it turns out many human recruiters

have a standard algorithm

when they're looking at a risumi.

So they start with your name, and then they look to see

where you went to school, and then finally,

they look at what your last job was.

If they don't see the pattern they're looking for...

(bell dings) ...that's all the time you get.

And in a sense, that's exactly what artificial intelligence

is doing, as well, in a very basic level.

It's recognizing sets of patterns and using that

to decide what the next step in its decision process would be.

♪ ♪

NARRATOR: What is commonly referred to

as artificial intelligence, or A.I.,

is a process called machine learning,

where a computer algorithm will adjust on its own,

without human instructions,

in response to the patterns it finds in the data.

These powerful processes can analyze more data

than any person can,

and find patterns never recognized before.

The principles for machine learning

were invented in the 1950s,

but began proliferating only after about 2010.

What we consider machine learning today

came about because hard drives became very cheap.

So it was really easy to get a lot of data on everyone

in every aspect of life.

And the question is, what can we do with all of that data?

Those new uses are things like predictive policing,

they are things like deciding whether or not a person's

going to get a job or not,

or be invited for a job interview.

NARRATOR: So how does such a powerful tool like machine learning work?

Take the case of a hiring algorithm.

First, a computer needs to understand the objective.

Here, the objective is identifying

the best candidate for the job.

The algorithm looks at risumis of former job candidates

and searches for keywords in risumis of successful hires.

The risumis are what's called training data.

The algorithm assigns values to each keyword.

Words that appear more frequently in the risumis

of successful candidates are given more value.

The system learns from past risumis the patterns

of qualities that are associated with successful hires.

Then it makes its predictions by identifying these

same patterns from the risumis of potential candidates.

♪ ♪

In a similar way,

the Chicago police wanted to find patterns in crime reports

and arrest records to predict who would be connected

to violence in the future.

They thought Papachristos's model could help.

Obviously we wanted to, and tried, and framed and wrote

all the caveats and made our recommendations to say,

"This research should be in this public health space."

But once the math is out there,

once the statistics are out there,

people can also take it and do what they want with it.

NARRATOR: While Papachristos saw the model as a tool to identify

future victims of gun violence,

CPD saw the chance to identify not only future victims,

but future criminals.

First it took me, you know, by, by surprise,

and then it got me worried.

What is it gonna do?

Who is it gonna harm?

♪ ♪

NARRATOR: What the police wanted to predict was who was at risk

for being involved in future violence.

Gimme all your money, man.

NARRATOR: Training on hundreds of thousands of arrest records,

the computer algorithm looks for patterns or factors

associated with violent crime

to calculate the risk that an individual

will be connected to future violence.

Using social network analysis,

arrest records of associates

are also included in that calculation.

The program was called the Strategic Subject List, or SSL.

It would be one of the most controversial

in Chicago policing history.

ANDY CLARNO: The idea behind the Strategic Subjects List,

or the SSL,

was to try to identify the people who would be

most likely to become involved as what they called

a "party to violence," either as a shooter or a victim.

♪ ♪

NARRATOR: Chicago police would use Papachristos's research

to evaluate what was called an individual's

"co-arrest network."

And the way that the Chicago Police Department

calculated an individual's network was through

kind of two degrees of removal.

Anybody that I'd been arrested with and anybody that they

would, had been arrested with counted as people who were

within my network.

So my risk score would be based on my individual history

of arrest and victimization, as well as the histories

of arrest and victimization of people within that

two-degree network of mine.

It was colloquially known as the "heat list."

If you were hot, you were on it.

And they gave you literally a risk score.

At one time, it was zero to 500-plus.

If you're 500-plus, you are a high-risk person.

♪ ♪

And if you made this heat list,

you might find a detective knocking on your front door.

♪ ♪

NARRATOR: Trying to predict

future criminal activity is not a new idea.

Scotland Yard in London began using this approach

by mapping crime events in the 1930s.

But in the 1990s,

it was New York City Police Commissioner William Bratton

who took crime mapping to another level.

BRATTON: I run the New York City Police Department.

My competition is the criminal element.

NARRATOR: Bratton convinced policing agencies across the country

that data-driven policing was the key

to successful policing strategies.

Part of this is to prevent crime in the first place.

♪ ♪

NARRATOR: Bratton was inspired

by the work of his own New York City Transit Police.

As you see all those,

uh, dots on the map,

that's our opponents.

NARRATOR: It was called Charts of the Future,

and credited with cutting subway felonies by 27%

and robberies by a third.

Bratton saw potential.

He ordered all New York City precincts

to systematically map crime,

collect data, find patterns, report back.

The new approach was called CompStat.

BRAYNE: CompStat, I think, in a way, is kind of a precessor

of predictive policing,

in the sense that many of the same principles there--

you know, using data tracking, year-to-dates,

identifying places where law enforcement interventions

could be effective, et cetera--

really laid the groundwork for predictive policing.

♪ ♪

NARRATOR: By the early 2000s,

as computational power increased,

criminologists were convinced this new data trove

could be used in machine learning to create models

that predict when and where

crime would happen in the future.

♪ ♪

REPORTER: L.A. police now say the gunmen opened fire

with a semi-automatic weapon.

NARRATOR: In 2008,

now as chief of the Los Angeles Police Department,

Bratton joined with academics at U.C.L.A.

to help launch a predictive policing system

called PredPol,

powered by a machine learning algorithm.

♪ ♪

ISAAC: PredPol started

as a spin-off of a set of, like,

government contracts that were related to military work.

They were developing

a form of an algorithm that was used to predict I.E.Ds.

(device explodes)

And it was a technique that was used

to also detect aftershocks and seismographic activity.

(dogs barking and whining, objects clattering)

And after those contracts ended,

the company decided they wanted to apply this

in the domain of, of policing

domestically in the United States.

(radio beeping)

NARRATOR: The PredPol model

relies on three types of historical data:

type of crime, crime location, and time of crime,

going back two to five years.

The algorithm is looking for patterns

to identify locations where crime is most likely to occur.

As new crime incidents are reported,

they get folded into the calculation.

The predictions are displayed on a map

as 500 x 500 foot areas that officers are then

directed to patrol.

ISAAC: And then from there, the algorithm says,

"Okay, based on what we know about the kind of

"very recent history,

"where is likely that we'll see crime

in the next day or the next hour?"

♪ ♪

BRAYNE: One of the key reasons

that police start using these tools

is the efficient and even, to a certain extent,

like in their logic,

more fair, um, and, and justifiable allocation

of their police resources.

♪ ♪

NARRATOR: By 2013,

in addition to PredPol, predictive policing systems

developed by companies like HunchLab, IBM, and Palantir

were in use across the country.

(radios running in background)

And computer algorithms

were also being adopted in courtrooms.

BAILIFF: 21CF3810, State of Wisconsin versus Chantille...

KATHERINE FORREST: These tools are used in pretrial determinations,

they're used in sentencing determinations,

and they're used in housing determinations.

They're also used, importantly, in the plea bargaining phase.

They're used really throughout the entire process

to try to do what judges have been doing,

which is the very, very difficult task

of trying to understand and predict

what will a human being do tomorrow, or the next day,

or next month, or three years from now.

ASSISTANT DISTRICT ATTORNEY: Bail forfeited.

He failed to appear 12/13/21.

Didn't even make it to preliminary hearing.

The software tools are an attempt to try to predict

it better than humans can.

MICHELLE HAVAS: On count one, you're charged with

felony intimidation of a victim.

SWEENEY: So, in the United States, you're innocent

until you've been proven guilty, but you've been arrested.

Now that you've been arrested,

a judge has to decide whether or not

you get out on bail,

or how high or low that bail should be.

You're charged with driving on a suspended license.

I've set that bond at $1,000.

No insurance, I've set that bond at $1,000.

ALISON SHAMES: One of the problems is,

judges often are relying on money bond

or financial conditions of release.

JUDGE: So I'm going to lower his fine

to make it a bit more reasonable.

So instead of $250,000 cash,

surety is $100,000.

SHAMES: It allows people who have access to money to be released.

If you are poor, you are often being detained pretrial.

Approximately 70% of the people in jail are there on pretrial.

These are people who are presumed innocent,

but are detained during the pretrial stage of their case.

NARRATOR: Many jurisdictions use pretrial assessment algorithms

with a goal to reduce jail populations and decrease

the impact of judicial bias.

SHAMES: The use of a tool like this takes historical data

and assesses, based on research,

associates factors that are predictive of the two outcomes

that the judge is concerned with.

That's community safety and whether that person

will appear back in court during the pretrial period.

♪ ♪

NARRATOR: Many of these algorithms are based on a concept called

a regression model.

The earliest, called linear regression,

dates back to 19th-century mathematics.

O'NEIL: At the end of the day, machine learning algorithms

do exactly what linear regression does,

which is predict--

based on the initial conditions, the situation they're seeing--

predict what will happen in the future,

whether that's, like, in the next one minute

or the next four years.

NARRATOR: Throughout the United States, over 60 jurisdictions

use predictive algorithms as part of the legal process.

One of the most widely used is COMPAS.

The COMPAS algorithm weighs factors,

including a defendant's answers to a questionnaire,

to provide a risk assessment score.

These scores are used every day by judges to guide decisions

about pretrial detention, bail, and even sentencing.

But the reliability of the COMPAS algorithm

has been questioned.

In 2016, ProPublica published an investigative report

on the COMPAS risk assessment tool.

BENJAMIN: Investigators wanted to see if the scores were accurate

in predicting whether these individuals

would commit a future crime.

And they found two things that were interesting.

One was that the score was remarkably unreliable

in predicting who would commit a, a crime in the future

over this two-year period.

But then the other thing that ProPublica investigators found

was that Black people were much more likely to be deemed

high risk and white people low risk.

NARRATOR: This was true even in cases when the Black person

was arrested for a minor offense and the white person

in question was arrested for a more serious crime.

BENJAMIN: This ProPublica study was one of the first to begin

to burst the bubble of technology

as somehow objective and neutral.

NARRATOR: The article created a national controversy.

But at Dartmouth, a student convinced her professor

they should both be more than stunned.

HANY FARID: As it turns out, one of my students, Julia Dressel,

reads the same article and said,

"This is terrible.

We should do something about it." (chuckles)

This is the difference between an awesome idealistic student

and a jaded, uh, professor.

And I thought, "I think you're right."

And as we were sort of struggling to understand

the underlying roots of the bias in the algorithms,

we asked ourselves a really simple question:

are the algorithms today, are they doing better than humans?

Because presumably, that's why you have these algorithms,

is that they eliminate some of the bias and the prejudices,

either implicit or explicit, in the human judgment.

NARRATOR: To analyze COMPAS's risk assessment accuracy,

they used the crowdsourcing platform Mechanical Turk.

Their online study included 400 participants

who evaluated 1,000 defendants.

FARID: We asked participants to

read a very short paragraph about an actual defendant.

How old they were,

whether they were male or female,

what their prior juvenile conviction record was,

and their prior adult conviction record.

And, importantly, we didn't tell people their race.

And then we ask a very simple question,

"Do you think this person will commit a crime

in the next two years?"

Yes, no.

And again, these are non-experts.

These are people being paid

a couple of bucks online to answer a survey.

No criminal justice experience,

don't know anything about the defendants.

They were as accurate as the commercial software

being used in the courts today,

one particular piece of software.

That was really surprising.

We would've expected a little bit of improvement.

After all, the algorithm has access

to huge amounts of training data.

NARRATOR: And something else puzzled the researchers.

The MTurk workers' answers to questions

about who would commit crimes in the future and who wouldn't

showed a surprising pattern of racial bias,

even though race wasn't indicated

in any of the profiles.

They were more likely to say a person of color

will be high risk when they weren't,

and they were more likely to say that a white person

would not be high risk when in fact they were.

And this made no sense to us at all.

You don't know the race of the person.

How is it possible that you're biased against them?

(radios running in background)

In this country, if you are a person of color,

you are significantly more likely, historically,

to be arrested,

to be charged, and to be convicted of a crime.

So in fact, prior convictions

is a proxy for your race.

Not a perfect proxy, but it is correlated,

because of the historical inequities

in the criminal justice system

and policing in this country.

(siren blaring)

MAN: It's my car, bro, come on, what are y'all doing?

Like, this, this is racial profiling.

NARRATOR: Research indicates a Black person

is five times more likely to be stopped without cause

than a white person.

Black people are at least twice as likely

as white people to be arrested for drug offenses,

even though Black and white people

use drugs at the same rate.

Black people are also about 12 times

more likely to be wrongly convicted of drug crimes.

FORREST: Historically, Black men have been arrested at higher levels

than other populations.

Therefore, the tool predicts that a Black man, for instance,

will be arrested at a rate and recidivate at a rate

that is higher than a white individual.

FARID: And so what was happening is, you know, the big data,

the big machine learning folks are saying,

"Look, we're not giving it race-- it can't be racist."

But that is spectacularly naive,

because we know that other things correlate with race.

In this case, number of prior convictions.

And so when you train an algorithm on historical data,

well, guess what.

It's going to reproduce history-- of course it will.

NARRATOR: Compounding the problem is the fact that

predictive algorithms can't be put on the witness stand

and interrogated about their decision-making processes.

FORREST: Many defendants have had difficulty

getting access to the underlying information

that tells them,

what was the data set that was used to assess me?

What were the inputs that were used?

How were those inputs weighted?

So you've got what can be, these days,

increasingly, a black box.

A lack of transparency.

NARRATOR: Some black box algorithms get their name

from a lack of transparency about the code

and data inputs they use,

which can be deemed proprietary.

But that's not the only kind of black box.

A black box is any system which is so complicated

that you can see what goes in and you can see what comes out,

but it's impossible to understand

what's going on inside it.

All of those steps in the algorithm

are hidden inside phenomenally complex math

and processes.

FARID: And I would argue that when you are using algorithms

in mission-critical applications,

like criminal justice system,

we should not be deploying black box algorithms.

NARRATOR: PredPol, like many predictive platforms,

claimed a proven record for crime reduction.

In 2015, PredPol published its algorithm

In 2015, PredPol published its algorithm

in a peer-reviewed journal.

William Isaac and Kristian Lum,

research scientists who investigate

predictive policing platforms,

analyzed the algorithm.

ISAAC: We just kind of saw the algorithm

as going back to the same one or two blocks

every single time.

And that's kind of strange,

because if you had a truly predictive policing system,

you wouldn't necessarily see it going to the same locations

over and over again.

NARRATOR: For their experiment,

Isaac and Lum used a different data set,

public health data, to map illicit drug use in Oakland.

ISAAC: So, a good chunk of the city was kind of

evenly distributed in terms of where

potential illicit drug use might be.

But the police predictions were clustering around areas

where police had, you know,

historically found incidents of illicit drug use.

Specifically, we saw significant numbers of neighborhoods

that were predominantly non-white and lower-income

being deliberate targets of the predictions.

NARRATOR: Even though illicit drug use was a citywide problem,

the algorithm focused its predictions

on low-income neighborhoods and communities of color.

ISAAC: The reason why is actually really important.

It's very hard to divorce

these predictions from those histories

and legacies of over-policing.

As a result of that, they manifest themselves in the data.

NARRATOR: In an area where there is more police presence,

more crime is uncovered.

The crime data indicates to the algorithm

that the heavily policed neighborhood

is where future crime will be found,

even though there may be other neighborhoods

where crimes are being committed at the same or higher rate.

ISAAC: Every new prediction that you generate

is going to be increasingly dependent

on the behavior of the algorithm in the past.

So, you know, if you go ten days, 20 days,

30 days into the future, right, after using an algorithm,

all of those predictions have changed the behavior

of the police department

and are now being folded back into the next day's prediction.

NARRATOR: The result can be a feedback loop

that reinforces historical policing practices.

SWEENEY: All of these different types

of machine learning algorithms are all trying to help us

figure out, are there some patterns in this data?

It's up to us to then figure out,

are those legitimate patterns, do they,

are they useful patterns?

Because the computer has no idea.

It didn't make a logical association.

It just made it, made a correlation.

MING: My favorite definition of artificial intelligence

is, it's any autonomous system

that can make decisions under uncertainty.

You can't make decisions under uncertainty without bias.

In fact, it's impossible to escape from having bias.

It's a mathematical reality

about any intelligent system, even us.

(siren blaring in distance)

NARRATOR: And even if the goal is to get rid of prejudice,

bias in the historical data can undermine that objective.

Amazon discovered this

when they began a search for top talent

with a hiring algorithm whose training data

depended on hiring successes from the past.

MING: Amazon, somewhat famously within the A.I. industry,

they tried to build a hiring algorithm.

They had a massive data set.

They had all the right answers,

because they knew literally who got hired

and who got that promotion in their first year.

(typing)

NARRATOR: The company created multiple models

to review past candidates' risumis

and identify some 50,000 key terms.

MING: What Amazon actually wanted to achieve

was to diversify their hiring.

Amazon, just like every other tech company,

and a lot of other companies, as well,

has enormous bias built into its hiring history.

It was always biased, strongly biased, in favor of men,

in favor, generally,

of white or sometimes Asian men.

Well, they went and built a hiring algorithm.

And sure enough, this thing was

the most sexist recruiter you could imagine.

If you said the word "women's" in your risumi,

then it wouldn't hire you.

If you went to a women's college,

it didn't want to hire you.

So they take out all the gender markers,

and all of the women's colleges--

all the things that explicitly says,

"This is a man," and, "This is a woman,"

or even the ones that, obviously, implicitly say it.

So they did that.

And then they trained up their new deep neural network

to decide who Amazon would hire.

And it did something amazing.

It did something no human could do.

It figured out who was a woman and it wouldn't hire them.

It was able to look through

all of the correlations that existed

in that massive data set

and figure out which ones most strongly correlated

with someone getting a promotion.

And the single biggest correlate

of getting a promotion was being a man.

And it figured those patterns out and didn't hire women.

NARRATOR: Amazon abandoned its hiring algorithm in 2017.

Remember the way machine learning works, right?

It's like a student who doesn't really understand

the material in the class.

They got a bunch of questions, they got a bunch of answers.

They got a bunch of questions, they got a bunch of answers.

And now they're trying to pattern match

for a new question and say, "Oh, wait.

"Let me find an answer that looks pretty much

like the questions and answers I saw before."

The algorithm only worked because someone has said,

"Oh, this person whose data you have,

"they were a good employee.

This other person was a bad employee," or, "This person

performed well," or, "This person did not perform well."

O'NEIL: Because algorithms don't just look for patterns,

they look for patterns of success, however it's defined.

But the definition of success is really critically important

to what that end up, ends up being.

And a lot of, a lot of opinion

is embedded in, what, what does success look like?

NARRATOR: In the case of algorithms,

human choices play a critical role.

O'NEIL: The data itself was curated.

Someone decided what data to collect.

Somebody decided what data was not relevant, right?

And they don't exclude it necessarily

intentionally-- they could be blind spots.

NARRATOR: The need to identify such oversights

becomes more urgent

as technology takes on more decision making.

♪ ♪

Consider facial recognition technology,

used by law enforcement

in cities around the world for surveillance.

In Detroit, 2018,

law enforcement looked to facial recognition technology

when $3,800 worth of watches

were stolen from an upscale boutique.

Police ran a still frame from the shop's surveillance video

through their facial recognition system to find a match.

How do I turn a face into numbers

that equations can act with?

You turn the individual pixels in the picture of that face

into values.

What it's really looking for are complex patterns

across those pixels.

The sequence of taking a pattern of numbers

and transforming it

into little edges and angles,

then transforming that into eyes and cheekbones and mustaches.

NARRATOR: To find that match,

the system can be trained on billions of photographs.

Facial recognition uses a class of machine learning

called deep learning.

The models built by deep learning techniques

are called neural networks.

VENKATASUBRAMANIAN: A neural network

is, you know, stylized as, you know, trying to model

how neural pathways work in the brain.

You can think of a neural network

as a collection of neurons.

So you put some values into a neuron,

and if they're, sufficiently, they add up to some number,

or they cross some threshold,

this one will fire and send off a new number to the next neuron.

NARRATOR: At a certain threshold,

the neuron will fire to the next neuron.

If it's below the threshold, the neuron doesn't fire.

This process repeats and repeats

across hundreds, possibly thousands of layers,

making connections like the neurons in our brain.

♪ ♪

The output is a predictive match.

Based on a facial recognition match,

in January 2020, the police arrested Robert Williams

for the theft of the watches.

The next day, he was released.

Not only did Williams have an alibi,

but it wasn't his face.

MING: To be very blunt about it, these algorithms are probably

dramatically over-trained on white faces.

♪ ♪

So, of course, algorithms that start out bad

can be improved, in general.

The Gender Shades project found that

certain facial recognition technology,

when they actually tested it on Black women,

it was 65% accurate, whereas for white men, it was 99% accurate.

How did they improve it? Because they did.

They built an algorithm

that was trained on more diverse data.

So I don't think it's completely a lost cause

to improve algorithms to be better.

MAN (in ad voiceover): I used to think my job was all about arrests.

NEDY: LESLIE KEN There was a commercial a few years ago

that showed a police officer going to a gas station

and then waiting for the criminal to show up.

MAN: We analyze crime data,

spot patterns,

and figure out where to send patrols.

They said, "Well, our algorithm will tell you exactly

where the crime, the next crime is going to take place."

Well, that's just silly, uh, it, that's not how it works.

MAN: By stopping it before it happens.

(sighs)

MAN: Let's build a smarter planet.

♪ ♪

JOEL CAPLAN: Understanding what it is about these places

that enable crime problems

to emerge and/or persist.

NARRATOR: At Rutgers University,

the researchers who invented

the crime mapping platform called Risk Terrain Modeling,

or RTM,

bristle at the term "predictive policing."

CAPLAN (voiceover): We don't want to predict, we want to prevent.

I worked as a police officer a long time ago,

in the early 2000s.

Police collected data for as long as police have existed.

Now there's a greater recognition

that data can have value.

But it's not just about the data.

It's about how you analyze it, how you use those results.

There's only two data sets that risk terrain modeling uses.

These data sets are local,

current information about crime incidents within a given area

and information about environmental features

that exist in that landscape,

such as bars, fast food restaurants,

convenience stores, schools, parks, alleyways.

KENNEDY: The algorithm is basically

the relationship between these environmental features

and the, the outcome data, which in this case is crime.

The algorithm provides you with a map

of the distribution of the risk values.

ALEJANDRO GIME NEZ-SANTANA: This is the highest-risk area,

on this commercial corridor on Bloomfield Avenue.

NARRATOR: But the algorithm isn't intended for use just by police.

Criminologist Alejandro Giminez-Santana

leads the Newark Public Safety Collaborative,

a collection of 40 community organizations.

They use RTM as a diagnostic tool

to understand not just where crime may happen next,

but why.

GIME NEZ-SANTANA: Through RTM, we identify this commercial corridor

on Bloomfield Avenue, which is where we are right now,

as a risky area for auto theft due to car idling.

So why is this space

particularly problematic when it comes to auto theft?

One is because we're in a commercial corridor,

where there's high density of people

who go to the beauty salon or to go to a restaurant.

Uber delivery and Uber Eats,

delivery people who come to grab orders that also,

and leave their cars running

create the conditions for this crime

to be concentrated in this particular area.

What the data showed us was,

there was a tremendous rise in auto vehicle thefts.

But we convinced the police department

to take a more social service approach.

NARRATOR: Community organizers convinced police

not to ticket idling cars,

and let organizers create

an effective public awareness poster campaign instead.

And we put it out to the Newark students

to submit in this flyer campaign,

and have their artwork on the actual flyer.

GIME NEZ-SANTANA: As you can see, this is the commercial corridor

on Bloomfield Avenue.

The site score shows a six,

which means that we are at the highest risk of auto theft

in this particular location.

And as I move closer to the end

of the commercial corridor, the site risk score is coming down.

NARRATOR: This is the first time in Newark

that police data for crime occurrences

have been shared widely with community members.

ELVIS PEREZ: The kind of data we share is incident-related data--

sort of time, location, that sort of information.

We don't discuss any private arrest information.

We're trying to avoid a crime.

NARRATOR: In 2019,

Caplan and Kennedy formed a start-up at Rutgers

to meet the rising demand for their technology.

Despite the many possible applications for RTM,

from tracking public health issues

to understanding vehicle crashes,

law enforcement continues to be its principal application.

Like any other technology,

risk terrain modeling can be used for the public good

when people use it wisely.

♪ ♪

We as academics and scientists, we actually need to be critical,

because it could be the best model in the world,

it could be very good predictions,

but how you use those predictions

matters, in some ways, even more.

REPORTER: The police department

had revised the SSL numerous times...

NARRATOR: In 2019,

Chicago's inspector general contracted the RAND Corporation

to evaluate the Strategic Subject List,

the predictive policing platform

that incorporated Papachristos's research on social networks.

PAPACHRISTOS: I never wanted to go down this path

of who was the person that was the potential suspect.

And that problem is not necessarily

with the statistical model, it's the fact that someone

took victim and made him an offender.

You've criminalized someone who is at risk,

that you should be prioritizing saving their life.

NARRATOR: It turned out that some 400,000 people were included on the SSL.

Of those, 77% were Black or Hispanic.

The inspector general's audit revealed

that SSL scores were unreliable.

The Rand Corporation found the program had no impact

on homicide or victimization rates.

(protesters chanting)

NARRATOR: The program was shut down.

But data collection continues

to be essential to law enforcement.

♪ ♪

O'NEIL: There are things about us that we might not even

be aware of that are sort of being collected

by the data brokers

and will be held against us for the rest of our lives--

held against people forever, digitally.

NARRATOR: Data is produced and collected.

But is it accurate?

And can the data be properly vetted?

PAPACHRISTOS: And that was one of the critiques

of not just the Strategic Subjects List,

but the gang database in Chicago.

Any data source that treats data as a stagnant, forever condition

is a problem.

WOMAN: The gang database has been around for four years.

It'll be five in January.

We want to get rid of surveillance

in Black and brown communities.

BENJAMIN: In places like Chicago,

in places like L.A., where I grew up,

there are gang databases with tens of thousands

of people listed, their names listed in these databases.

Just by simply having a certain name

and coming from a certain ZIP code

could land you in these databases.

Do you all feel safe in Chicago?

DARRELL DACRES: The cops pulled up out of nowhere.

Didn't ask any questions, just immediately start beating on us.

And basically were saying, like,

what are, what are we doing over here, you know, like,

in this, in this gangbang area?

I was already labeled as a gangbanger

from that area because of where I lived.

I, I just happened to live there.

NARRATOR: The Chicago gang database

is shared with hundreds of law enforcement agencies.

Even if someone is wrongly included,

there is no mechanism to have their name removed.

If you try to apply for an apartment,

or if you try to apply for a job or a college,

or even in a, um, a house, it will show

that you are in this record of a gang database.

I was arrested for peacefully protesting.

And they told me that, "Well,

you're in the gang database."

But I was never in no gang.

MAN: Because you have a gang designation,

you're a security threat group,

right?

NARRATOR: Researchers and activists

have been instrumental in dismantling

some of these systems.

And so we continue to push back.

I mean, the fight is not going to finish

until we get rid of the database.

♪ ♪

FERGUSON: I think what we're seeing now

is not a move away from data.

It's just a move away from this term "predictive policing."

But we're seeing big companies,

big tech, enter the policing space.

We're seeing the reality that almost all policing now

is data-driven.

You're seeing these same police departments

invest heavily in the technology,

including other forms of surveillance technology,

including other forms of databases

to sort of manage policing.

(chanting): We want you out!

NARRATOR: More citizens are calling for regulations

to audit algorithms

and guarantee they're accomplishing what they promise

without harm.

BRAYNE: Ironically, there is very little data

on police use of big data.

And there is no systematic data

at a national level on how these tools are used.

The deployment of these tools

so far outpaces

legal and regulatory responses to them.

What you have happening

is essentially this regulatory Wild West.

O'NEIL: And we're, like, "Well, it's an algorithm,

let's, let's just throw it into production."

Without testing it to whether

it "works" sufficiently, um, at all.

NARRATOR: Multiple requests for comment

from police agencies and law enforcement officials

in several cities, including Chicago and New York,

were either declined or went unanswered.

♪ ♪

Artificial intelligence must serve people,

and therefore artificial intelligence

must always comply with people's rights.

NARRATOR: The European Union is preparing to implement legislation

to regulate artificial intelligence.

In 2021, bills to regulate data science algorithms

were introduced in 17 states,

and enacted in Alabama, Colorado,

Illinois, and Mississippi.

SWEENEY: If you look carefully on electrical devices,

you'll see "U.L.," for Underwriters Laboratory.

That's a process that came about

so that things, when you plugged them in,

didn't blow up in your hand.

That's the same kind of idea

that we need in these algorithms.

O'NEIL: We can adjust it to make it better than the past,

and we can do it carefully,

and we can do it with, with precision

in an ongoing conversation about what it means to us

that it is, it's biased in the right way.

I don't think you remove bias,

but you get to a bias that you can live with,

that you, you think is moral.

To be clear, like, I, I think we can do better,

but often doing better

would look like we don't use this at all.

(radio running)

FARID: There's nothing fundamentally wrong

with trying to predict the future,

as long as you understand how the algorithms are working,

how are they being deployed.

What is the consequence of getting it right?

And most importantly is,

what is the consequence of getting it wrong?

OFFICER: Keep your hands on the steering wheel!

MAN: My hands haven't moved off the steering wheel!

MAN 2: Are you gonna arrest me?

MAN 1: Officer, what are we here for?

OFFICER: We just want to talk with...

♪ ♪

ANNOUNCER: This program is available with PBS Passport

and on Amazon Prime Video.

♪ ♪

♪ ♪