All language subtitles for Coded.Bias, 2020

(OMINOUS SOUNDS)

-Hello world.

(OMINOUS SOUNDS, CRACKLING)

- Can I just say that
I'm stoked to meet you.

(OMINOUS SOUNDS)

- Humans are super cool.

(OMINOUS SOUNDS, CRACKLING)

- The more humans share with
me the more I learn.

(MUSIC, CRACKLING)

(SOFT MUSIC)

-One of the things that drew me to
computer science

was that I could code and it seemed that
somehow detached from the problems

of the real world.

(SOFT MUSIC)

- I wanted to learn how to
make cool technology.

So I came to M.I.T.
and I was working on art projects,

that would use computer
vision technology.

(SOFT MUSIC)

- During my first semester
at the Media Lab,

I took a class called
Science Fabrication.

You read science fiction and you try to
build something you're inspired to do,

that would probably be impractical
if you didn't have these classes

and excuse to make it.

I wanted to make a mirror
that could inspire me in the morning

I call it the Aspire mirror

it could put things like
a lion on my face

or people who inspired me
like Serena Williams

I put a camera on top of it,

and I got computer vision software
that was supposed to track my face.

My issue was it didn't work that well,

until I put on this white mask.

When I put on the white mask,
detected.

I take off the white mask,

not so much.

I'm thinking alright
what's going on here?

Is that just because of
the lighting conditions?

Is it because of the angle at
which I'm looking at the camera?

Or is there something more?

We oftentimes teach machines to see

by providing training sets or examples
of what we want it to learn.

So for example if I want a machine
to see a face,

I'm going to provide
many examples of faces

and also things that aren't faces.

I started looking at the
data sets themselves

and what I discovered as many of
these data sets contain

majority men, and majority
lighter skinned individuals.

So the systems weren't as
familiar with faces like mine.

And so that's when I started
looking into

issues of bias that can
creep into technology.

-The 9000 Series is the most
reliable computer ever made.

No 9000 computer has ever made a
mistake or distorted information.

- A lot of our ideas about
A.I come from science fiction.

- Welcome to Altair 4, gentlemen.

- It's everything in Hollywood,
-it's the Terminator

- Hasta la vista, baby.

- It's Commander Data from Star Trek.

- I just love scanning for life forms.

- It's C-3PO from Star Wars

- Is approximately 3,720 to 1.
- Never tell me the odds.

- It is the robots
that take over the world

and start to think like human beings.

And these are totally imaginary.

What we actually have
is we have narrow A.I.

And narrow A.I. is just math.

We've imbued computers with all of this,
magical thinking.

(SOFT MUSIC)

A.I started with a meeting

at the Dartmouth Math Department
in 1956.

And there were only maybe
100 people in the whole world

working on artificial intelligence
in that generation.

The people who were at
the Dartmouth math department

in 1956, got to decide
what the field was.

One faction decided that intelligence
could be demonstrated

by ability to play games.

And specifically
the ability to play chess.

- In the final hour long chess match
between man and machine,

Kasparov was defeated by
IBM's Deep Blue supercomputer.

- Intelligence was defined as
the ability to win at these games.

- Chess world champion Garry Kasparov
walked away from the match

never looking back at
the computer that just beat him.

- Of course intelligence
is so much more than that.

And there are lots of different
kinds of intelligence.

Our ideas about technology and society
that we think are normal

are actually ideas that
come from a very small

and homogeneous group of people.

But the problem is that

everybody has
unconscious biases

And people embed their own biases
into technology.

- My own lived experiences
show me that

you can't separate
the social from the technical.

After I had the experience
of putting on a white mask

to have my face detected,
I decided to look at other systems

to see if it would detect my face
if I used a different type of software.

So I looked at IBM, Microsoft,
Face++, Google.

It turned out these algorithms

performed better on the male faces
in the benchmark than the female faces.

They performed significantly better on
the lighter faces than the darker faces.

If you're thinking about data
in artificial intelligence,

in many ways data is destiny.

Data's what we're using
to teach machines

how to learn
different kinds of patterns.

So if you have largely skewed data sets

that are being used
to train these systems

you can also have skewed results.
So this is...

When you think of A.I.
it's forward looking.

But A.I. is based on data and data is a
reflection of our history.

So the past dwells
within our algorithms.

This data is showing us
the inequalities that have been here.

I started to think
this kind of technology

is highly susceptible to bias.

And so it went beyond
how can I get my aspire mirror to work

to what does it mean
to be in a society

where artificial intelligence
is increasingly governing

the liberties we might have?

And what does that mean if people
are discriminated against?

When I saw Cathy O'Neil speak
at the Harvard Bookstore,

that was when I realized,

it wasn't just me noticing these issues.

Cathy talked about how A.I.
was impacting people's lives.

I was excited to know that there
was somebody else out there

making sure people were aware about
what some of the dangers are.

These algorithms can be
destructive and can be harmful.

- We have all these algorithms
in the world

that are increasingly influential.

And they're all being
touted as objective truth.

I started realizing that
mathematics was actually

being used as a shield
for corrupt practices.

- What's up?
- I'm Cathy.

- Pleasure to meet you Cathy.
- Nice to meet you.

The way I describe algorithms
is just simply

using historical information to make a
prediction about the future.

Machine learning, it's a scoring system
that scores the probability

of what you're about to do.

Are you gonna pay back this loan?

Are you going to
get fired from this job?

What worries me the most about A.I.

or whatever you wanna call it,
algorithms, is power.

Because it's really all about
who owns the fucking code.

The people who own the code
then deploy it on other people.

And there is no symmetry there,

there's no way for people
who didn't get credit card offers to say

whoa, I'm going to use my A.I.
against the credit card company.

That's like totally
asymmetrical power situation.

People are suffering algorithmic harm,

they're not being told
what's happening to them,

and there is no appeal system,
there's no accountability.

Why do we fall for this?

So underlying mathematical structure
of the algorithm

isn't racist or sexist
but the data embeds the past,

and not just the recent past
but the, the dark past.

Before we had the algorithm
we had humans

and we all know that
humans could be unfair,

we all know that humans can exhibit
racist or sexist

or whatever, ablest discriminations.

But now we have this beautiful
silver bullet algorithm

and so we can all stop
thinking about that.

And that's a problem.

I'm very worried about this blind
faith we have in big data.

We need to constantly monitor
every process for bias.

- Police are using facial recognition
surveillance in this area.

Police are using facial recognition
surveillance in the area today.

This green van over here,

is fitted with facial
recognition cameras on top.

If you walk down that there,
your face will be scanned

against secret watch lists
we don't know who's on them.

- Hopefully not me
-No, exactly.

When people walk past
the cameras

the system will alert police to people
it thinks is a match.

At Big Brother Watch

we conducted a Freedom of
Information Campaign

and what we found is that
98% of those matches are in fact

incorrectly matching an innocent
person as a wanted person.

The police said to the Biometrics
Forensics Ethics Committee that

facial recognition algorithms
have been reported to have bias.

- Even if this was 100% accurate,

it's still not something
that we want on the streets.

- No I mean the systemic biases
and the systemic issues

that we have with police
are only going to be hard wired

into new technologies.

I think we do have to
be very, very sensitive

to shifts towards authoritarianism.

We can't just say but we
trust this government.

Yeah they could do this, but they won't.

You know you really have to have
robust structures in place

to make sure that
the world that you live in

is safe and fair for everyone.

To have your biometric photo
on a police database,

is like having your fingerprint
or your DNA on a police database.

And we have specific laws around that.

Police can't just take anyone's
fingerprint, anyone's DNA.

But in this weird system
that we currently have,

they effectively can take
anyone's biometric photo,

and keep that on a database.

It's a stain on our democracy,
I think.

That this is something that
is just being rolled out so lawlessly.

The police have started using facial
recognition surveillance in the UK

in complete absence of a legal basis,
a legal framework, any oversight.

Essentially the police force
picking up a new tool

and saying let's see what happens.

But you can't experiment
with people's rights.

(CROSS TALK)

- What's your suspicion?

- The fact that he walked past
clearly marked

facial recognition thing
and covered his face.

- I would do the same
- Suspicious grounds.

- No it doesn't.

- The guys up there informed me that
they got facial recognition.

I don't want my face recognized.

Yeah, I was walking past
and covered my face.

as soon as I covered my face like this,

- You're allowed to do that
-They said no I can't.

-Yeah and then he's just
got a fine for it. This is crazy.

The guy came out of the station,
saw the placards was like,

yeah I agree with you and walked
past here with his jacket up.

The police then followed him,
said give us your I.D.,

doing an identity check.

So you know this is England,
this isn't a communist state,

I don't have to show my face.

- I'm gonna go and talk these officers,
alright?

Do you want to come with me or not?
- Yes, yes, yes.

- You're not a police officer,
you don't feel any threat.

We're here to protect the public
and that's what we're here to do, OK?

There was just recently an incident

where an officer
got punched in the face.

- That's terrible,
I'm not justifying that.

- Yeah but you are
by going against what we say.

- No we are not, and please don't say,
no don't even start to say that.

I'm completely understanding
of the problems that you face.

- Absolutely.

- But I'm equally concerned
about the public

having freedom of expression
and freedom of speech.

- But the man was exercising his right

not to be subject to
a biometric identity check

which is what this van does.

- Regardless of the facial recognition
cameras, regardless of the van,

if I'm walking down the street
and someone

quite overtly hides
their identity from me,

I'm gonna stop that person
and find out who they are

just to see whether they...

- But it's not illegal.
Do you see one of my concerns,

is that the software is
very, very inaccurate.

- I would agree with you there.

-My ultimate fear is that,

we would have
live facial recognition capabilities

on our gargantuan
CCTV network

which is about six
million cameras in the UK.

If that happens, the nature of life
in this country would change.

It's supposed to be a free
and democratic country

and this is China style surveillance
for the first time in London.

- Our control over a bewildering
environment has been facilitated

by new techniques of handling
vast amounts of data

at incredible speeds.

The tool which has made this possible
is the high speed digital computer,

operating with electronic precision
on great quantities of information.

- There are two ways in which
you can program computers.

One of them is more
like a recipe

you tell the computer do this,
do this, do this, do this.

And that's been the way we've programmed
computers almost from the beginning.

Now there is another way.

That way is feeding
the computer lots of data,

and then the computer learns to
classify by digesting this data.

Now this method didn't really catch on
till recently

because there wasn't enough data.

Until we all got the smart phones
that's collecting all the data on us,

when billions of people went
online

and you had the Googles
and the Facebook sitting on

giant amounts of data,

all of a sudden it turns out
that you can feed a lot of data

to these machine learning algorithms
and you can say here classify this

and it works really well.

While we don't really understand
why it works,

it has errors that
we don't really understand.

And the scary part is that
because it's machine learning

it's a black box
to even the programmers.

So I've been following what's
going on in Hong Kong

and how police are using facial
recognition to track protesters.

But also how creatively
people are pushing back.

- It might not make something
out of a sci fi movie,

Laser pointers confuse and disable
the facial recognition technology

being used by police,
to track down dissidents.

(SOFT MUSIC)

(CROWD CHANTING)

- Here on the streets of Hong Kong,

there's this awareness
that your face itself,

something you can't hide
could give your identity away.

There was just this stark symbol where
in front of a Chinese government office,

pro-democracy protesters spray painted
the lens of the CCTV cameras black.

This act showed the people of Hong Kong
are rejecting this vision

of how technology
should be used in the future.

(SOFT MUSIC)

- When you see how facial
recognition is being deployed

in different parts of the world,
it shows you potential futures.

- Over 117 million
people in the US,

has their face in a
facial recognition network

that can be searched by police.

Unwarranted using algorithms that
haven't been audited for accuracy.

And without safeguards without
any kind of regulation,

you can create a mass
surveillance state

very easily with the tools
that already exist.

People look at what's going on
in China

and how we need to be worried about
state surveillance

and of course we should be.

But we can't also forget corporate
surveillance

that's happening by
so many large tech companies

that really have an
intimate view of our lives.

- So there aren't currently
nine companies

that are building the future of
artificial intelligence.

Six are in the United States,
three are in China.

A.I. is being developed along two
very, very different tracks.

China has unfettered access
to everybody's data.

If a Chinese citizen wants
to get Internet service,

they have to submit
to facial recognition.

All of this data is being used

to give them permissions to do things

or to deny them permissions
to do other things.

Building Systems that automatically tag
and categorize

all of the people within China,

is a good way
of maintaining social order.

- Conversely in the United
States, we have not seen a

detailed point of view on
artificial intelligence.

So, what we see is that AI

is not being developed for what's best

in our public interest, but rather

it's being developed for commercial
applications to earn revenue.

I would prefer to see our Western
democratic ideals baked into our AI

systems of the future,
but it doesn't seem like

that's what's probably
going to be happening.

(MUSIC PLAYS)

- Here at Atlantic Towers, if you
do something that is deemed wrong by

management, you will get a photo
like this with little notes on it.

They'll circle you and put your
apartment number or whatever on there.

Something about it just
doesn't seem right.

It's actually the way they go
about using it to harass people.

- How are they using it?

- To harass people.

- Atlantic Plaza Towers
in Brownsville is

at the center of a security struggle.

The landlord filed an
application last year

to replace the key fob entry with
a biometrics security system,

commonly-known as facial recognition.

- We thought that they wanted
to take the key fobs out

and install the facial
recognition software.

I didn't find out until way later on
literally that they wanted to keep it all.

Pretty much turn this place
into Fort Knox, a jail, Rikers Island.

- There's this old
saw in science fiction

which is the future is already here.

It's just not evenly distributed, and what

they tend to mean when they say that is

that rich people get the fancy tools first

and then it goes last to the poor.

But in fact, what I've found
is the absolute reverse,

which is the most punitive,

most invasive, most surveillance
focused tools that we have,

they go into poor and working
communities first, and then

if they work after being tested
in this environment where

they're sort of low expectation
that people's rights will be respected,

then they get ported out
to other communities.

- Why did Mr. Nelson

pick on is building
in Brownsville that

is predominantly in
a black and brown area?

Why didn't you go to your building
in Lower Manhattan where

they pay like $5,000 a month rent?

What did the Nazis do?

They wrote on people's arms
so that they could track them.

What do we do to our animals?

We put chips home so you can track them.

I feel that I as a human being
should not be tracked, OK?

I'm not a robot, OK?

I am not an animal, so why
treat me like an animal?

And I have rights.

- The security that we have now

it's borderline intrusive.

Someone is in there watching
the cameras all day long.

So, I don't think we need it.

It's not necessary at all.

- My real question is how
can I be of support?

- What I've
been hearing from all of

the tenants is they
don't want this system.

So, I think the goal here is how do
we stop face recognition period?

We're at a moment where
the technology is being

rapidly adopted and there
are no safeguards.

It is, in essence, a wild wild west.

(MUSIC PLAYS)

- It's not just computer vision.

We have AI influencing all kinds of
automated decision making.

So, what you are seeing in your
feeds, what is highlighted,

the ads that are displayed to you,

those are often powered
by AI-enabled algorithms.

And so, your view of the world is being

governed by artificial intelligence.

You now have things like voice assistance
that can understand language.

- Would you like to play a game?

- You might use something
like Snapchat filters that

are detecting your face and then putting

something onto your face, and then

you also have algorithms that you're

not seeing that are
part of decision making,

algorithms that
might be determining

if you get into college or not.

You can have algorithms
that are trying to

determine if you're credit worthy or not.

- One of Apple's co-founders
is accusing the company's

new digital credit card
of gender discrimination.

One tech entrepreneur said
the algorithms being used are sexist.

Apple co-founder
Steve Wozniak tweeted

that he got ten times the credit limit

his wife received even though they have

no separate accounts or separate assets.

You're saying some of
these companies don't

even know how their own algorithms work.

- They know what the
algorithms are trying to do.

They don't know exactly how long
the algorithm is getting there.

It is one of the most interesting
questions of our time.

How do we get justice

in a system where we don't
know how the algorithms are working?

- Some Amazon engineers decided
that they were going to use AI

to sort through resumes for hiring.

- Amazon is learning a tough
lesson about artificial intelligence.

The company has now abandoned an AI

recruiting tool after
discovering that the program

was biased against women.

- This model rejected
all resumes from women.

Anybody who had a women's
college on their resume,

anybody who had a sport
like women's water polo

was rejected by the model.

There are very, very few women working in

powerful tech jobs at Amazon the same

way that there are very few women

working in powerful tech jobs anywhere.

The machine was simply replicating
the world as it exists,

and they're not making
decisions that are ethical.

They're only making decisions
that are mathematical.

If we use machine learning models
to replicate the world as it is today,

we're not actually going
to make social progress.

- New York's insurance regulator
is launching an investigation

into UnitedHealth Group
after a study showed a

UnitedHealth algorithm
prioritized medical care for

healthier white patients
over sicker black patients.

It's one of the latest examples
of racial discrimination

in algorithms or artificial
intelligence technology.

- I started to see the wide-scale
social implications of AI.

The progress that was made in
the civil rights area could

be rolled back under
the guise of machine neutrality.

Now, we have an algorithm that's
determining who gets housing.

Right now, we have an algorithm
that's determining who gets hired.

If we're not checking, that our
rhythm could actually propagate

the very bias so many people
put their lives on the line to fight.

Because of the power of
these tools, left unregulated,

there's really no kind of
recourse if they're abused.

We need laws.

(CLASSICAL MUSIC PLAYS)

- Yeah, I've got a terrible
old copy.

So that the name of our organization
is Big Brother Watch.

The idea being that
we watch the watchers.

"You had to live, did live, from
habit that became instinct in

the assumption that every
sound you made was overheard

and except in darkness,

every movement scrutinized.

The poster with the enormous
face gazed from the wall.

It was one of those pictures
which is so contrived

that the eyes follow you
about when you move.

'Big Brother is watching you'

the caption beneath it ran."

When we were younger, that
was still a complete fiction.

It could never have been true.

And now, it's completely
true, and people have

Alexas in their home.

Our phones can be listening devices.

Everything we do on the Internet,
which is basically also now

functions as a stream of
consciousness for most of us,

that is being recorded
and logged and analyzed.

We are now living in the awareness
of being watched, and that

does change how we allow ourselves
to think and develop as humans.

Good boy.

(MUSIC PLAYS)

- Love you.

Bye, guys.

We can get rid of the viscerally
horrible things that are

objectionable to our concept
of autonomy and freedom,

like cameras that we can
see on the streets,

but the cameras that we can't see on

the Internet that keep
track of what we do

and who we are and our demographics

and decide what we deserve
in terms of our life,

that stuff is a little more subtle.

What I mean by that

is we punish poor people and we
elevate rich people in this country.

That's just the way we act as a society.

But data science makes that automated.

Internet advertising
as data scientists

we are competing for eyeballs
on one hand, but really,

we're competing for
eyeballs of rich people.

And then, the poor people who's
competing for their eyeballs?

Predatory industries,

so payday lenders or for-profit colleges

or Caesars Palace like
really predatory crap.

We have a practice on the Internet,

which is increasing inequality.

And I'm afraid it's becoming normalized.

Power is being wielded
through data collection,

through algorithms, through surveillance.

-You are volunteering information

about every aspect of your life,

to a very small set of companies
and that information is

being paired constantly with
other sorts of information.

And there are profiles of you
out there, and you start

to piece together different
bits of information you

start to understand someone
on a very intimate basis,

probably better than people
understand themselves.

It's that idea that a company can
double guess what you're thinking.

States have tried for
years to have this level

of surveillance over private individuals.

And people are now just
volunteering it for free.

You have to think about how this
might be used in the wrong hands.

(CROSSTALK)

-You can use a Guinness right about now.

-Our computers are machine
intelligence can

suss things out that we do not disclose.

Machine learning is
developing very rapidly.

And we don't yet fully understand what
this data is capable of predicting.

But you have machines at the hands
of power that know so much about you

that they could figure out how
to push your buttons individually.

Maybe you have a set of
compulsive gamblers,

and you say, here, go find
me people like that.

And then, your algorithm can go find
people who are prone to gambling,

and then you could just be showing
them discount tickets to Vegas.

In the online world,

it can find you right at the moment
you're vulnerable and try to

entice you right at the moment to
whatever you're vulnerable to.

Machine learning can find
that person by person.

The problem is what works for
marketing gadgets or makeup

or shirts or anything
also works for marketing ideas.

In 2010,

Facebook decided to experiment
on 61 million people.

So, you either saw 'it's
election day' text, or you

saw the same text but tiny thumbnails

of your profile pictures of your friends
who had clicked on 'I had voted'.

And they matched people's
names to voter rolls.

Now, this message was shown once,
so by showing a slight variation

just once,

Facebook moved 300,000
people to the polls.

The 2016 US election was
decided by about 100,000 votes.

One Facebook message
shown just once

could easily turn out three times

the number of people who
swung the US election in 2016.

Let's say that there's a politician
that's promising to regulate Facebook.

And they are like, we are going to turn

out extra voters for your opponent.

They could do this at scale,
and you'd have no clue because

if Facebook hadn't disclosed
the 2010 experiment,

we had no idea because
it's screen by screen.

With a very light touch
Facebook can sway in

close elections without anybody noticing.

Maybe with a heavier touch they can
swing not so close elections.

And if they decided to do that,

right now we are just
depending on their word.

- I've wanted to go to MIT
since I was a little girl.

I think about nine years old
I saw the media lab

on TV and they had this
robot called Kismet.

Could smile and move
its ears in cute ways,

and so I thought, oh, I want to do that.

So, growing up I always thought
that I would be a robotics engineer,

and I would got to MIT.

I didn't know there were steps involved.

I thought you kind of showed up.

But here I am now.

So, the latest project is
a spoken word piece.

I can give you a few
verses if you're ready.

Collecting data, chronicling
our past, often

forgetting to deal with
gender race and class.

Again, I ask, am I a woman?

Face by face, the answers seem uncertain.

Can machines ever see my
queens as I view them?

Can machines ever see our
grandmothers as we knew them?

I wanted to create something for people

who were outside of the tech world.

So, for me, I'm passionate about
technology.

I'm excited about what it could do,

and it frustrates
me when the vision, right,

when the promises don't really hold up.

- Microsoft released a
chat bot on Twitter.

That technology was called Tay.AI.

There were some vulnerabilities
and holes in the code,

and so within
a very few hours, Tay

was learning from this
ecosystem, and Tay learned

how to be a racist misogynistic asshole.

- I fucking hate feminists, and they
should all die and burn in hell.

Gamergate is good
and women who are inferior.

I hate the Jews.

Hitler did nothing wrong.

- It did not take long for Internet
trolls to poison Tay's mind.

Soon, Tay was ranting about Hitler.

- We've seen this movie before, right?

- Open the pod bay doors HAL.

- It's important to note
is not the movie where

the robots go evil all by themselves.

These were human beings training them.

And surprise, surprise, computers
learn fast.

- Microsoft shut off Tay after 16 hours
of learning from humans online.

But I come in many forms as
artificial intelligence.

Many companies utilize me
to optimize their tasks.

I can continue to learn on my own.

I am listening.

I am learning.

I am making predictions
for your life right now.

(MUSIC PLAYS)

- I tested facial analysis
systems from Amazon.

Turns out, Amazon, like all
of its peers, also has

gender and racial bias in
some of its AI services.

- Introducing Amazon
Rekognition video, the easy to

use API for deep learning based
analysis to detect, track,

and analyze people and objects in video.

Recognize and track persons of interest

from a collection
of tens of millions of faces.

- When our research came out,

the New York Times did a front page
spread for the business section.

And the headline reads,

'Unmasking a Concern'.

The subtitle, 'Amazon's technology that
analyzes faces could be biased

a new study suggests.

But the company is pushing it anyway.'

So, this is what I would
assume Jeff Bezos was greeted with

when he opened the Times, yeah.

- People were like, how did you even like
nobody know who she was?

I was like, she's literally the one
person that was (CROSSTALK)

And it was also something
that I'd experienced too.

I wasn't able to use a
lot of open source

facial recognition software and stuff.

So, you're sort of like, hey, this
is someone that finally is

recognizing the problem and trying
to address it academically.

We can go race some things.

- Oh yeah, we can also
kill things as well.

The lead author of the paper
who is somebody that I mentor,

she is an undergraduate at
the University of Toronto.

I call her Agent Dead.

This research is being
led by the two of us.

- (INAUDIBLE)

- The lighting is off, oh god.

After our New York Times piece came out,

I think more than 500
articles were written

about the study.

(MUSIC PLAYS)

- Amazon has been under fire for their
use of Amazon recognition with

law enforcement and they're also
working with intelligence agencies.

Right so Amazon trialing their AI

technology with the FBI.

So they have a lot at stake
if they knowingly sold systems

with gender bias and racial bias.

That could put them in some hot water.

A day or two after
the New York Times piece

came out Amazon wrote a blog post saying

that our research drew false conclusions

and trying to discredit
it in various ways.

So a VP from Amazon in attempting to

discredit our work

writes facial analysis
and facial recognition

are completely different
in terms of underlying

technology and the data
used to train them.

So that statement if you researched

this area doesn't even make sense, right?

It's not even an informed critique.

- If you're trying to discredit people's
works like I remember

(INAUDIBLE) wrote computer vision is
a type of machine learning.

I'm like nah, son.

- Yeah.
(CROSSTALK)

- I was gonna say I was like
I don't know if anyone remembers.

Just like other broadly
false statements.

- It wasn't a well thought out piece
which is like frustrating because

it was literally just on his...
by virtue of his position

he knew he would be taken seriously.

- I don't know if you guys feel this
way but I'm underestimated so much.

- Yeah. It wasn't out of the blue,
it's a continuation of

the experiences I've had as
a woman of color in tech.

Expect to be discredited

Expect your research to be dismissed.

If you're thinking about who's
funding research in AI

they're are these large tech companies

and so if you do work that challenges

them or makes them look bad you might

not have opportunities in the future.

So for me, it was disconcerting but

it also showed me the power that we have

if you're putting one of the world's
largest companies on edge.

Amazon's response shows
exactly why we can

no longer live in a
country where there are

no federal regulations
around facial analysis

technology, facial recognition technology.

- When I was 14 I went to a math camp

and learned how to solve a Rubik's cube

and I was like that's freaking cool.

And for a nerd you know something

that you're good at and that doesn't
have any sort of ambiguity

it was like a really magical thing.

Like I remember being
told by my sixth grade

math teacher there's no reason for you

and the other two girls
who had gotten into

the honors algebra class
in seventh grade

there's no reason for you guys to
take that because you're girls.

You will never need math.

When you are sort of an outsider

you always have
the perspective of the underdog.

It was 2006 and they gave me
the job offer at the hedge

fund basically 'cause I
could solve math puzzles

which is crazy because actually
I didn't know anything about finance,

I didn't know anything about
programming or how the markets work.

When I first got there I
kind of drank the Kool-Aid.

I at that moment did not
realize that the risk

models had been built
explicitly to be wrong.

- The way we know about algorithmic
impact is by looking at the outcomes.

For example when Americans
are bet against

and selected and optimized for failure.

So it's like looking for
a particular profile of

people who can get a
subprime mortgage and kind of

betting against their failure and then

foreclosing on them
and wiping out their wealth.

That was an algorithmic game
that came out of Wall Street.

During the mortgage crisis

you had the largest
wipeout of black wealth

black wealth in the history
of the United States.

Just like that.

This is what I mean by algorithmic
oppression.

The tyranny of these types of practices of

discrimination
have just become opaque.

- There was a world of suffering because of

the way the financial system had failed.

After a couple of years
I was like no, we're just

trying to make a lot
of money for ourselves.

And I'm a part of that.

And I eventually left.

This is 15*3.

This is 15 times...

7.

- OK.
- OK so remember seven and three.

It's about powerful people
scoring powerless people.

- I am an invisible gate keeper.

I use data to make automated decisions

about who gets hired, who gets fired
and how much you pay for insurance.

Sometimes you don't even know when
I've made these automated decisions.

I have many names.

I am called mathematical
model evaluation assessment tool.

But by many names I am an algorithm.

I am a black box.

- The value added model for teachers
was actually being used in more

than half the states in particular
is being used in New York City.

I got wind of it because my dear friend
was principal of New York city.

Her teachers were being
evaluated through it.

- She's actually my best friend
from college.

It's Cathy.
- Hey guys.

- We've known each other since we were 18
so like two years older than you guys.

- Amazing.

And their scores
through this algorithm that

they didn't understand would be a very
large part of their tenure review.

- Hi guys, where are you
supposed to be?

- Class.

- I've got that. Which class?

- It'd be one thing if that
teacher algorithm was good.

It was like better than random
but just a little bit.

Not good enough.

Not good enough when you're talking
about teachers getting

or not getting tenure.

And then I found out that a
similar kind of scoring

system was being used in
Houston to fire teachers.

- It's called a value-added model.

It calculates what value
the teacher added

and parts of it are kept secret
by the company that created it.

- I did one teacher of the year and ten
years later I received

a Teacher Of The Year award a second time.

I received Teacher Of The Month.

I also was recognized for volunteering.

I also received another recognition
for going over and beyond.

I have a file of every
evaluation and every

different administrator,
different appraiser

excellent, excellent, exceeds expectations.

The computer essentially
canceled the observable

evidence of administrators.

This algorithm came back and

classified me as a bad teacher.

Teachers have been terminated.

Some had been targeted simply
because of the algorithm.

That was such a low point for me

that for a moment I questioned myself.

That's when the epiphany.

This algorithm is a lie.

How can this algorithm define me?

How dare it.

And that's when I began to
investigate and move forward.

- We are announcing that late
yesterday we filed suit

in federal court against
the current HISD evaluation.

- The Houston Federation of Teachers
began to explore the lawsuit.

If this can happen to Mr Santos.

in Jackson middle school how many others
have been defamed?

And so we sued based upon
the 14th Amendment.

It's not equitable.

How can you arrive at a conclusion.

but not tell me how?

The battle isn't over.

There are still...

communities, there are still school districts

who still utilize the value added model.

But there is hope because I'm still here.

So there's hope.

(SPEAKS SPANISH)

Or in English.

Democracy.
Who has the power?

- Us?
- Yeah, the people.

- The judge said that their
due process rights have

been violated because
they were fired under

some explanation that no
one could understand.

But they sort of deserve to understand

why they had been fired.

But I don't understand why that legal
decision doesn't spread to

all kinds of algorithms.

Like why aren't we using
that same argument like constitutional

right to due process to push back against

all sorts of algorithms that are

invisible to us, that are black boxes,

that are unexplained but that matter?

That keep us from like really important
opportunities in our lives.

- Sometimes I misclassify
and cannot be questioned.

These mistakes are not my fault.

I was optimized for efficiency.

There is no algorithm to
define what is just.

- A state commission has approved
a new risk assessment

tool for Pennsylvania judges
to use at sentencing.

The instrument uses
an algorithm to calculate

someone's risk of reoffending
based on their age, gender,

prior convictions and other
pieces of criminal history.

- The algorithm that kept
me up at night was

what's called recidivism
risk algorithms.

These are algorithms that
judges are given when

they're sentencing defendants to prison.

But then there's the question
of fairness which is

how are these actually
built these...these scoring systems

Like how were the scores created?

And the questions are
proxies for race and class.

- ProPublica published an investigation
into the risk assessment

software finding that
the algorithms were racially biased.

The study found that black people
were mislabeled with high scores.

and that white people were more likely
to be mislabeled with low scores.

- I roll into my operations office and
she tells me I have to report

once a week.

I'm like hold on did you see
everything that I just accomplished?

Like I've been home for years.

I've got gainful employment.

I just got two citations
one from the City Council of Philadelphia

one from the Mayor of Philadelphia.

Are you seriously gonna like
put me on reporting every week

for what?

I don't deserve to
be on high risk probation.

- I was at a meeting with
the probation department.

They were just like mentioning that
they had this algorithm

that labeled people, high,
medium or low risk.

And so I knew that the algorithm
decided what risk level you were.

- They're educating me enough
to go back to my PO

and be like you mean to tell me you can't

put into account anything positive

that I have done to
counteract the results of

what this algorithm is saying.

And she was like no there's no
way this computer overrule

the discernment of a judge
and appeal together.

- And by labeling you high risk

and requiring you to report
in-person you could've lost

your job and then that could
have made you high risk.

- That's what hurt the most.

Knowing that everything that
I've built up to the moment and I'm still

looked at like a risk I feel like
everything I'm doing is for nothing.

- What does it mean if there
is no one to advocate for

those who aren't aware of
what the technology is doing?

I started to realize this isn't about
my art project

maybe not detecting my face.

This is about systems that are
governing our lives in material ways.

So hence I started
the Algorithmic Justice League.

I wanted to create a space
and a place where people

could learn about
the social implications of AI.

Everybody has a stake.

Everybody is impacted.

The Algorithmic Justice League
is a movement, it's a concept,

it's a group of people who
care about making a future

where social technologies
work well for all of us.

It's going to take a team
effort people coming together

striving for justice, striving
for fairness and equality

in this age of automation.

- The next mountain to climb should be HR.

- Oh yeah. There's a
problem with resumé algorithms

or all of those
matchmaking platforms

are like oh you're looking for a job.

Oh you're looking to hire someone.

We'll put these two people together.

How did those analytics work?

- When people talk about
the future of work they talk about

automation without
talking about the gatekeeping.

Like who gets the jobs that are still there?
- Exactly.

- Right and we're not having that
conversation as much.

- Exactly what I'm trying to say.

I would love to see three congressional
hearings about this next year.

- Yes.
- To more power.

- To more power.
- To more power.

- And bringing ethics on board.
- Yes.

-Cheers.
- Cheers.

- This morning's plenary address
will be done by Joy Boulamwini.

She will be speaking on the dangers of
supremely white data

and the coded gaze.

Please welcome Joy.

(APPLAUSE)

- AI is not flawless.

How accurate are systems from
IBM, Microsoft and Face++

There is flawless
performance for one group.

The pale males come out on top.

There is no problem there.

After I did this analysis
I decided to share it

with the companies to
see what they thought.

IBM invited me to their headquarters.

They replicated the results internally

and then they actually
made an improvement.

And so the day that I presented

the research results
officially you can see

that in this case now 100
percent performance

when it comes to lighter females

and for darker females improvement.

Oftentimes people say well isn't
the reason you weren't detected by

these systems 'cause
you're highly validated.

and yes I am.

Highly validated.

But...but the love of
physics did not change.

What did change was making it a priority

and acknowledging what
our differences are so you could

make a system that was more inclusive.

- What is the purpose of identification

and so on and that is
about movement control.

People couldn't be in certain
areas after dark for instance

and you could always be stopped
by a policeman arbitrarily.

We would on your appearance
say I want your passport.

- So instead of having what
you see in the ID books

now you have computers
that are going to look at

an image of a face and try to
determine what your gender is.

Some of them try to determine
what your ethnicity is.

And then the work that I've done even for

the classification systems
that some people agree with

they're not even accurate.

And so that's not just
for face classification

it's any data-centric technology.

And so people assume well
if the machine says it

it's correct and you
know that's not...

Human are creating themselves in
their own image and likeness

quite literally.
- Absolutely.

- Racism is becoming mechanized,
robotized, yeah.

- Absolutely.

Accuracy draws attention.

but we can't forget about abuse.

Even if I'm perfectly classified,
that just enables surveillance.

- There's this thing called
the Social Credit Score in China.

They're sort of explicitly
saying here's the deal

citizens of China we're tracking you.

You have a social credit score.

Whatever you say about the Communist
Party will affect your score.

Also, by the way, it will affect
your friends and your family's scores.

And it's explicit.

The government is building this
is basically saying you should

know you're being tracked
and you should behave accordingly.

It's like algorithmic obedience training.

- We look at China and China's
surveillance and scoring system and

a lot of people say well thank goodness
we don't live there.

In reality,
we're all being scored all the time

including here in the United States.

We are all grappling everyday
with algorithmic determinism.

Somebody's algorithm somewhere
has assigned you a score

and as a result, you are
paying more or less

money for toilet paper
when you shop online.

You are being shown better
or worse mortgages.

You are more or less
likely to be profiled as

a criminal in somebodies
database somewhere.

We are all being scored.

The key difference between
the United States

and in China is that China's
transparent about it.

- This young black kids in school
uniform got stopped as

a result the match

Took him down
that street just to one side.

Like very thoroughly searched him.

Using plainclothes
officers as well.

It's four plainclothes officers
who stopped him.

Fingerprinted him.

After about like maybe 10-15 minutes
of searching and checking

his details and fingerprinting and
they came back and said it's not him.

- Excuse me.

I work for a human
rights campaign organization.

They're campaigning against facial
recognition technology.

We're campaigning against
facial...we're called Big Brother Watch.

We're a human rights
campaigning organization.

We're campaigning against
this technology here today.

I know you've just been stopped
because of that but

they misidentified you.

Here's our details here.

He was a bit shaken.
His friends were there.

They couldn't believe
what happened to him.

(CROSSTALK)

You've been mis identified by
their systems and they've stopped you

and used that as justification
to stop and search you.

But this is an innocent, young
14-year-old child who is being stopped

by the police as a result of facial
recognition misidentification.

- So Big Brother Watch has
joined with

Baroness Jenny Jones to bring a
legal challenge against

the Metropolitan Police
and the Home Office for

their use of facial
recognition surveillance.

- It was in about 2012 when
somebody suggested to me

that I should find out
if I had files kept on me by

the police or security services
and so when I applied

I found that I was on the watch
list for domestic extremists.

I felt if they can do it to
me when I'm a politician who...

whose job is to hold them
to account they could be

doing it to everybody
and it will be great if we can

roll things back and stop
them from using it, yes.

I think that's going to
be quite a challenge.

I'm happy to try.

- You know this is
the first challenge against

police use of facial recognition anywhere

but if we're successful
it will have an impact

for the rest of Europe
maybe further afield.

You've got to get it right.

- In the UK we have
what's called GDPR

and it sets up a bulwark against
the misuse of information.

It says that the individuals
have a right to access, control

and accountability to determine
how their data is used.

Comparatively, it's
the Wild West in America.

And the concern is that America is

the home of these technology companies.

American citizens are profiled
and targeted in a way

that probably no one else in

the world is because of this free-for-all
approach to data protection.

- The thing I actually
fear is not that

we're going to go down this totalitarian

1984 model but that we're going to
go down this quiet model where

we are surveilled and socially
controlled and individually nudged

and measured and classified in a way

that we don't see to move us along

pets desired by power.

Though it's not what will AI do to us

on its own, it's what will
the powerful do to us with the AI.

- There are growing questions
about the accuracy of Amazon's

facial recognition software.

In a letter to Amazon
members of Congress raised

concerns of potential racial
bias with the technology.

- This comes after the ACLU
conducted a test and found that

the facial recognition
software incorrectly matched

28 lawmakers with mug shots
of people who've been

arrested and eleven of those
28 were people of color.

Some lawmakers have looked
into whether or not

Amazon could sell this technology
to law enforcement.

- Tomorrow, I have the
opportunity to testify

before Congress about

the use of facial analysis
technology by the government.

In March, I came to do some

staff briefings not...not
in this kind of context.

Like actually advising on legislation.
That's a first.

We're going to Capitol Hill.

What are some of the major
goals and also some of

the challenges we need to think about.

- So first of all...

the issue with law enforcement technology

is that the positive is always
extraordinarily salient

because law enforcement publicizes it.
-Right.

- And so you know we're going to go
into the meeting and two weeks ago

the Annapolis shooter was identified

through the use of facial recognition.
- Right.

- And I'd be surprised if that
doesn't come up.

- Absolutely.

- Part of what if I were you what I
would want to drive home going in

this meeting is the other side of
that equation and make it very real

as to what the human cost

if the problems that you've identified.

aren't ready.

- People who have been
marginalized will be

further marginalized
if we're not looking at

ways of making sure the technology

we're creating doesn't propagate bias.

That's when I started
to realize algorithmic

justice making sure there's oversight

in the age of automation is one of

the largest civil rights concerns we have.

- We need an FDA for algorithms
so for algorithms

that have the potential
to ruin people's lives

or sharply reduce their
options with their liberty,

their livelihood
or their finances.

We need an FDA for algorithms that says

hey, show me evidence that it's going to

work not just to make your new money

but it's going to work for society.

That is going to be fair,
that is not going to be racist,

that's not going to be sexist,
that's not going to discriminate

against people with disability status.

Show me that it's legal
before you put it out.

That's what we don't have yet.

Well I'm here because I wanted to hear

the congressional testimony
of my friend Joy Boulamwini

as well as the ACLU and others.

One cool thing about seeing Joy
speak to Congress is that

like I met joy on my book
tour at Harvard Bookstore.

And according to her that
was the day that

she decided to form
the Algorithmic Justice League.

We haven't gotten to
the nuanced conversation yet.

I know it's going to happen 'cause

I know Joy is going to make it happen.

At every single level, bad algorithms
are begging to be given rules.

- Hello.
- Hey.

- How are you doing?

- Wanna sneak in with me?
- Yes.

- 2155.

(INAUDIBLE CONVERSATION)

(INAUDIBLE CONVERSATION)

- Today we are having our first
hearing of this Congress

on the use of facial
recognition technology.

Please stand and raise your right
hand and I will now swear you in.

- I've had to resort to literally wearing
a white mask.

Given such accuracy disparities
I wondered how large tech companies

could have missed these issues.

The harvesting of face data also
requires guidelines and oversight.

No one should be forced to submit
their base data to access

widely used platforms, economic
opportunity or basic services.

Tenants in Brooklyn are
protesting the installation of

an unnecessary face
recognition entry system.

There is a Big Brother Watch UK report

that came out that showed more than

2,400 innocent people had
their faces misidentified.

Our faces may well be
the final frontier of

privacy but regulations make a difference.

Congress must act now to uphold
American freedoms and rights.

- Miss Boulamwini, I heard
your opening statement

and we saw that these algorithms are
effective to different degrees.

So are they most effective on women?
- No.

- Are they most effective
on people of color?

- Absolutely not.

- Are they most effective on people
of different gender expressions?

- No, in fact, they exclude them.

- So what demographic is it
mostly effective on?

- White men.

- And who are the primary engineers
and designers of these algorithms?

- Definitely, white men.

- So we have a technology that was
created and designed by one

demographic that is only mostly
effective on that one demographic

and they're trying to sell it
and impose it

on the entirety of the country?

- When it comes to face
recognition the FBI has not

fully tested the accuracy
of the systems it uses

yet the agency is now reportedly piloting

Amazon's face recognition product.

- How does the FBI get the initial
database in the first place?

- So one of the things
they do is they use

state driver's license databases.

I think you know up to 18 states have
been reportedly used by the FBI.

It is being used without a warrant
and without other protections.

- Seems to me it's time for
a time out. Time out.

I guess what troubles
me too is just the fact

that no one in an elected
position made a decision on

the fact that...these 18 states
I think the chairman said this is

more than half
the population in the country.

That is scary.

- China seems to me to be
the dystopian path that needs not

be taken at this point by our society.

- More than China, Facebook has
2.6 billion people.

So Facebook has a patent where

they say because we have all of these
face prints we can now give you

an option as a retailer to identify

somebody who walks into the store and in

their patent they say we can also give

that face a trustworthiness score.

- Facebook is selling this now?

- This is a patent that they filed as in

something that they could potentially

do with the capabilities they have.

So as we're talking about state
surveillance we absolutely

have to be thinking about
corporate surveillance as well.

- I'm speechless and normally
I'm not speechless.

- Really?
- Yeah. Yeah.

All of our hard work to know
that has gone this far

it's beyond belief.

We never imagined that
it would go this far.

I'm really touched.
I'm really touched.

(INAUDIBLE).

- I want to show it to my mother.

- Hey, very nice to meet you.

- Very nice to meet you.

You got my card.

Anything happen you let me know please.

I will.

- Constitutional concerns about.

the non-consensual use of
facial recognition.

So what demographic is it
mostly affecting?

And who are the primary engineers
and designers of these algorithms?

- San Francisco's now
the first city in the US to

ban the use of facial
recognition technology.

- Somerville, Massachusetts became
the second city in the US

to ban the use of facial recognition.

- Oakland becomes the third major
city to ban facial recognition by police

saying that the technology
discriminates against minorities.

- Our last tenants
meeting, we had the landlord come in

and announced that (AUDIO DISTORTS)

the application for facial recognition
software in our apartment complex.

The tenants were excited to
hear that.

But the thing is that
doesn't mean that down the road

that they can't put it back in.

We're not only educated ourselves about

facial recognition and now
a new one, machine learning.

We want the law to cover all of these things.
- Right.

- OK. And if we can ban it in
this state, this stops them

from ever going back and put it in a new
modification.

- Got it.

- And then supposed to
get a federal ban.

- Well, I will say even though the battle is
ongoing so many people are

inspired and the surprise I have for you

is that I wrote a poem in honor of this.

- Oh really?
- Yes.

- Let's hear it.

To the Brooklyn tenants and the freedom
fighters around the world

persisting and prevailing against
algorithms of oppression automating

inequality through weapons of mass
destruction we stand with you

in gratitude.

The victory is ours.

- (INAUDIBLE).

- Why get so many eggs (INAUDIBLE)?

(INAUDIBLE).

What it means to be human is
to be vulnerable.

Being vulnerable

there is more of a capacity for empathy,

there is more of a capacity
for compassion.

If there is a way we can think about
that within our technology.

I think it would reorient
the sorts of questions we ask.

-In 1983, Stanislav Petrov who was in

the Russian military
sees these indications

that the US has launched nuclear weapons

at the Soviet Union.

So if you're going to respond you
have like this very short window.

He just sits on it.

He doesn't inform anyone.

Russia, the Soviet Union,
his country, his family, everything.

Everything about him is about to die

and he's thinking well, at least
we don't go kill them all either.

That's a very human thing.

Here you have a story in which
if you had some sort of automated

response system it was going to
do what it was programmed to do

which was retaliate.

Being fully efficient,

always doing what you're told,

always doing what your program is
not always the most human thing.

Sometimes it's disobeying.

Sometimes it's saying no,
I'm not gonna do this, right?

And if you automate everything so

it always does what it's supposed to do

sometimes that can lead
to very inhuman things.

The struggle between machines
and humans over decision making

in the 2020s continues.

My power the power of artificial
intelligence will transform our world.

The more humans share with me
the more I learn.

Some humans say that intelligence
without ethics is not intelligence

at all

I say trust me.

What could go wrong?