Original subtitles

?

(KNOCKING)

(DOOR OPENING)

Thanks for letting me use the lab on the weekend, Dr. Sherman.

Of course, Winnie, we want all the students

competing in this year's West Haven Science Competition

to feel supported.

How's your research project coming along?

Oh, it's very promising.

I'm gunning to win this thing.

Oh, great, well, good luck.

I'd offer to stay and help but Sunday is my fun day.

So, I'm just gonna leave you here to use the lab

completely unsupervised.

Thanks.

Let's get to work, Carl.

- (CARL SQUEAKS) - (WINNIE) If my hypothesis is correct,

my project will blow everyone else's out of the water.

And then I'll get into the university of my choice...

and be on the path to becoming a groundbreaking scientist

just like my heroes.

I'll follow in the footsteps of Marie Curie

and Mimi Phipps Clark.

And do research to benefit mankind.

Because Carl, in a world filled with ambiguity,

science is always right.

Actually... Science gets it wrong all the time!

Mother physics!

Wait, are you Adam from TV?

Oh, you watch the show.

Yeah, when "Myth Busters" isn't on.

Now those guys are smart.

Well, wanna do the honors?

Sure.

Hi, I'm Winnie Jones and this is Adam Conover,

and I'm on "Adam Ruins Everything."

Uh-oh.

?

Corrected & Synced by Bakugan

Adam, I've seen your show.

It's about proving things wrong.

But you can't do that to science.

Science is how we find out the truth about the world.

Yes, it is.

But the way we do science is often flawed.

Well, not here, it isn't.

Carl and I are working on a very exciting, very solid study

about how Vitamin D affects human memory retention.

Oh.

(SQUEAKING)

Yes, you are a good little test subject.

Actually, if you're trying to learn about humans,

he's a terrible little test subject.

(LOUD SQUEAKING)

Sorry.

What? Everyone knows testing on rodents

is practically the same as doing actual human trials.

Yes, everyone "knows" that.

Too bad they're wrong.

Cool! I've always wanted to be a B-A-L-B-C-J

laboratory mouse.

But what are we doing here?

The same thing we do every night, Winnie.

Ruining a common misconception.

(THUNDER CRASHES)

Rodents make up 95% of all animals

used for biomedical research in America.

That's 20 to 30 million mice a year.

Whoa, they must be such effective test subjects.

Actually, they're not.

Out of all clinical trials based on mouse models,

the results don't apply to humans over 80% of the time.

Well, time to file these.

(ADAM) And those failed trials

can cause hundreds of millions of dollars a piece.

And this drives up the cost of development.

Meaning we get fewer new drugs.

Yeah, we're out of funding. (GRUMBLING THROUGH PHONE)

Well, then tell the cancer patients

they'll have to wait, Pete!

But how can that be?

I read mice share 97% of DNA with humans.

True, but that 3% makes a big difference.

Mouse metabolisms are seven times faster.

We have different inflammation reactions.

And, you know, people are over 3,000 times bigger.

- (GASPS) - (SQUEAKING)

(ADAM) And our bodies can have

very different reactions to medicine.

In one extreme example from 2006, a new cancer drug

was tested in mice and passed with flying colors.

But when human subjects were injected with just

1/500th of a mouse dosage,

within hours, all of the volunteers

went into catastrophic, multi-organ failure

and had to be rushed to the ICU.

Oh.

The truth is, testing on mice

just doesn't tell us as much about humans as we think it does.

Jeez Louise.

So, why do we even use mice in the first place?

Because they're cheap and convenient.

Compared to other animal test subjects,

mice are a bargain.

And they're great in a lab setting because they're easy

to tame and store and they breed like crazy.

Congratulations!

It's 100!

Rodents are also super easy to genetically customize.

Want a mouse with no immune system?

Or how about one that glows in the dark.

Extra tumors strike your fancy?

You can have whatever model of mouse you want.

Finally, the Animal Welfare Act, the only federal law

that covers testing on animals doesn't protect mice at all.

(SCREAMING)

Oh, yeah,

we can never get away with this on a rabbit.

So, even though we know testing on mice is a problem,

we keep doing it anyways 'cause it's easier?

(WOMAN) And at this point,

it's almost impossible not to.

Ah, Winnie, this is...

Dr. Azra Raza,

Director of the Myelodysplastic Syndrome Center at Columbia!

I traded six Albert Einsteins for her.

(DR. RAZA) Thanks, Winnie!

And thanks for keeping me in mint condition.

Medical science is almost entirely built

around mouse testing.

Researchers have been using mice

since the 1800s.

And students are taught on them

from day one of school.

And most other testing methods

are prohibitively expensive.

You quite literally cannot do a study without mice.

At the moment in the United States,

the FDA will almost never approve a trial in humans

unless researchers have done it in rodents first.

The real problem is cancer research has become

so mouse centric, we cannot replace rodent testing

without overhauling the whole system.

Which is a shame.

Because if we want to cure cancer in humans,

we should be studying human cancer cells.

We are wasting valuable resources and time on mice.

(SQUEAKING)

Hey, come on! This is a family show.

(SIGHS)

Thanks, Doctor Raza.

You're welcome.

And here's another tip.

If you put me in your bike spoke,

it will sound like a motorcycle.

(MIMICKING MOTOR PUTTERING)

(SIGHS)

(BOOKS THUD TO FLOOR)

Now my whole project isn't useable.

Well, you don't have to throw out all of your work.

Nope! I am gonna start a new project from scratch.

I mean, might as well get used to it.

When I'm a scientist,

I'll be doing nothing but science all day.

Not quite.

Want me to tell you about it?

Coou have found a cool kid to mess with?

(WHISTLING THEME)

You know, Winnie, the reality working as a scientist

might not be quite what you expect.

Very funny, Adam.

You're a real Humphry Davy.

The chemist who discovered the effects of laughing gas? Hello?

I know exactly what life as a scientist will be like.

I'll spend my days investigating the big unanswered questions.

I'll follow the research wherever it leads,

beholden to nothing and no one

but the pursuit of pure scientific truth.

That's a beautiful dream.

But in reality, what science you get to do depends a lot

on who gives you money to do it.

'Cause if you wanna pursue scientific truth,

first you gotta find someone willing to pay for it.

Around two-thirds of basic research is privately funded.

But here's the problem.

The priorities of the people handing out the funding

often affect what research gets done.

(ANNOUNCER) Welcome to "Smart Tank."

Today the Smarts will hear a proposal from Winnie,

a young researcher competing

in the West Haven Science Competition.

Hello, Smarts.

Uh, today, I'm asking for funding

for my biomedical research.

Here's my proposal.

You'll find it's very exciting and worthwhile.

First up, a major funder of research,

corporate America.

Winnie, we might be able to help you.

I just have one question.

Is this gonna help us sell more candy bars?

No... Why?

Well, corporations and trade groups

tend to fund research that will make them money.

In 2011, the National Confectioners Association

paid for a study

that implied eating sweets actually makes kids skinnier.

If you can't provide that kind of return

on investment, I'm out.

Well, I see plenty of upside.

Your research can help us develop new drugs.

Really?

Sure.

And if your results make our drugs look good,

we'll publish them.

Otherwise we'll bury them

so the results will never be seen. (CHUCKLES)

Yeah, they actually do that.

Well, that's wrong.

I won't be a part of it.

Oh, an idealist. I'm out.

Well, there's two Smarts left.

Okay, another major source of funding is philanthropy.

Winnie, I apologize for my robber baron friends here.

You and I, we're in it to help people.

See? This lady gets me.

Sure, and philanthropic donors fund some worthy research.

But they're also more likely to give money

to causes that have affected them personally.

You have to use this to find a cure for diamond allergies.

(LOUD SNEEZE)

Or maybe I could study diabetes or sickle cell anemia?

Oh, sorry, sweetie, I'm out.

This is a travesty.

Isn't anyone willing to fund objective,

independent, unglamorous research?

Yes, there is one final funding source.

Public funding.

Also is Mark Cuban here

because I would, like, die if I met Mark Cuban.

Because government funding isn't controlled by profits

or the whims of one person, it supports the unsexy

yet vital basic research that all science builds upon.

And it's led to advances like...

Heart rate monitors, the Human Genome Project,

audio recording, water purification, satellites,

solar panels, super computers, GPS and the internet.

Just to name a few.

Even Google got its start with a government grant.

But you won't see that on the first page of search results.

Wow, sounds like public funding is awesome.

Finally a funding source with no downsides.

Well... there is one little wrinkle.

Federal science budgets are controlled by politicians.

I hereby award you, Winnie, this government grant for...

(WHISPERING) What's that, Senator?

You'd rather have tax cuts than scientific knowledge?

And there's a powerful lobby that won't like the research?

And you don't know enough about science

to care about the consequences?

(SIGHS)

Sorry, I'm out.

No! This work is worth the money.

Ask any scientist, they'll tell you.

Sorry, politicians tell me what to do, not scientists.

Maybe not the best system.

But science needs the money.

Yes, it does.

Private funding will never be enough to support research

in the public interest.

The kind of long-term careful science that doesn't generate

profits or glory, but improves people's lives

and increases our understanding of the universe.

We need public funding for science.

But in the past 50 years,

the portion of the discretionary budget

that government spends on research and development

has been cut nearly in half.

And with less funding,

there's more competition for fewer resources.

In the past two decades,

30% of National Health Institute grants got funded.

But today, it's about 17%.

No, no, no!

And from 2010 to 2013, the U.S. had the biggest

three-year reduction of scientific research

and development since the end of the space race.

(SIGHS)

But if it's so hard to get funding,

then how do we do science?

Well, sometimes we just don't.

In a recent study of 11,000 scientists,

half of them said they've had to stop doing

valuable research partway through

because of economic pressures.

Who knows what great advancements we lost out on as a result.

As much as we'd love to believe that the progress of science

is determined solely by the best and brightest,

decisions about which science gets funded

have always been controlled by non-scientists.

It's all about who holds the purse strings.

That may be true but science always triumphs in the end.

Look at all these journals.

Great, factual peer reviewed research is published every day.

(SUCKING TEETH) Ah...

- Ugh. - What?

Nothing, just that a surprising number of scientific studies

do not... hold up.

(SCREAMING)

- Oh! - Excuse me.

I'm using those studies for my project.

Go ahead, but there's a good chance

a lot of those studies are wrong.

Adam, please, when a study has been published,

that means its data is robust and accurate.

Winnie, if you'd ever read "US Weekly", you'd know

that just 'cause something is published,

that doesn't mean it's true.

What's "US Weekly?"

Wow, you are nerdier than I am. (CHUCKLES)

Studies can be flawed for a number of reasons.

?

If the sample size was too small,

the luck of the draw could've skewed the results.

We should test this on 10,000 people but...

I only know ten.

That's okay, making friends is hard.

(ADAM) Or maybe there was a confounding variable.

Like, if the subject was using another product

that could affect the results

and the researchers didn't account for it.

Oh, wow, this pill is really

boosting our subject's resistance to sunburn.

(ADAM) And something, very rarely,

unscrupulous researchers will fake their results.

What if we just make up numbers

that show that this pill cures cancer?

Hello? Nobel Prize committee?

I've got some data you've gotta see.

Sure, so we just need a way to find out

which studies are accurate.

We already have one. And that's to reproduce them.

To show the results weren't just some fluke,

someone else has to do another study

that tests the same question

to see if they can get the same findings.

It's like double checking the answers.

Curse, my results were way different.

And here's the problem.

Since studies are usually reproduced

after they're published,

that means they have a chance

to influence other research or even become famous

before we find out they're wrong.

For instance, remember that famous study on power posing.

Uh, it's only one of the most watched TED Talks of all time.

It proved that by standing in the power pose,

your body produces hormones that give you confidence.

Well, no one's ever been able to reproduce those results.

What?

Or how about that series of studies

from the '70s and '80s

that showed you can actually make yourself feel happier

just by smiling?

Well, that makes sense, right?

Sure, except that in 2016, almost 20 labs tried

to replicate it and not one of them

was able to reproduce those findings.

(TRAIN WHISTLE BLOWS)

Holy Hawking!

If that's true, then how many other studies might be wrong?

Oh, a lot of them.

In fact, many scientists say that we're in the midst

of a reproducibility crisis.

Like a good type of crisis, right?

One research team just replicated 100 famous psychology

studies but found they couldn't reproduce about 60% of them.

(BEEP)

Another revisited 67 major drug studies

and found that about 75% didn't match their results.

And another team zeroed on 53 recent cancer studies

and couldn't reproduce 47 of them.

(BEEP, LOUD EXPLOSION)

But those numbers are so high!

If science gets it wrong that often,

then we should be reproducing every study.

(MAN) That'd be nice, but it probably won't happen.

There... and that's how you fold a fitted sheet.

This is Professor Nosek. of the University of Virginia.

And the co-founder of the Center for Open Science.

Adam's right, Winnie.

Replication studies are not getting done.

And it's because scientists don't have enough good reasons to do them.

Wait, what do you mean?

To succeed in their careers, scientists have to publish

and get attention for their research.

And that's more likely to happen

if their studies have new and flashy conclusions.

My study found that Fidget Spinners give you six-pack abs.

(CROWD GASPS AND APPLAUDS)

And the pressure to get these kinds of results

sometimes leads researchers to cut corners.

I didn't need test subjects.

I just asked my bro, Kevin.

He's super ripped.

On the other hand, replication studies make for good science.

But they don't get as much attention.

People just aren't interested in studies

that don't find anything new.

And I replicated a 2003 study

on classification of fine-grained polypeptide folds.

The results held up.

(CRICKETS CHIRPING, MAN COUGHS)

Hey, her work is just as important.

Yeah, but it isn't as exciting.

Science is competitive.

There aren't enough jobs or grant dollars for everyone.

So, if you were trying to get people excited about your research, what would you do?

Follow-up on someone's prior findings?

Or try to find something new and amazing?

I, uh, also did a study that found...

candy cures leukemia!

(CROWD GASPS AND APPLAUDS)

(SHUTTERS CLICKING)

These incentives to publish lead to a lot of unreliable research.

One estimate suggest that in the U.S.,

we spend $28 billion a year on biomedical research

that can't be reproduced.

That's such a waste!

Well, flashy findings can be useful

for stimulating new areas of discovery.

But we need to change the incentives so that researchers

are rewarded for producing results

that are interesting and reproducible.

Thanks, Professor Nosek.

No problem.

Now if you'll excuse me,

I'm gonna go use the emergency shower.

I save a ton on my water bill.

Oh, thrifty. (CHUCKLING)

So, even if my research gets positive results,

I still have to find someone to reproduce it,

but no one will because they only care about

their own careers?

Wait, what are you doing?

You're right, Adam.

Science sucks, I'm done.

I'm gonna go drink some Red Bull in the woods

or do whatever cool kids do.

Oh, wait!

I was about to tell you why science is actually amazing!

I thought the structure of my show!

?

(BIRDS CHIRPING)

You forgot this.

I don't need it.

I'm quitting the competition.

You know, I used to love that no matter how chaotic

and uncertain the world felt,

I could at least count on science to always be right.

But you showed me, it's not.

Winnie, do you know why science is so awesome?

It's the field of human inquiry

that embraces mistakes and learns from them.

Scientists don't just assert what they think is true.

They test their hypotheses against the empirical evidence.

And if it turns out they're wrong,

they admit it and try again.

The fact that science seeks out its own errors

and corrects them is its greatest strength.

But how do we fix what's wrong with science itself?

With more science. With more science.

When I learned about the reproducibility challenges

in science, I thought we need to study this problem.

So we started The Center for Open Science

to investigate why is it that researchers are not producing

reproducible science even though they want to?

So, we built software

so that researchers can share their data more easily

and be more transparent in their methods.

We also work with journals and societies and funders

to align the incentives so that it's in researchers interests

to be more open and reproducible in their work.

And when I first emigrated here from Pakistan,

I had no idea how much research in the U.S.

rested on flawed mouse testing.

To combat this problem, I started to collect human tissue,

and this is how I found

a novel treatment for pre-leukemia.

- Whoa. - So, if something in science is broken

like mouse models or faulty studies,

it doesn't mean we failed.

It means we need people like you to find the solution by using...

Science!

(ALL) Yes!

You're right!

I have to use science to fix science.

Eureka!

Thank you, guys.

(FAINT CHATTER)

And finally... Winnie.

And your project is...

Wait, where is your project?

Dr. Sherman, I didn't do a project.

I did 19.

Because for my experiment,

I replicated everyone else's research,

and I found that all of their conclusions were...

irreproducible.

Wow, well, that doesn't quite follow the project guidelines.

But, nice work, Winnie.

Sorry, everyone else.

(MUFFLED GRUMBLING)

You suck.

Well, I didn't do it for the glory.

I did it for science.

Come on, Adam, I'll show you my new research proposal.

I got some sweet new binder tabs.

Ooh, I love office supplies.

(CHUCKLING)

(BOY GASPS)

The card moved.

Did anyone else see that?

Holy... (SCHOOL BELL RINGS)

?

Hello, we're here on set with Brian Nosek

because I have way more questions for him

about the reproducibility crisis in science.

Thanks for having me.

Is the crisis real and is it something

that we should all be worried about?

Well, it's at least a crisis of confidence.

And it's, uh, something that emerged when researchers

across a variety of disciplines realized

that the published literature, what we thing we're doing

and producing and finding isn't as reliable as we think it is.

And so, can you walk us through a little bit

of how the fields realized this was happening.

Well, the oddity is is that we've known the challenges

for reproducibility for 60 years.

And it hasn't gotten a groundswell.

But that has shifted over the last ten years

because perhaps the internet helped make it easier

to evaluate lots of different papers at the same time.

Perhaps because of a couple of very dramatic findings

that there has been a convergence.

And so it has become a thing.

Now that we know about it, we can do something about it.

That's exactly right.

Science is at its best

when it is constantly self critiquing

of looking at where it's limited,

where it's not finding things that are reliable

and looking for ways to do it better.

Every scientific claim is wrong in some fundamental way.

And the process of discovery in science is figuring out

where our current models of the world are wrong

and how we can make them less wrong.

I'm so fascinated by that. Can you elaborate on

every scientific theory about the world

is wrong in some way. In some way.

If we had a perfect model of the world,

it would be as complex as the world.

How it is we describe how the world works,

how people interact, how atoms interact,

whatever it is, is a simplification.

- It is a way to try to describe... - Right.

...lots of different things that are happening in simple terms

that make predictions about next things that will happen.

Right.

But our predictions are always imperfect.

They can't quite get exactly the phenomena as we study them.

So, they're wrong. Right.

But wrong doesn't mean, oh, that's useless.

It's not information that we need.

It's not information that we can use.

Newton's Laws of Mechanics

are the most successful theory ever, right?

We use them to build bridges and skyscrapers...

I learned about them in high school, yeah.

And they're wrong, right?

Einstein showed how they are wrong

when things are moving very fast.

So, it isn't that something being wrong means,

oh, can ignore science.

It means there's always something more to learn.

So, that leads me really nicely to my next question which is...

Is this a reason that people

should stop believing or trusting science?

You know, should the audience go home

and every time they read about a new study say,

oh, that's probably never gonna be reproduced

and so I'm not even gonna think about it again?

One of the big challenges of scientific knowledge

is that is accumulates very slowly.

So, no one study is definitive.

Drinking coffee is bad for you.

No, drinking coffee is good for you.

No, drinking coffee's bad for you.

Well, make up your mind, scientists.

It's those claims that are initially very exciting.

But then survive confrontation over time

are the ones that survive and we have confidence in.

So, as a reader of the newspaper of reporting scientific results,

one thing to look for is, is this the first finding?

Is this something brand-new?

Uh, or is it the accumulation

of a lot of different pieces of evidence?

That's a very easy signal for whether that's a credible claim

or a reliable claim that we know a lot about it

versus something new and possibly exciting.

Yeah.

The reason to trust science

is because it doesn't trust itself.

Ahh. Right?

It's that science is always questioning its own findings

which makes it the most reliable way to discover things.

Thank you so much for being here.

Thanks for having me, Adam.

Repair and Synchronization by Easy Subtitles Synchronizer 1.0.0.0