NARRATOR: The future unfolds before our eyes...
but is it always beyond our grasp?
What was once the province of the gods
has now come more clearly into view
through mathematics and data.
Out of some early observations about gambling
arose tools that guide
our scientific understanding of the world
and more...
through the power of prediction.
(car speeding, alarm blaring, waves crashing)
From our decisions about the weather...
NEWSCASTER: The strongest hurricane ever on record...
NARRATOR: To finding someone lost at sea...
COAST GUARD CREWMAN: Commencing search pattern.
Keep a good look out!
NARRATOR: Every day mathematics and data combine
to help us envision what might be...
It's the best crystal ball that humankind can have.
NARRATOR: Take a trip on the wings of probability
into the future.
MONA CHALABI: We're thinking about luck
or misfortune,
but they just basically are a question of math, right?
"Prediction by the Numbers"--
right now, on "NOVA."
âª
Major funding for "NOVA is provided by the following:
âª
NARRATOR: The Orange County Fair, held in Southern California.
In theory, these crowds hold a predictive power
that can have startling accuracy,
but it doesn't belong to any individual, only the group.
And even then, it has to be viewed
through the lens of mathematics.
The theory is known as the "wisdom of crowds,"
a phenomenon first documented about a hundred years ago.
Statistician Talithia Williams is here
to see if the theory checks out, and to spend some time
with the fair's most beloved animal,
Patches, a 14-year-old ox.
(snorts)
TALITHIA WILLIAMS: It was a fair kind of like this one
where, in 1906,
Sir Francis Galton came across a contest
where you had to guess the weight of an ox,
like Patches you see here behind me.
âª
NARRATOR: After the ox weight-guessing contest was over,
Galton took all the entries home and analyzed them statistically.
To his surprise,
while none of the individual guesses were correct,
the average of all the guesses
was off by less than one percent.
That's the wisdom of crowds.
But is it still true?
So, here's how I think we can test that today.
What if we ask a random sample of people here at the fair
if they can guess how many jelly beans they think are in the jar.
And then, we take those numbers and average them
and see if that's actually close
to the true number of jelly beans.
âª
Guess how many jelly beans are in here.
Come on, guys, everybody's got to have their guess.
I see your mind churning.
1,227.
846.
Probably like 925?
I think a thousand.
WILLIAMS: So just write your number down.
Uh huh, there you go.
Can I have a jelly bean?
(laughter)
NARRATOR: The 135 guesses gathered from the crowd vary wildly.
WILLIAMS: The range of our guesses was,
from the smallest was 183, the largest was 12,000.
So you can tell, folks were really guessing.
But when we take the average of our guesses, we get 1,522.
So the question is,
how close is our average to the actual number of jelly beans?
Well, now's the moment of truth.
âª
All right, so the real number of jelly beans was 1,676.
The average of our guesses was off by less than ten percent.
So there actually was some wisdom in our crowd.
NARRATOR: Though off by about ten percent,
the average of the crowd's estimates
was still more accurate
than the vast majority of the individual guesses.
Even so, the wisdom of crowds does have limits.
It can be easily undermined by outside influences
and tends to work best on questions with clear answers,
like a number.
The steps Talithia took
reflect a process going on all around us these days
in the work of statisticians.
Thanks, everybody.
So we collected this data,
right, we analyzed it mathematically,
and we got an estimate that was pretty close
to the actual true value.
That's math and statistics at work.
âª
NARRATOR: We didn't always use math and statistics to make predictions.
The Romans studied the flights and cries of birds.
The Chinese cracked "oracle" bones with a hot metal rod
and read the results
19th-century Russians used chickens.
(chicken clucking)
Throughout history, we've sought the future
in moles on people's faces,
clouds in the sky,
or a pearl cast into an iron pot.
And that list of things used for predicting goes on and on.
But more recently-- that is the last couple hundred years--
to see into the future, we've turned to science
and made some remarkable predictions
from the existence of Neptune,
or radio waves,
or black holes,
to the future location of a comet with such precision
we could land a space probe on it.
(probe beeping)
But if you pop the hood of science,
inside you'll find a field of applied mathematics
that's made many of those predictions possible:
statistics.
Statistics is kind of unique.
It's not an empirical science itself, but it's not pure math,
but it's not philosophy either.
It's the framework, the language,
the rules by which we do science.
From that, we can make decisions,
we can make conclusions, we can make predictions.
That's what... that's what statisticians try to do.
Why I love statistics is that
it predicts the likelihood of future occurrences,
which really means it's the best crystal ball
that humankind can have.
âª
NARRATOR: Ultimately, all the predictive power of statistics
rests on a revolutionary insight from about 500 years ago--
that chance itself can be tamed
through the mathematics of probability.
Viva Las Vegas!
Here's a city full of palaces
built on understanding probability
and fueled by gambling,
which may seem a funny place
to find mathematician Keith Devlin.
But mathematics and gambling
have been tied together for centuries.
Today in a casino, you'll find roulette,
slot machines,
blackjack.
Playing craps is also known as "rolling the bones,"
which is more accurate than you might think.
KEITH DEVLIN: Humans have been gambling
since the beginnings of modern civilization.
The ancient Greeks, the ancient Egyptians,
would use the ankle bones of sheep as a form of early dice.
NARRATOR: Surprisingly, while the Greeks laid the foundation
for our mathematics, they didn't spend any effort
trying to analyze games of chance.
DEVLIN: It seems to have never occurred to them,
or indeed to anybody way up until the 15th, 16th century,
that you could apply mathematics
to calculate the way these games would come out.
âª
NARRATOR: 16th-century Italian mathematician Gerolamo Cardano
made a key early observation:
that the more times a game of chance is played,
the better mathematical probability
predicts the outcome,
later proven as the law of large numbers.
Examples of the law of large numbers at work surround us.
DEVLIN: When I flip this coin,
we have no way of knowing
whether it's going to come up heads or tails.
âª
That time it was heads.
On the other hand, if I were to toss a coin 100 times,
roughly 50% of the time it would come up heads,
and 50% of the time it would come up tails.
We can't predict a single toss.
We can predict the aggregate behavior over a 100 tosses.
That's the law of large numbers.
NARRATOR: In fact, casinos are a testament
to the iron hand of the law of large numbers.
The games are designed
to give the casinos a slight edge over the gambler.
Take American roulette:
on the wheel are the numbers one through 36,
half red and half black.
Betting a dollar on one color or the other
seems like a 50-50 proposition.
But the wheel also has two green slots with zeros.
If the ball lands in those,
the casino wins all the bets on either red or black.
And that's the kind of edge
that makes the casino money over the long run.
Customers are gambling.
The casino is absolutely not gambling.
Because they may lose money,
they may lose a lot of money to one or two players,
but if you have thousands and thousands of players,
by the law of large numbers,
you are guaranteed to make money.
âª
NARRATOR: The law of large numbers
comes into play outside of gambling too.
In basketball, a field goal or shooting percentage
is the number of baskets made
divided by the number of shots taken.
But early in the season,
when it's based on a low number of attempts,
that percentage can be misleading.
JORDAN ELLENBERG: At the beginning of the season, a less skilled player
might get off a few lucky shots in a row.
And at that point,
they'd have a super-high shooting percentage.
âª
NARRATOR: Meanwhile, a very skilled player
might miss a few at the beginning of the season
and have a low shooting percentage.
(buzzer)
But as the season goes on,
and the total number of shots climbs,
their shooting percentages will soon reflect
their true skill level.
(buzzers and chimes going off)
That's the law of large numbers at work.
A small sample, like just a few shots, can be deceptive,
while a large sample, like a lot of shots,
gives you a better picture.
âª
The gambling observations that led to the law of large numbers
were a start, but what really launched probability theory
and opened up a door to a whole new way
of thinking about the future,
was a series of letters
exchanged between two French mathematicians,
Blaise Pascal and Pierre de Fermat in the 1650s,
about another gambling problem that had been kicking around
for a few centuries.
A simplified version of the problem goes like this:
two players-- let's call them Blaise and Pierre--
are flipping a coin.
Blaise has chosen heads, and Pierre tails.
The game is the best of 5 flips,
and each has put money into the pot.
They flip the coin three times, and Blaise is ahead two to one.
But then the game is interrupted.
What is the fair way to split the pot?
The question is: how do they divide up the pot
so that it's fair to what might have happened
if they'd been able to complete the game.
NARRATOR: Fermat suggested imagining the possible future outcomes
if the game had continued.
There are just two more coin flips,
creating four possible combinations.
Heads-heads, heads-tails, tails-heads, and tails-tails.
In the first three, Blaise wins with enough heads.
Pierre only wins in the last case,
so Fermat suggested that a three-to-one split
was the correct solution.
The key breakthrough
was imagining the future, mathematically,
something even Pascal had trouble with.
Because what Fermat did was say, "Let's look into the future,
"look at possible futures,
"and we'll count the way things could have happened
in different possible futures."
It was a simple arithmetic issue,
but the idea of counting things in the future
was just completely new,
and Pascal couldn't wrap his mind around it.
NARRATOR: Eventually Pascal accepted Fermat's solution,
as did others,
and today, that exchange of letters is regarded
as the birth of modern probability theory.
People realized the future wasn't blank.
You didn't know exactly what was going to happen,
but you could calculate with great precision
what the likelihood of things happening were.
You could make all of the predictions
we make today and take for granted.
You could make them using mathematics.
NARRATOR: It was a fundamental insight,
and one of the doors that led to the modern world.
Inherent in all our attempts to predict the future--
from the stock market
to insurance
to web retailers trying to figure out
what you might buy next,
is the idea that with the right data,
the likelihood of future events can be calculated.
(thunder roaring)
In fact, one of the great success stories
in the science of prediction
yields a forecast that many of us check every day
to answer the question, "Do I need an umbrella
or a storm shelter?"
âª
The hurricane season of 2017
will be remembered for its ferocity and destruction.
The strongest ever on record...
The Puerto Rico and the San Juan that we knew yesterday
is no longer there.
(helicopter hovering)
NARRATOR: The storms formed and gained in intensity with surprising speed,
leaving forecasters to emphasize the uncertainty
of where they might land.
Maria is now a Category Three hurricane.
Exactly what it's going to look like, we just don't know yet.
There's still great uncertainty...
NARRATOR: In weather forecasting,
the only certainty is uncertainty.
LOUIS UCCELLINI: One thing we know
for sure is we cannot give you a perfect forecast.
Given the nature of how we make a forecast,
from the global observations to equations running on computers,
stepping out in time,
I don't think there'll ever be a perfect forecast.
NARRATOR: To fight that uncertainty,
forecasters have turned to more data-- lots more data.
Here at the National Weather Service
Baltimore-Washington office,
meteorologist Isha Renta
prepares for the afternoon launch of a weather balloon.
Twice a day, every day,
all across the U.S. and around the world,
at the very same time,
balloons are released to take a package of instruments
up through the atmosphere.
It transmits readings about every ten meters in height.
ISHA RENTA: It's my understanding that they have developed
other ways to get vertical profiles of the atmosphere,
but still the accuracy and the resolution
that the weather balloon will give you is a lot higher.
So that's why we still depend on them.
âª
NARRATOR: The data from Isha's weather balloon ends up
at the National Center for Environmental Prediction
in College Park, Maryland,
the starting point for nearly all weather forecasts
in the United States.
Her information becomes one drop in a very large bucket
of data taken in each day.
Temperature, pressure, wind speed,
and direction in the atmosphere.
Tens of thousands of point observations are used
every hour of every day as kind of a starting point.
That's where we begin the simulation,
from those observations.
NARRATOR: It all becomes part of a process,
which has been described
as one of the great intellectual achievements
of the 20th century: numerical forecasting.
âª
The first step in numerical forecasting
is to break a nearly 40-mile- thick section of the atmosphere
into a three-dimensional grid.
Then, each grid point is assigned numerical values
for different aspects of the weather,
based on the billions of measurements
continually pouring into the Weather Service.
So you'll have an understanding
of temperature, pressure,
and values in terms of wind and wind direction
at each one of these points
within this grid that covers the globe.
NARRATOR: From there, equations from the physics
of fluids and thermodynamics are applied to each grid point.
CARBIN: Not only do you change the characteristics
at each grid point, but the changes at those grid points
affect neighboring grid points,
and then neighboring grid points affect other grid points.
And so you evolve the atmosphere through time
in this three-dimensional space.
NARRATOR: And remarkably, the approach works.
It's amazingly crazy that it works.
It's remarkable how well it does work,
given that we're making grand assumptions
about the initial state, so to speak,
or the beginning state of any forecast.
NARRATOR: And that initial state turns out to be absolutely crucial.
âª
In the early days of numerical forecasting,
it seemed like a definitive weather prediction
extending far into the future might soon be possible.
But research in the 1960s
showed that slight errors in measuring the initial state
grow larger over time, leading predictions astray.
So as you step ahead in time,
the forecast will become less accurate.
NARRATOR: Ironically, that sensitivity to initial conditions
also suggested a way to improve the accuracy
of numerical weather forecasts.
Thanks to the power of today's computers,
forecasters can run their weather simulations not once
but several times.
For each run, they slightly alter the initial conditions
to reflect the inherent error built into the measurements
and the uncertainty in the model itself.
The process is called ensemble forecasting,
and the results are called spaghetti plots.
CARBIN: We're looking at about 100 different forecasts here
for the jet stream at about six days ago.
We have the actual jet stream
drawn as the white line on here today,
and you can see most of the forecasts six days ago
were well north of where we actually find the jet stream
this morning.
And then we'll go to a five-day forecast
and a four-day forecast and a three-day forecast
and then down to two days and the day of the event.
And you can see how the model forecasts all converge
on that solution, which is what you would expect them to do.
But you go back to the six-day forecast,
you can see the large spread in the ensemble solutions
for this particular pattern.
NARRATOR: In the end, meteorologists turn to statistical tools
to analyze weather forecasts
and often use probabilities to express the uncertainty
in the results.
That's the "40% chance of rain" you might hear
from your local forecaster.
Meteorology is probabilistic at its very core,
and I believe that the general public knows
there is uncertainty inherent in everything we say,
but we're getting better.
âª
Our forecasts for three days out now are as accurate
as one-day forecasts were about 10 years ago.
And this continues to improve.
So the science has advanced beyond my wildest dreams,
and it's hard to even see where it might go in the future.
NARRATOR: Just like in meteorology,
for the rest of science,
the ultimate test of our understanding
is our ability to make accurate predictions.
On a grand scale, scientific theories
like Einstein's general theory of relativity
have to make predictions
that can be tested to become accepted.
In that case, it took four years
before a full solar eclipse revealed
that light passing near the sun curved,
just as predicted by Einstein's theory--
the first proof he was right
that the sun's mass distorts the fabric of space-time--
what we experience as gravity.
(child babbling)
In fact, the scientific method demands a hypothesis
which leads to a prediction of results
from a carefully designed experiment
that will test its claim.
Surprisingly, it wasn't until the 1920s and '30s
that a British scientist, Ronald A. Fisher,
laid out guidelines for designing experiments
using statistics and probability as a way of judging results.
âª
As an example, he told the story of a lady
who claimed to taste the difference
between milk poured into her tea
and tea poured into her milk.
Fisher considered ways to test that.
What if he presented her with just one cup to identify?
If she got it right one time, you'd probably,
"Well, yeah, but she had a 50-50 chance, just by guessing,
of getting it right."
So you'd be pretty unconvinced that she has the skill.
NARRATOR: Fisher proposed that a reasonable test of her ability
would be eight cups,
four with milk into tea,
four with tea into milk,
each presented randomly.
The lady then had to separate them back into the two groups.
Why eight?
Because that produced 70 possible combinations
of the cups, but only one with them separated correctly.
If she got it right,
that wouldn't prove she had a special ability,
but Fisher could conclude, if she was just guessing,
it was an extremely unlikely result,
a probability of just 1.4 percent.
Thanks mainly to Fisher,
that idea became enshrined in experimental science
as the "p-value"-- p for probability.
If you assume your results were just due to chance,
that what you were testing had no effect,
what's the probability you would see those results
or something even more rare?
If you assume that there's a process
that is completely random,
and you find that it's pretty unlikely to get your data,
then you might be suspicious that something is happening.
You might conclude, in fact, that it's not a random process.
That this is interesting to look at what else might be going on,
and it passes some kind of sniff test.
NARRATOR: Fisher also suggested a benchmark:
only experimental results
where the p-value was under .05--
a probability of less than five percent--
were worth a second look.
In other words,
if you assume your results were just due to chance,
you'd see them less than one time out of 20.
Not very likely.
He called those results "statistically significant."
Statistically significant.
Now this is a terrible word.
It could be quite insignificant.
You could be detecting a very, very, very small effect,
but it would be called, in the mathematical lingo,
"significant."
âª
NARRATOR: Since Fisher's day,
p-values have been used as a convenient yardstick for success
by many, including most scientific journals.
Since they prefer to publish successes,
and getting published is critical to career advancement,
the temptation to massage and manipulate experimental data
into a good p-value is enormous.
There's even a name for it: "p-hacking."
P-hacking is when researchers consciously or unconsciously
guide their data analysis to get the results that they want,
and since .05 is kind of the-the bar for being able to publish
and call something real, and get all your grant money,
it's usually guiding the results
so that you arrive at that p of .05.
âª
NARRATOR: How much p-hacking really goes on is hard to know.
What may be more important is to remember
what was originally intended by a p-value.
The p-value was always meant to be a detective, not a judge.
If you do an experiment
and find the result that is statistically significant,
that is telling you, that is an interesting place to look
and research and understand further what's going on,
not "don't study this anymore because the matter is settled."
NARRATOR: In a sense, a low p-value is an invitation
to reproduce the experiment, to help validate the result,
but that doesn't always happen.
In fact, there are few career incentives for it.
Journals and funders prefer novel research.
There is no Nobel Prize for replication.
Another solution to p-hacking and the overemphasis on p-values
may simply be greater transparency.
WILLIAMS: More and more, what people are doing
is publishing their data.
And so it's becoming harder and harder to lie with statistics,
because people will just probe and say,
"Well, give me the set you analyzed
and let me see how you got this result."
NARRATOR: Statistics continues to play a fundamental role in science,
but really anywhere data is collected,
you'll find statisticians are at work,
looking for patterns, drawing conclusions,
and often making predictions--
though they don't always work out.
The presidential election of 2016
was a tough one for pollsters,
the folks who conduct and analyze opinion polls.
Hillary Clinton was the overwhelming favorite
to beat Donald Trump right up to election day.
Trump is headed for a historic defeat.
He's going to lose by a landslide.
I think that she's going to have a very good night.
NARRATOR: The "New York Times" put Trump's chances at 15%.
One pollster on election night gave him one percent.
A projection of a 99% chance of winning, is that correct?
The odds are overwhelming
of a Hillary Clinton victory on Tuesday.
I would be very surprised if anything else happened
NARRATOR: And, of course, Trump won, and Clinton lost.
CHALABI: People were repeatedly told,
"Hillary Clinton is the candidate
most likely to win this election," and she didn't.
And I think that really left people feeling almost lied to,
almost cheated by these numbers.
NARRATOR: So what was going on with the polls?
And exactly how do people predict elections?
One way is just by asking people who they'll vote for.
PATRICK MURRAY: One of the great things about polling
is that we don't have to talk to everybody
in order to find out what the opinions are of everybody.
We can actually select something called a sample.
âª
NARRATOR: Sampling is a familiar idea.
To see if the soup is right,
you taste a teaspoon, not the whole pot.
To test your blood at the doctor,
they typically draw less than an ounce,
they don't drain you dry.
But in many circumstances,
finding a representative sample is harder than it sounds.
WILLIAMS: Let's suppose that this
is the population of about a thousand people in a city,
and we want to know, "Are people for or against
converting a park into a dog park?"
And so these green beads down here
are going to represent people who were for it,
and the red beads are folks who are against it.
NARRATOR: Talithia's first step is to take advantage
of an unlikely ally in sampling: randomness.
MURRAY: The beauty of randomization
is that as long as you throw everything from your population
into one pot and randomly pull it out,
you can be sure that you're within
a certain percentage points
of the actual value that's in that pot.
NARRATOR: So the plan is to randomly sample the beads-- but how many?
That depends on how much accuracy Talithia wants.
One measure is the margin of error--
the maximum amount the result from the sample
can be expected to differ from that of the whole population.
It's the plus or minus figure, often a percentage,
you see in the fine print in polls.
But there's also confidence level.
Inherently, there is uncertainty
that any sample really represents a whole population.
The confidence level tells you how sure you can be
about your result.
A 90% confidence level means,
on average, if you ran your poll or sample 100 times,
90 of those times, it would be accurate,
within the margin of error.
Talithia knows the total number of beads is a thousand.
And she's settled on a plus-or-minus five-percent
margin of error at a 90% confidence level.
That means she needs a sample size of at least 214 beads.
WILLIAMS: Here are the results:
We got a 103 red beads and 111 green,
so about 48% of our population would vote against,
and about 52% would vote for.
Now, remember that margin of error that we talked about,
that plus-or-minus five percent?
So once you take that into account,
those numbers really aren't that different at all.
So I guess you could say, this puppy is too close to call.
NARRATOR: In fact, within the margin of error, the stats got it right.
There were an equal number of red and green beads in the jar.
While the sampling error built in from the mathematics
can be quantified,
there are other errors that can't.
MURRAY: The other parts of the error-- how we word our questions,
how the respondents feel that day,
the responsibility to predict
what their behavior is going to be somewhere down the line--
all those sources of error
are something that we can't calculate.
NARRATOR: And there's a catch to random sampling for polls too.
A few decades back,
when just about every household had a landline,
finding a random sample meant randomly dialing phone numbers.
MURRAY: Into the 1970s and the 1980s, we were getting,
you know, 90% response rates.
If we randomly chose a phone number,
somebody on the other end of that phone would pick it up
and would do the interview with us.
(phone rings)
NARRATOR: Those days are over.
Thanks to caller I.D. and answering machines,
people often don't answer their landlines anymore--
if they even have one.
Response rates are way down.
NATE SILVER: Only about ten percent of people
respond to polls.
So you're kind of crossing your fingers
and hoping the people you reach
are the same as the ones that are actually going to vote.
For example, we found in 2016, pollsters were not reaching
enough white voters without college degrees.
RICHARD DE VEAUX: If there's a bias in the data, you cannot recover from it.
As we've seen from some recent elections.
(chuckles)
NARRATOR: After Donald Trump's surprise win,
many wondered if polling was broken.
But if you look at the polls themselves,
and not the headlines,
on average, polls on the national and state level
were off by historically typical amounts.
SILVER: So when I hear people say, "Oh, the polls were wrong,"
then it probably reflects people's interpretations
about the polls being wrong,
where people, for various reasons,
looked at the polls, and they said,
"These numbers prove to me that Clinton's going to win."
When we looked at the polls, we said,
"These numbers certainly make her a favorite,
"but they point toward an election that's fairly close
and quite uncertain, actually."
NARRATOR: And in 2016,
the U .S. presidential election was just that close.
Trump's victory depended on fewer votes
than the seating capacity of some college football stadiums--
spread across three states:
Pennsylvania, Wisconsin, and Michigan.
And there were some problems with the polls in those states
that led to underestimating Trump's support,
according to a postmortem by a consortium of pollsters.
âª
Nate Silver, the founder of the website FiveThirtyEight,
is one of the biggest names in polling--
even though he doesn't generally conduct polls.
SILVER: Our job is to take other people's polls
and to translate that
in terms of a probability, to say basically whether--
um, who's ahead,
which is usually pretty easy to tell, um,
but then how certain or uncertain is the election
is the more difficult part.
NARRATOR: Like a meteorologist,
Nate presents his predictions as probabilities.
On the morning of Election Day 2016,
he gave Clinton about a 70% chance of winning
and Trump about a 30% chance.
That's like rolling a ten-sided die
with seven sides that are Clinton
and three that are Trump.
SILVER: People who make probabilistic forecasts,
they're not saying that politics is intrinsically random.
They're saying that we have imperfect knowledge of it,
and that if you think you can be more certain than that,
you're probably fooling yourself based on how accurate polls,
other types of political data are.
âª
NARRATOR: Ultimately, interpreting a probability
depends on the situation.
While a 30% chance might seem slim,
if you learned the flight you were about to board
crashed three out of every ten trips,
would you get on the plane?
FLIGHT ATTENDANT: As this plane only makes it to its destination
seven out of ten times,
please pay attention to our short safety briefing.
NARRATOR: Or if a weather forecaster said
there's only a 30% chance of rain,
and then it rained--
would you care?
ELLENBERG: If it does rain,
no one demands to know, "Why did it rain?
We have to get to the bottom of this."
We can say like, "It just did."
It might have rained, it might not have rained.
As it happened, it did.
I do think there's a certain natural resistance
to seeing things that maybe we care about
more than whether it's going to rain or not,
like elections, in that same way.
NARRATOR: As 2016 shows,
predicting who will win the U.S. presidency,
a one-time contest between two unique opponents,
is far from easy.
But in at least one field,
there are literally decades of detailed statistics
on how the contests played out--
baseball.
âª
Baseball has always been a game of numbers--
box scores, batting averages, ERAs, RBIs.
But while stats have always been part of baseball,
in the last 20 years, their importance has skyrocketed
due to sports analytics,
the use of predictive models to improve a team's performance.
To some extent every business, not just sports,
is really trying to predict the next event, you know.
Whether you're on Wall Street,
or if you're in the tech business,
what's the new new thing.
And for us, it's future player performance.
NARRATOR: Billy Beane was one of the first
to adopt the quantitative approach in the late '90s,
when he was the general manager of the Oakland Athletics.
Stuck with the low payroll of a small-market team,
he abandoned decades of subjective baseball lore
and committed the organization to using statistical analyses
to guide the team's decision-making.
It very much became a mathematical equation
putting together a baseball team.
NARRATOR: Billy's stats-driven approach started to attract attention
when the Oakland A's finished in the playoffs
in four consecutive years
and set a league record with 20 wins in a row.
Then it was lionized
and even given a name in a best-selling book and movie,
"Moneyball."
Brad Pitt plays Billy.
BRAD PITT: If we win on our budget with this team,
we'll have changed the game.
âª
NARRATOR: While "Moneyballing" didn't lead
to a league championship for the Oakland A's,
it did change the game.
Today, every Major League Baseball team
has a sports analytics department,
trying to predict and enhance future player performance
through data,
analyzing everything
from the angle and speed of the ball coming off the bat--
to which players should be brought up
from the minor leagues or traded.
BEANE: I'll never pretend to be a math whiz,
I just understand its powers and its application.
When you run a Major League Baseball team,
which is a great job,
and every kid who dreams of doing it,
I can tell you it's everything you've thought of.
But when they ask me,
"What do I have to do to do that?"
My answer is always the same.
I say, "Go study and get an A in math."
NARRATOR: While sports analytics has transformed baseball,
Moneyballing has found its way into many unrelated fields.
Proponents of data-driven decision making and prediction
have applied the approach to areas as diverse
as popular music and law enforcement.
Moneyballing has been enabled
by the vast amounts of information
gathered through the internet, so-called "Big Data."
Our current output of data
is roughly 2.5 quintillion bytes a day.
(splashing)
But what about the opposite situation,
when there's very little data, yet actions need to be taken--
for example when searching for people lost at sea?
How do you even begin to predict where they might be?
The U.S. Coast Guard's Sector Boston Command Center.
From this secure set of rooms,
the Coast Guard coordinates all operations in the Boston area,
including national security, drug enforcement,
and search and rescue.
(phone ringing)
Good morning, Coast Guard Sector Boston Command Center,
Mr. Fleming speaking.
PHONE CALLER: Uh, good morning, sir...
NARRATOR: A caller reports
that a friend went paddleboarding
earlier in the morning, but he's now overdue.
(alarm blaring)
The Coast Guard initiates a search
with a 45-foot response boat...
EDWARD NYGREN: Engaging...
NARRATOR: ...out of Boston Harbor.
NYGREN: Coming up.
NARRATOR: Unfortunately, a paddle craft in trouble
has grown increasingly common.
NYGREN: You are required to have a life jacket on.
The reason for that is in 2015,
I think we had 625 deaths nationwide--
a number of those people that were recovered
were recovered without a life jacket.
NARRATOR: The Command Center also launches another boat
out of Station Point Allerton, in Hull.
CREWMAN: Short tack disconnected.
LUKE SCHAFFER: Stand clear of lines.
(horn blowing)
NARRATOR: The caller said the missing person typically paddled
between Nantasket Beach and Boston Light,
about three miles away.
But with all the unknowns--
where he got into trouble and how he may have drifted--
the search area could be as large as 20 square miles.
Search and rescue operations
are often based on unique circumstances
and require action, despite incomplete information.
To attack problems like that,
statisticians turn to an idea that originates
with an 18th-century English clergyman
interested in probability-- Thomas Bayes.
Imagine you are given a coin to flip,
and you want to know if it is fair, 50-50 heads or tails,
or weighted to land more on heads than tails.
The traditional approach in statistics and science
doesn't assume either answer and uses experiments to find out.
In this case that involves flipping the coin a lot.
Or you could approach the problem like a Bayesian.
Unlike traditional statistics,
that means starting with an initial probability
based on what you know.
In this case, all the coins you've ever come across
in a lifetime of flipping coins have been fair.
It seems likely this one is probably fair too.
Next, you also flip the coin,
updating the probability as you go.
Let's say it starts off with several heads in a row.
That might make you wonder,
increasing your probability estimate that it's weighted.
But as you flip it more times, those start to look like chance.
In the end, your best estimate
is that it is probably a fair coin,
but you are open to any new information.
Like it belongs to your uncle the con man, "Crooked Larry."
NYGREN: Sector 659.
Our estimated time of arrival is one-one-five-eight.
NARRATOR: Bayesian inference creates a rigorous mathematical approach
to calculating probabilities based on new information.
And it sits at the heart of the Coast Guard's
Search and Rescue Optimal Planning System: SAROPS.
BRIAN FLEMING: He's been missing since 7:30 this morning,
so I'm going to go ahead and do a SAROPS drift.
NARRATOR: SAROPS takes information
about the last-known position of the object of the search...
FLEMING: What's the direction of the wind?
NARRATOR: ...along with the readings of currents and winds
and combines them with information
about how objects drift in the water
to simulate thousands of possible paths
the target may have taken.
These get processed into probabilities,
indicated by color,
and turned into search plans to be executed.
SCHAFFER: SAROPS is really a workhorse for the Coast Guard.
It does a lot of the calculations for us.
It provides us with a lot of valuable search patterns
and search-planning options.
MAN (on phone): I thought he was pretty far off shore but, you know,
he said he was okay, so I kept going.
NARRATOR: Word of the search has spread.
A boater calls in a sighting from earlier in the day.
What I did is I went in and put that information into SAROPS,
and it changed everything.
NARRATOR: SAROPS quickly recalculates all the probabilities
and generates a new search plan.
The area has shifted about three miles farther out to sea.
NYGREN: We are on-scene, commencing search pattern now.
âª
Keep a good look out.
CREWMAN: Roger, coming up.
We're assessing the situation on scene.
âª
SCHAFFER: Any object you see in the water, please take a closer look at.
âª
Paddleboarder, port side
âª
CREWMAN (on radio): Roger, we have located a paddleboarder
with zero-one person on board.
Off the port corridor!
CREWMAN: Starboard side.
I have a visual
CREWMAN: All right.
NARRATOR: As it turns out, the search has been a drill.
Hours earlier, the paddleboard was placed in the water
by another Coast Guard ship and allowed to drift.
The instruments mounted on it are there to measure wind
and record the path it's taken,
information that will later be used
to tweak the drift simulations in SAROPS,
though the system performed quite well today.
SCHAFFER: The object was right in the middle of our search patterns.
So SAROPS was actually dead-on accurate
in predicting where we needed to search
to find the missing paddleboarder.
âª
FLEMING: To be able to call a family and say,
"Your family and friends is coming home,"
is absolutely a call
that all of us should have the chance to make,
and, fortunately, because of stuff like this,
we do get to make that call.
âª
NARRATOR: The computational complexity of updating probabilities
held the Bayesian approach back for most of the 20th century.
But today's computing power has unleashed it on the world.
It's in everything from your spam filter
to the way Google searches work to self-driving cars.
Some even find in the Bayesian embrace of probability,
similarities to how we learn from experience.
And they've built it into computers,
Making it part of a powerful new force:
machine learning.
SEBASTIAN THRUN: In the past, when we programmed computers,
we tended to really write down, in excruciating detail,
a set of rules that would tell the computer
what to do in every single contingencies.
NARRATOR: But there's another approach--
to treat the computer
like a child learning to ride a bike.
No one teaches a child to ride using a set of rules.
There may be some tips,
but ultimately, it is trial and error--
experience-- that's the instructor.
THRUN: The new thing, the new kid on the block
is machine learning,
specifically something called deep learning.
Here, we don't inform the computer of the rules,
but through examples.
So similar to, like, a small child
that falls down and learns from this experience,
we just let the computer learn from examples.
NARRATOR: Suppose you want to train a computer
to recognize pictures of cats.
By scanning through thousands of labeled pictures--
some cats, some not--
the computer can develop its own guidelines
for assessing the probability that a picture is a cat.
And these days
computers are doing far more than just looking for cats.
THRUN: Some of the best computers now
can learn how to beat the world's best Go champion
or to discover documents in stacks of documents,
work that highly paid lawyers normally do,
or diagnose diseases.
At Stanford, we recently ran a study
to understand whether a machine-learning algorithm
can compete with top-notch, Stanford-level,
board-certified dermatologists
in spotting things like skin cancer.
And lo and behold, we found that our machine-learning algorithm,
our little box,
is as good as the best human doctor in finding skin cancer.
NARRATOR: That raises a lot of questions:
should we trust software over our doctors?
Or are diagnostic programs like Sebastian's
the intelligent medical assistants of tomorrow,
a new tool but not a substitute?
And there are other concerns.
If you asked a person riding a bike exactly how they do it,
they'd be hard-pressed to put it into words.
The same is true
with so-called "black box" machine learning applications
like Sebastian's:
no one, including Sebastian,
knows how it detects skin cancer.
Like the bicyclist, it just does,
which may be fine for diagnostic software,
but not for other aspects of medicine,
like treatment decisions.
DANIELA WITTEN: If what you're doing is deciding
what dose of chemotherapy to give a patient,
I think most people would be uncomfortable
with that being a black box.
People would want to understand
where those predictions are coming from.
NARRATOR: The same can be true
for evaluating who should get a home loan,
or who should get fired from their job for poor performance,
or who gets paroled,
all situations
in which black box machine learning software are in use.
WITTEN: These are algorithms
that can have a big effect on people's lives.
And we have to understand, as a society,
what is going into those algorithms
and what they're based on,
in order to make sure that they're not perpetuating
social problems that we already have.
âª
NARRATOR: We live in an age when the fusion of data, computers,
probability, and statistics
grants us more predictive power than we've ever known before.
We can see the tangible benefits,
and some of the dangers,
while also wondering where this will all go.
We're really seeing a new science of statistics
developing under our feet.
That's exciting,
and I think it must be a little bit like
what it was like when the theory of probability
was first being developed
by Pascal and Fermat and people around them,
that people were sort of saying,
"My God, these are questions that mathematics
can really have something to say about."
I think that must have been what it was like
when statistics in its traditional form
was being developed in the first part of the 20th century,
and suddenly people were just asking
whole new kinds of questions
that they couldn't even have approached before.
And I think we're having another moment like that now.
âª
NARRATOR: While tomorrow will always remain uncertain,
mathematics will continue to guide the way,
through the power of probability,
and prediction by the numbers.
This "NOVA" program is available on DVD.
To order, visit shopPBS.org or call 1-800-PLAY-PBS.
"NOVA" is also available for download on iTunes.
âª