r NOVA is provided by the following...
And...
And by the Corporation for Public Broadcasting
and by contributions to your PBS station from:
MAN: We're ten seconds away on tape two.
NARRATOR: Dave Ferrucci is a nervous parent.
He's spent the last four years building
a revolutionary new computer,
and it's about to face its biggest test
in front of an audience of millions.
WATSON: Hello, my name is Watson.
I hope we will have a good game today,
but first I have to test my voice.
NARRATOR: When the cameras roll,
the computer, called Watson, will make history
as it competes on the popular quiz show Jeopardy!
ANNOUNCER: This is Jeopardy!
It's frightening, right?
It's a different experience.
It's a very different experience for a scientist
to sit here and have this happen live.
MAN: Six, five, four...
NARRATOR: Ferrucci has reason to fear.
Watson is playing for a million-dollar jackpot
against the game's toughest competitors:
Brad Rutter, Jeopardy!'s biggest money winner,
and Ken Jennings, famous for winning 74 consecutive games.
KEN JENNINGS: There's some contestant's pride.
I want to beat my human competition but, you know,
as a species I would like mankind to beat
the big, bad computer.
WATSON: Let's finish "Leaders of World War II."
NARRATOR: This big, bad computer is the culmination
of four years' intensive work.
IBM has put Watson through hundreds of practice games,
with a stand-in host and real contestants.
CRAIN: After Germany invaded the Netherlands,
this queen, her family, and cabinet fled to London.
Maria?
Who is Beatrix?
CRAIN: No. Watson?
WATSON: Who is Wilhelmina?
CRAIN: That is correct.
ERIC BROWN: It's a human standing there with their carbon and water,
versus the computer with all of its silicon,
and its main memory and its disc.
NARRATOR: It seems like it should be easy for the computer to win,
with its enormous memory and processing power.
But the human brain makes an intimidating opponent,
especially on Jeopardy!
Jeopardy! questions are tricky.
They have puns in them, they have little jokes in them.
Just understanding the question is a pretty big deal.
CRAIN: This trusted friend
was the first nondairy powdered creamer.
Watson?
WATSON: What is milk?
(laughter)
CRAIN: No! Maria?
What is Coffee-mate?
CRAIN: Thank you.
FERRUCCI: Humans communicate very fluently
in, you know, natural language,
and that's where computers struggle dramatically.
NARRATOR: Now, Watson is on the verge of conquering that challenge.
CRAIN: Here we go.
A garment worn by a child, perhaps aboard an operatic ship.
Watson?
WATSON: What is pinafore?
CRAIN: Yes. How did you get that?
NARRATOR: If Watson wins on Jeopardy!, it will be a major breakthrough
in a quest that's gone on for decades:
the audacious dream to build a machine as smart as a person;
the quest for artificial intelligence-- AI.
I am Elektro.
My brain is bigger than yours.
When we started doing AI,
the goal was, why can't we build a person?
We all know how to make people, that's easy.
What if we could build one out of silicon?
NARRATOR: Pioneers drew their inspiration
from the world of science fiction.
As a child, Isaac Asimov turns up.
So here I'm an adolescent and the robots,
intelligent machines, are a part of my life.
At your service.
Let's see about getting them built.
NARRATOR: In the early days, computers grew rapidly more powerful,
quickly mastering complex equations.
The first programs we wrote at M.I.T.
solved problems that only very educated people could solve,
like problems in calculus and then algebra.
NARRATOR: The computer pioneers thought they were on a fast track
to building humanlike intelligence.
I confidently expect that within 10 or 15 years,
we will find emerging from the laboratories
something not too far from the robot of science fiction fame.
PATRICK WINSTON: In the beginning we thought,
"Well, maybe ten or 15 years and we'll have something
that's really smart."
WINOGRAD: In the beginning, people really were amazed
at how much computers could do.
When you see something that's improving very fast,
you simply assume it will continue improving that fast
indefinitely.
NARRATOR: In the '60s, confidence was so high
it inspired one of the most iconic film characters
of all time.
MAN: Hello, HAL, do you read me?
Do you read me, HAL?
HAL: Affirmative, Dave. I read you.
When I was a kid, I saw 2001: A Space Odyssey
and HAL was just the best thing ever.
DAVE: Open the pod bay doors, please, HAL.
HAL: I'm sorry, Dave, I'm afraid I can't do that.
You know, it was a murdering psychopath.
HAL?
But, it was intelligent, could talk to people,
could see people, could lip read,
could do all this stuff.
A machine that could do that--
and I'd never even seen a real computer at that time.
I was mesmerized.
I was instantly mesmerized by the character HAL.
There's this one part in the movie
where one of the actors in the movie is sketching quietly,
and HAL asks him...
HAL: Have you been doing some more work?
DAVE: A few sketches.
HAL: May I see them?
He says, "Oh, I'm sketching."
And he says, "Can you show me the sketch?"
And he holds it up.
HAL: Can you hold it a bit closer?
RAO: That's when I got goosebumps,
because that's such a human thing to say.
HAL: I think you've improved a great deal.
And I said, "This is wonderful.
"What does it take?
What does it take to build something like this?"
HAL: Bishop takes knight's pawn.
NARRATOR: More than 40 years after the creation of HAL,
no one has answered that question.
No real computer or robot has been able to interact
with humans as seamlessly as Hollywood imagines.
Tell me, what is love?
NARRATOR: The problem is our own human computer, the brain,
a complex entity that's defied any attempts at replication.
We just had no idea how sophisticated the brain was.
NARRATOR: The computer has always been king
when it comes to calculation
and processing huge amounts of data.
Pen, as in "the pen" what's the middle letter?
KIDS: "E"!
NARRATOR: But simple skills that humans master early in life,
like understanding language or recognizing objects,
continue to baffle researchers.
You know, people vastly misjudged
how subtle we are when we're intelligent.
People just hugely underestimated that.
NARRATOR: But the dream of building a computer that could talk
and match wits with humans never really died,
and a few years ago a new plan was hatched,
sparked by an unlikely event.
ANNOUNCER: This is Jeopardy!
ALEX TREBEK: In Wagner's operas,
this eldest Valkyrie is stereotypically dressed
in a horned helmet and breastplate.
Ken?
Who is Brunhilde?
NARRATOR: In 2004, Ken Jennings' 74 game-winning streak
on Jeopardy! set the country abuzz
and caught the eye of an IBM executive while out to dinner.
All of a sudden the entire restaurant cleared out
to the bar that I'm sitting at to go see Ken Jennings.
TREBEK: This "Grand Ole Opry" comedy star used to wear a straw hat
with the $1.98 price tag still attached.
Ken?
Who is Minnie Pearl?
Minnie Pearl, howdy!
NARRATOR: Charles Lickel wondered if a computer could ever play
as well as Ken Jennings.
So he pitched the idea to some of IBM's top scientists.
For the ones that knew Jeopardy!,
they said, "Charles, that's just too hard."
I think the prevailing view was
these questions were difficult to understand,
difficult to even comprehend what was being asked.
Yeah, I was like, "No way!" I was like, "No way!"
NARRATOR: But one researcher, Dave Ferrucci, was intrigued.
My view was maybe this isn't as completely impossible
as we think it is.
NARRATOR: For over 40 years,
Jeopardy! has been pop culture's IQ test.
Clues are given as answers.
And contestants have to respond in the form of a question.
"Mothers," $1,600.
TREBEK: It's a larger vessel
that guards and supplies smaller ones.
Christina.
What is the mothership? Mothership, yes.
JENNINGS: The show's central conceit
is a little syntactic reversal,
whereby they give you an answer and you supply a question.
You don't say "George Washington";
you say, "Who is George Washington?"
NARRATOR: To win, contestants need to be human encyclopedias.
It's essentially everything under the sun.
You know the categories at the same second
Alex tells the folks at home the category.
One, you have to have a broad knowledge,
because we have 13 categories on each show.
Kate, start.
"Ph" for $400.
TREBEK: For the record, Thomas Edison invented
the first practical one of these in 1877.
NARRATOR: Contestants also have to be fast.
What is the phonograph?
Good!
NARRATOR: They typically have three seconds or less
to come up with an answer.
TREBEK: The mortar and pestle
is a symbol of this profession.
Ariel.
What is a pharmacist? Pharmacist is right.
NARRATOR: To compete on Jeopardy!,
IBM's computer must have an enormous knowledge base
because it will not be connected to the Internet.
But the far bigger challenge for the machine
will be understanding clues,
which can be extremely convoluted or obscure.
TREBEK: You'll find this flower before "Pickle Bottom"
in a line of handbags and bedding.
And that would be petunia.
Back to you, Ariel.
JENNINGS: There'll be a lot of puns.
There'll be double meanings.
And these are things that computers, historically,
are terrible at.
NARRATOR: Human language is a minefield for computers.
Consider this sentence...
How's it go?
I shot an elephant wearing my pajamas.
Was I wearing the pajamas?
Was the elephant wearing the pajamas?
So there are different interpretations,
different ways to parse the sentence.
The word "shot"-- what's really going on there?
There's already ambiguity in there.
Could actually be shooting, sort of a hunting shooting.
If I'm a photographer and I'm immersed in that context?
I may interpret that as shooting with a camera.
Which one did I mean?
You have to look at the context.
NARRATOR: But a computer has no context.
It's just an electronic brain in a box.
In 2006, Ferrucci tackles this challenge,
along with the best and brightest programmers from IBM
and the country's top AI labs.
To start, they run a test.
GONDEK: We had an existing
state-of-the-art system that people had worked on
for a number of years,
and we tried applying that to the Jeopardy! challenge.
NARRATOR: They feed one of IBM's most sophisticated computer programs
hundreds of Jeopardy! questions like this one:
The correct answer is "Who is Edmund Halley?"
The computer says, "Who is Peter Sellers?"
The computer ran a search through a million documents,
looking for key words from the clue.
It homed in on a description of one of the Pink Panther films,
in which one character was a "paramour" or mistress.
The star of the movie?
Peter Sellers.
It's probably the last answer a human would come up with.
But it's typical for computers.
The team has a long way to go.
Just how far becomes clear when they compare the computer
to the best human players.
They create a graph called "the cloud."
Each dot represents a Jeopardy! champion's performance.
Jennings is at the top.
What you see is a cloud and it's around...
they answer around 50%, the winners do,
and they get around 90% correct.
NARRATOR: And where is the computer in this cloud?
GONDEK: If you ask it to answer all the questions,
it would be giving you 10% of the questions right.
You can't go on Jeopardy! like that.
I mean, the best humans are 90%, 92%.
We weren't even close.
NARRATOR: To win at Jeopardy!, the team will need a whole new way
to tackle human language,
one that takes advantage of the computer's basic strengths.
At its electronic core,
a computer speaks a very simple language:
binary code, on or off.
But with that simple code it can follow instructions
and solve complex problems
once reserved for intellectual giants.
It used to be the case that intelligence was chess, right?
If you can play chess, that's intelligence.
NARRATOR: Computers have mastered the game.
Chess is easy for computers
because the rules are very well defined and very clear.
NARRATOR: The rules of chess are relatively simple.
A board of 64 squares.
Each piece-- pawn, knight, queen-- can move a certain way,
and there's a single goal: take out your opponent's king.
For humans, it is the ultimate game of strategy.
The way computers play chess
is not at all the way people play chess.
We humans look at the board and have conceptual ideas like
control the center, attack on the right.
Very different from the way computers play chess.
NARRATOR: A chess-playing computer looks at
virtually every possible move it could make and every response,
every way the game could play out.
Computers play chess through searching a tree of moves
down to a very deep level,
looking ahead on every possible path.
But they do it by brute force--
by going 20, 30, 40 moves ahead
and seeing all the bad things that can happen.
A person can't look that many moves ahead, broadly.
NARRATOR: This is the power behind the most famous chess game
in the history of AI,
when in 1997, another IBM computer named Deep Blue
beat the reigning world champion, Gary Kasparov.
REPORTER (archival): The chess world champion walked away from the match,
never looking back at the computer that just beat him.
NARRATOR: The victory makes Deep Blue look pretty smart.
But is it?
WINSTON: Deep Blue...
it's only acting as if it's intelligent.
It's not really intelligent in the way that we humans are.
BROOKS: It's good at one thing-- it's playing chess.
It can't do anything else;
it has no other understanding of the world.
It's just about chess moves.
NARRATOR: This lack of understanding
has hampered every computer program
that's tackled human language.
A perfect example is a program from the 1960s called ELIZA.
ELIZA was one of the first programs
that had anything resembling human conversation.
It was a dialogue--
you typed things in, it typed things back.
FEMALE VOICE: How do you do?
Please tell me your problem.
I'm feeling sad.
Then it types back,
"Did you come to me because you're feeling sad?"
NARRATOR: ELIZA was programmed to respond like a psychiatrist.
But it had no real insight.
Instead, it followed simple rules
and rearranged key phrases.
So if I say "I'm dead,"
it responds, "Do you enjoy being dead?"
It doesn't have any understanding
that dead is a different kind of condition.
It really is just doing this sort of fill-in-the-blanks
kind of pattern matching.
NARRATOR: Anyone who tried to solve the language problem
hit the same brick wall--
the computer's profound ignorance
of what we take for granted every day.
There is just so much more that we know
that we don't know we know.
I mean, just we know all kinds of stuff,
like you press the up button in the elevator,
that means it's going to go up.
Or, milk is white, or water is wet.
I mean, there's just stuff that we know
that we don't even realize we know.
That's one of the things that makes it hard.
NARRATOR: All the commonsense knowledge a human brain collects naturally
seems much too complex to program into a computer.
But that hasn't stopped one scientist from trying.
DOUG LENAT: So we have actually manually entered about six million rules.
That's about 3% of what it's going to need to know
in terms of actually spanning what you and I would call
"human common sense."
NARRATOR: For the last 25 years,
Doug Lenat has been leading a team
trying to create humanlike intelligence
by teaching a computer common sense, rule by rule.
The program is called CYC, and at headquarters
the walls are covered with logic diagrams.
In a way, the magic of this, the power of this,
is if you just tell it each rule one by one by one
and you give it general logical reasoning capabilities,
that's all you need to do.
NARRATOR: So far, CYC has six million rules
and can answer a lot of commonsense questions.
Like this one:
LENAT: Can a can cancan?
(cancan music playing)
NARRATOR: CYC says no and it explains why.
Here it's essentially saying the reason
why cans can't cancan is that cans are inanimate objects,
and it knows that cancan dancing requires
at least partially having a brain and using it.
(cancan music continues)
It's just enough for CYC to get the right answer
for the right reason.
(music ends, applause)
NARRATOR: 25 years ago,
many experts considered rules and logic the best hope
for building artificial intelligence.
But it's become clear these alone are not enough.
It's not just a matter of piling in more and more stuff.
There are basic principles that we didn't understand;
putting in more and more stuff
doesn't get you basic principles.
NARRATOR: At IBM, as Dave Ferrucci and his team tackle
the Jeopardy! challenge,
they know that facts and rules are just the beginning.
We couldn't write rules
for every combination of word and phrases and context.
NARRATOR: They need a new system, more fluid and flexible,
to navigate the twists and turns
of many different kinds of Jeopardy! questions.
They name their system Watson after IBM founder Thomas Watson.
The electronic Watson consists of 2,800 processors.
That's like 6,000 high-end home computers.
All together, it's the size of 10 refrigerators.
The team starts filling his memory banks
with about 10 million documents,
most downloaded from the Internet.
Because when Watson plays Jeopardy!,
he must stand alone, just like his human competitors.
FERRUCCI: All kinds of content, okay--
encyclopedias, dictionaries, thesauri, books, plays,
you name it.
NARRATOR: The entire World Book Encyclopedia, Wikipedia,
the Internet Movie Database,
much of the New York Times archive and the Bible
are just some of Watson's resources.
And to build on Watson's foundation of data and rules,
the team turns to a powerful tool in the computing world.
It's called "machine learning."
Machine learning is just like human learning from examples.
VON AHN: Before people would just write rules, write rules by hand.
Nowadays, it's all based on examples.
NARRATOR: To understand how machine learning works,
consider for a moment the letter "A".
What if you had to describe it to a computer?
It's a real problem faced by the U.S. Postal Service,
whose computers must decipher all kinds of addresses,
printed and handwritten.
VON AHN: We all know what an "A" looks like.
I know when I see it,
but there's just way too many different types of "A"s.
There are fonts where the "A" is just a triangle pointing up,
that's an "A".
Pretty quickly, you realize, there is no simple set of rules
that you can write down currently
for a program to determine
whether a letter is an "A" or not.
NARRATOR: Humans might not be able to come up with the rules
that reliably identify all kinds of "A"s,
but it turns out a computer can do it for itself
if you give it enough examples.
The way you do it is you just get an "A",
send it to the program and say that's an "A".
Here is another "A", different one, that's an "A".
Here's another "A", it's a different one, that's an "A".
Then you would give it another example
and you would give it another example
and you would do that a million times.
NARRATOR: The computer hunts for patterns among all those examples
and it finds them.
So the next time it meets a letter "A"--
even one it hasn't seen before-- it will recognize it.
This is machine learning,
and it's a crucial element of Watson's programming.
The team trains Watson, but here, instead of letters,
the examples are tens of thousands
of old Jeopardy! questions
along with a cheat sheet of all the correct answers.
Using machine learning, Watson will hunt for patterns
between the type of question, the correct answer
and the kinds of evidence that support that answer.
Now we do this over thousands of questions,
so we come up with some way to weigh the evidence on average,
so that we come up with the right answer.
NARRATOR: Now, when he's faced with a brand new question,
Watson uses what he learned from these patterns,
and declares his confidence in each possible answer.
FERRUCCI: In the end we get a list that says,
"Here's the top answer, and we're 75% sure it's right."
NARRATOR: Watson has now become a complex architecture
of rules, raw data and machine learning
that enables him to use statistics
to choose the right answer.
To test out this system,
the team scours the halls for IBM employees
who can play Jeopardy!
and everyone squeezes into a conference room.
HOST: "In 1978, New Jersey Monthly reporter Steven Levy
famously found this man's brain."
Watson?
WATSON: What is Einstein's?
HOST: Albert Einstein's, yes.
(laughter)
HOST: "The Fifth Amendment says
"that private property shall not be taken for public use
without this."
Watson?
WATSON: What is "just compensation"?
HOST: Yes.
NARRATOR: With this new system, Watson surges into the winners' cloud.
We took a huge jump with machine learning.
HOST: Watson with a commanding lead, $24,863.
We saw a huge jump in performance.
We were, like, "Whoo!"
NARRATOR: Up to now,
appearing on the TV show has only been a dream.
But Watson is performing so well, Dave Ferrucci decides
it's time to call Jeopardy!
In December 2009, Jeopardy! producers arrive at IBM
to size up Dave Ferrucci's new creation.
Like any human contestant, Watson must audition
to earn a spot on the show.
FERRUCCI: We spent all this time, you know, developing this system
and pushing its capabilities,
then here you are sitting here, all the executives are there...
(laughter)
HARRY FRIEDMAN: You hear "computer," you think,
well, of course a computer should have all the answers.
You hear about Q&A technology.
Well, isn't this just a big search engine?
FERRUCCI: And they're waiting to see,
you know, what really is going to happen.
And you just don't know, you don't know.
NARRATOR: To impress the executives, IBM builds a makeshift studio,
hires comedian Todd Crain to act as game show host,
and brings in former TV contestants.
GONDEK: That was one of the tensest days I've ever had.
Because we had never seen it play against Jeopardy! players.
CRAIN: Select again, David.
GONDEK: And I remember, like, the day before, we're tuning everything.
I was putting in the best strategy that we had.
I was putting the best stuff that we had and I thought,
"Well, this is just gonna kill 'em!"
CRAIN: Miranda?
What is "The Cat's in the Cradle"?
That is correct.
What is "I am the Walrus"?
Yes!
MIRANDA: What is "Crocodile Rock"?
Yes.
GONDEK: They were just like professional athletes.
It was a really tough few games for us.
NARRATOR: In the first round, it seems that Watson is auditioning
not for a game show, but a sit-com.
Where do we go next?
WATSON: L, underscore, underscore, underscore, underscore, O
Whoa... for 1,000.
NARRATOR: There are suddenly unexpected bugs that need fixing.
We weren't dealing with Roman numerals well.
So it'll say like "Henry the 5th," we would say, "Henry Vee."
CRAIN (reading):
...along with his weak-minded half-brother, Ivan the Fifth."
CRAIN: Watson?
WATSON: What is "Peter"?
CRAIN: More specific?
WATSON: What is "Peter 'Eye'"?
(laughter)
No.
Carrie or David?
Carrie? Who is Peter the Great?
That is correct.
NARRATOR: In Final Jeopardy!, where contestants must place bets
and write down the answer, things only get worse.
Under the category "Flags," the clue is...
NARRATOR (reading):
(Final Jeopardy theme music playing)
You need to know
a little bit something about 18th-century flags.
David, let's see if you did.
What is the four-word motto we're looking for, David?
What is, "Don't tread on me"?
That is correct!
Let's see if Watson got it right.
(laughter)
NARRATOR: Watson didn't recognize
the word "motto,"
and after scanning through millions of documents,
he found the word "terrorism" associated with "September 11"
so frequently, that seemed like the best answer.
By the time they break for lunch,
it's humans, two; Watson, zero.
And it's not clear if Watson will ever be ready
for primetime.
This was taking a risk for me
in the sense that you're sitting here and saying, "You know what,
I think this is possible," and then you fall flat on your face
and people say,
"Well, we're never going to believe Ferrucci again."
Did I expect to get fired?
No, but maybe.
(chuckles)
NARRATOR: But after lunch, the producers are treated
to a different side of Watson.
We came back, and the third game was neck and neck,
incredibly competitive.
"In Act Three of an 1846 Verdi opera...
(continues reading):
Watson?
WATSON: What is Attila?
Be more specific?
WATSON: What is Attila the Hun?
Thank you very much.
Attila the Hun-- I'll take that.
NARRATOR: That afternoon, Watson climbs back in the game.
"Wordsworth said they soar, but never roam."
Watson?
WATSON: What is skylark?
CRAIN: That is correct.
It's a device clamped to the wheel of a parked car
with overdue tickets.
Watson?
WATSON: What is boot?
CRAIN: Be more specific?
WATSON: What is Denver boot?
CRAIN: That is correct.
This African-American folklore laborer:
(continues reading):
Watson?
WATSON: What is John Henry?
That is correct. Select again, Watson.
NARRATOR: It may appear that Watson has redeemed himself,
but the producers are troubled by his erratic performance.
Their verdict: Watson isn't strong enough for Jeopardy!
At least not yet.
(beeping)
Why is Watson so erratic?
To understand his weaknesses,
you have to appreciate the complexity of the task.
Consider this clue...
The correct response is "What is The Matrix?"
But how can Watson figure that out?
First, he breaks down the clue into grammatical parts,
identifying key words and phrases.
Then, Watson's powerful search engines churn
through millions of documents,
including the Internet Movie Database.
What we do next is we take these documents
and we pull out candidate answers.
And we'll pull out, okay, Keanu Reeves.
That could be a candidate.
We'll pull out Nokia, we'll pull out The Matrix.
NARRATOR: Other movies starring Keanu Reeves
also become possible answers.
GONDEK: We'll pull out The Matrix 2, we'll pull out Speed,
Bill and Ted's Excellent Adventure, all this stuff.
Whoa!
NARRATOR: And Watson pulls out other famous sci-fi flicks,
like Blade Runner.
It generates hundreds of possible answers.
NARRATOR: With hundreds of choices,
how can Watson pick the one answer that's correct?
FERRUCCI: Next thing that Watson is going to do,
it's going to take those answers and say,
"Well, let's assume all of them might be right."
So these are its competing hypotheses.
NARRATOR: Watson starts considering evidence
for and against each candidate,
using rules like "a movie is sometimes called a flick."
GONDEK: And we'll look at things like,
well, it's looking for a flick.
Is this candidate answer a flick?
Is The Matrix a flick?
Yes.
Is Speed a flick?
Yes.
Is Keanu Reeves a flick?
No-- so we're starting to learn something.
NARRATOR: Within a matter of milliseconds,
Watson analyzes every possible answer
in hundreds of different ways
and scores each piece of evidence
behind every answer in the list.
That's a lot of scores.
GONDEK: The problem is, you have all these different scorers
and they don't agree.
Some of the scorers are going to say
"The Matrix" is the right answer.
Some of the scorers are going to say
"Keanu Reeves" is the right answer.
Some are going to think Matrix 2 is the right answer.
NARRATOR: And a lot of scorers think
"Blade Runner" is the right answer,
because it shows up so often as a sci-fi flick.
So you need someone at the end
to listen to all these different votes
and decide what's going to be the best answer.
NARRATOR: This is where Watson's machine learning kicks in.
Having studied thousands of other Jeopardy! questions
and their correct answers, Watson has learned
what evidence is important and what's not.
What machine learning will start to do is learn
how to weigh them differently and say,
weighing questions like this,
calling on a phone, not calling on a phone-- not so important.
This other stuff... Do I have a sci-fi movie?
Is the person named a character in that movie?
Very, very important for questions like this.
NARRATOR: In this case, he successfully weighs the evidence
and identifies sci-fi flicks from 1999,
starring Keanu Reeves.
So he picks the one answer matching all those elements--
"The Matrix."
Watson's elaborate system doesn't always work,
but without machine learning, he wouldn't stand a chance.
Machine learning isn't just important for Watson.
It's driving a revolution in computing.
It plays a major role
in computer models that predict the weather days in advance.
And all those recommendations you get from Amazon or Netflix?
No human is writing up rules about your likes and dislikes.
Instead, computers are comparing your preferences
to millions of other customers' and finding patterns
and learning about you.
Today machine learning is conquering many problems
once thought too complex for computers,
like speech recognition.
TOM MITCHELL: In the earlier days,
people decided that they would try to program computers
to recognize speech-- "Which word am I saying now?"
Pick up the big block
at the right side.
NARRATOR: In the '60s, this voice-directed block world
was the height of technology.
Computers could be programmed to recognize the audio signals
of specific words and phrases.
Pick up every small block.
NARRATOR: But they had to be reprogrammed for every new speaker
because everyone's speech is slightly different.
Even though it's very easy for you and I to recognize
the word "ice cube"...
Ice cube.
It's very difficult for us to write down the rules
that would allow a computer to look at the microphone signal
and see that it's "ice cube."
NARRATOR: But now, computers are trained
with millions of examples of human speech.
MITCHELL: Here's the microphone signal and this is the word "ice cube."
Ice cube.
MITCHELL: Here's another one...
Ice cube.
You end up
with much more successful speech recognition systems.
NARRATOR: Today, speech recognition software, though not perfect,
is remarkably accurate and getting better all the time.
All the ones we have today are based on machine learning,
simply because that works the best.
NARRATOR: And some programs are taking it a step further.
Where do you come from?
NARRATOR: Not only transcribing your speech...
(speaking Chinese)
NARRATOR: ...but translating it into a foreign language as well.
COMPUTER: Shanghai.
It's very nice to meet you.
(computer translating into Chinese)
NARRATOR: Every language has so much ambiguity--
double meanings and metaphors.
What are your specials today?
NARRATOR: Accurate computer translation seemed impossible
with the old rules-based approach.
(speaking Chinese)
COMPUTER: Today's special is roast beef fried rice.
All of these interactions are so complex
that you couldn't in your lifetime write all these rules.
It's just too enormous, too daunting an effort.
My wife asked me to buy some crackers.
(speaking Japanese)
COMPUTER: What kind of crackers?
Rice crackers.
NARRATOR: Alex Waibel fed a computer millions of examples
of English text, together with their translations,
into about a dozen different languages.
Now he's got a program that can run on your phone or iPad.
Do you often go shopping here?
NARRATOR: And translates on the go.
(computer voicing translation in Japanese)
(speaking Japanese)
NARRATOR: Machine learning has been so successful,
mastering more and more tasks once only done well by humans...
The computer program translates between languages.
NARRATOR: Some researchers believe it may be a crucial building block
for making real artificial intelligence.
(computer voicing translation in Japanese)
There are two ways of building intelligence.
You either know how to write down the recipe
or you let it grow itself.
And it's pretty clear that we don't know
how to write down the recipe.
Machine learning is all about giving it the capability
to grow itself.
NARRATOR: Some people find the idea
of a machine that can learn threatening.
But when Watson's having a bad day,
it's hard to imagine him threatening anyone.
Did you see the Daily Double?
No. So we were way ahead, we were almost locking out.
CRAIN: The other Daily Double!
We get the Daily Double, there's like two or three clues left.
So what do we do?
We bet big so we can lock 'em out.
WATSON: I'll wager $5,200.
(chuckling): Well...!
NARRATOR: The team is nervous.
They know Watson will never make the cut on Jeopardy!
unless he can stop making dumb mistakes.
Here's your clue.
It was about letters
and it was, "a woman wrote to this '40s artist."
CRAIN (reading):
We answered with "Rembrandt."
WATSON: Who is Rembrandt?
Really?
NARRATOR: Although Watson recognizes most dates,
he doesn't know that the '40s refer to the 1940s.
It's a '40s artist, we answered with Rembrandt.
There's a time... a time problem.
We all know that it was Jackson Pollock.
NARRATOR: Watson loses that game.
We got the Daily Double wrong and Final Jeopardy!
NARRATOR: And his opponents show him no mercy.
We both beat him!
Good for you!
Humans! Whoo!
Sometimes the reason something is the right answer
is very obvious to a human,
like, for example, it may be asking for a "she" or a "he."
CRAIN (reading):
Watson?
WATSON: Who is Richard Nixon?
(laughter)
Oh, here you go. Right?
CRAIN: Patricia?
Who is Pat Nixon?
CRAIN: That's correct.
Richard Nixon was never a first lady.
I want to understand what went on there.
Our new gender stuff is not in the system.
It's not in the system.
CHRIS WELTY: People take offense at being called the wrong gender.
Watson doesn't care about stuff like that.
It's making statistical judgments
based on how different pieces of evidence have gone together
in questions and answers that we've given it.
CRAIN (reading):
WATSON: What is Jimmy Dur-an-tay?
More specific?
WATSON: Sorry, all I know is, "What is Jimmy Dur-an-tay?"
Saying it slower doesn't make it right.
FERRUCCI: Now you hear how Todd Crain makes fun of the computer?
You know, I had my kids...
I had them sign the confidentiality agreement
and come in and see a couple of these games
and their comment was,
"Why does the host make fun of Watson, Daddy?"
This "What are you doing?" website's name also refers
to a type of nervous laugh.
CRAIN: Watson?
WATSON: What is evil laugh?
Oh, no!
In terms of comedy duos,
he is the best straight man in the business.
You're gonna kick yourself... Twitter.
CRAIN: Because he doesn't... he doesn't get it.
He doesn't get why his inappropriate answer is funny,
and you can't ask for better writing than that.
Once in a while, okay, we can all take a joke,
but over and over again, and Watson's defenseless, right?
So he's making fun of and criticizing
a defenseless computer
that represents people with real feelings, real families.
All right, maybe if I don't have any feelings,
my kids have feelings.
NARRATOR: And feelings are intensified by the fact
there's just one more month
before the Jeopardy! producers will return to make a decision.
Watson, you have control.
NARRATOR: The pressure is on to boost Watson's strengths
and eliminate his weaknesses.
His strengths are obvious.
He dominates when it comes to purely factual questions.
BROWN: Watson usually does very well at these factoid questions,
looking up facts-- history, geography, entertainment.
CRAIN (reading):
CRAIN: Watson?
WATSON: Who is Victor Flemming?
CRAIN: That is correct.
Watson. You choose again.
WATSON: For 1600, please.
CRAIN (reading):
Watson?
WATSON: Who is John Ford?
Good for 1600.
WATSON: Who is Mike Nichols?
CRAIN: Good for 12.
Last clue on the board for $800. Here it is.
(reading):
Watson?
WATSON: What is The Pink Panther?
I just want to check-- your buzzers are working?
(contestants laugh)
Just wanted to check in and wake you from your nap.
NARRATOR: Watson has made it into the cloud,
but nowhere near championship level.
Ferrucci must come up with a plan
to somehow step up his performance
before the Jeopardy! producers return.
The team has spent almost four years,
enormous intellectual and emotional energy,
and tens of millions of dollars on developing Watson.
But all this effort isn't just for a machine
that can play Jeopardy!
IBM has much bigger goals.
FERRUCCI: I'm already looking sort of beyond Jeopardy!
I'm thinking, where can we go from here?
NARRATOR: Ferrucci imagines a day when Watson might perform
much like the computer in Star Trek.
The captain just starts talking to the computer,
and they say, "Computer."
CAPTAIN KIRK: Computer.
COMPUTER: Ready.
Could a storm of such magnitude cause a power surge
in the transporter circuits...
FERRUCCI: It's an information seeking tool
that's capable of understanding your question.
KIRK: ...creating a momentary interdimensional contact
with a parallel universe?
FERRUCCI: And dialoguing with you to make sure that you get what you want.
COMPUTER: Affirmative.
Do you have one of those?
I... I don't have one of those, right?
NARRATOR: You don't need to be
a starship commander to find this helpful.
In a world overflowing with data,
intelligent expert systems that can answer vital questions
have been the Holy Grail.
Now, we could be close to a Watson, MD.
Think of it this way.
There's a bunch of information--
new diagnosis, new treatment options, new discoveries.
Can anybody keep that all in their head all at once?
NARRATOR: A machine that could access and organize all that information
could help doctors analyze symptoms
and wade through piles of medical journals.
FERRUCCI: It changes the paradigm in which we work with computers.
That's the vision.
WATSON: How am I doing?
I hope we will have a good game today,
but first I have to test my voice...
NARRATOR: Before that vision can be fulfilled,
before Watson can even compete on Jeopardy!,
the team needs to get more bugs out of his system.
One of Watson's most embarrassing weaknesses
is he cannot hear.
Instead, Watson receives each Jeopardy! clue
as an electronic text message
at the same moment his competitors see it on the board.
As a result, he doesn't know the other contestants' answers.
CRAIN (reading):
Bill?
What's a mosquito?
CRAIN: No.
Watson?
WATSON: What is mosquito?
CRAIN (laughing): No.
Harvey?
What's a horsefly?
CRAIN: Yes-- thank you for not saying mosquito.
Good job, good for $2,000, Harvey.
NARRATOR: Ferrucci and the team have been working furiously
to boost Watson's performance.
As part of their plan, Watson will now receive correct answers
electronically after they're revealed.
If the fix works, it will be in the nick of time.
The Jeopardy! producers are back,
and they're about to determine Watson's fate.
This was a measure of our progress, and we wanted to hear,
"Yeah, you're there, you've made it."
NARRATOR: The new and improved Watson gets his first big test
with a category called "Celebrations of the Month."
CRAIN (reading):
Watson?
WATSON: What is holiday?
No. That's not even close, really.
NARRATOR: Watson fails, because he doesn't get
that in this category,
the answer must be the name of a month--
something his human competition quickly figures out.
CRAIN: Arthur?
ARTHUR: What is April?
CRAIN: What is April?
April 18th is...
We don't understand the question.
We don't understand the category, basically.
CRAIN (reading):
NARRATOR: But now he electronically receives
these correct responses.
CRAIN: Arthur? What is June?
Good for 200.
NARRATOR: Can he learn from the answers?
GONDEK: What Watson does here is,
it sees that all the answers I've seen have been the month
in which this thing in the clue occurs.
CRAIN: Matt.
What is November?
Good for four!
So then it knows in the next clue to look for, um,
what month does this thing occur in?
MATT: Celebrations for six.
CRAIN: Celebrations for six.
(reading):
Watson?
WATSON: What is May?
GONDEK: Yes! He got it!
CRAIN: Very nicely done, Watson.
He figured it out.
It took us four.
Yeah, yeah.
NARRATOR: Thanks to the team's efforts,
Watson is soaring higher in the cloud
and now approaching the level of champions like Ken Jennings.
Watson is on a roll.
Choose again.
WATSON: "You're Tripping," for 1600.
CRAIN (reading):
Watson?
WATSON: What is Montreal?
Yes.
WATSON: Who is Zebulon Pike?
CRAIN: Good.
WATSON: What is Providence...
Aquarius... Texas... Beau Brummell?
CRAIN: Watson?
WATSON: What is "Ciao"?
That was right and cute all at the same time.
NARRATOR: The Jeopardy! executives have seen enough.
HARRY FRIEDMAN: I think we've gone
from impressed to blown away.
Very nicely done, Watson.
NARRATOR: Finally, Watson will get his chance on Jeopardy!
A computer playing against human champions.
ALEX TREBEK: They say it can think.
NARRATOR: In a game that is a very symbol of intelligence.
But can it think like a Jeopardy! champion?
NARRATOR: It will be a contest the world has never seen.
Good luck, Watson.
NARRATOR: But does this mean
that the dream of artificial intelligence is coming true?
BROOKS: So artificial intelligence,
to me, is trying to get computers to do stuff
that if people did them,
you'd say, "Oh, they're demonstrating their peopleness.
That's what makes humans humans, that stuff they're doing."
NARRATOR: But without experience or emotion,
can a computer like Watson ever learn and understand the world
the way we humans do from early childhood on?
Right now, no machine can understand
the meaning of a play,
what it means to be King Lear or Macbeth or Hamlet.
LAURENCE OLIVIER: To be... or not to be.
That is the question.
No machine can understand
the parables of the Judeo-Christian Bible.
All they can do is grovel through data
and find regularities.
NARRATOR: But for Dave Ferrucci,
that kind of understanding was never the goal.
It's not going to emerge as a human,
because it doesn't connect the information to human experience,
to human cognition.
When you think about a great symphony,
and when a human sits down with that,
that music is affecting that human at an emotional level.
The computer doesn't have that human experience,
doesn't have that human emotion.
It's not human, it's a computer.
NARRATOR: Watson may never experience the world the way we do,
but with his enormous knowledge base,
his skill at interpreting language
and his ability to learn...
Yes, he got it!
WATSON: What is May?
He figured it out.
NARRATOR: Could he actually be considered "intelligent"?
Oh my God.
It is more intelligent than the average Jeopardy! player
in answering Jeopardy! questions.
That's... impressively intelligent.
NARRATOR: The time has come for Watson to take the stage,
where his intelligence will be put to the ultimate test
in front of millions of Jeopardy! viewers.
Am I having fun?
It's nerve-wracking.
NARRATOR: For the big match, Watson now has
a physical presence, an avatar.
WATSON: My name is Watson.
How now brown cow?
NARRATOR: The team has been working right up till the end.
I think I dream about Jeopardy! questions now,
I have nightmares about Jeopardy! questions.
I talk to people in the form of a question.
NARRATOR: Have they done enough?
They're about to find out,
as Watson meets the world's two best Jeopardy! players:
Brad Rutter and Ken Jennings.
GONDEK: We've never had
this caliber of player.
And there's a reason why Ken won 74 games in a row.
There's a reason why Brad has never been beat...
by a human.
NARRATOR: The whole team has been waiting four years for this moment.
FERRUCCI: I knew this was going to happen,
but I never imagined quite like this.
What do you say we play Jeopardy!?
NARRATOR: With another stand-in host, Watson meets his opponents
for an exhibition round.
HOST (reading):
HOST: Watson?
WATSON: What is Jericho?
HOST: Correct.
WATSON: 400. Same category.
HOST: "This mystery author and her archaeologist hubby dug
in hopes of finding the lost Syrian city of Urkesh."
HOST: Watson?
WATSON: Who is Agatha Christie?
HOST: Correct.
NARRATOR: Watching from the wings
is Jeopardy!'s host, Alex Trebek.
He doesn't get everything right, but he doesn't miss very much.
HOST: Watson?
WATSON: Let's finish "Chicks Dig Me."
(audience laughing)
HOST (reading):
HOST: Ken?
What is Neanderthal?
HOST: You're right.
NARRATOR: By the end of this ten-minute game,
man and machine are neck and neck.
And no one dares predict what will happen
when they face off in the looming showdown.
FERRUCCI: It's going to be edge-of-your-seat.
It's going to be nerve-wracking.
What really is going to happen?
And you just don't know, you don't know.
TREBEK: I suspect that this will just be the jumping-off point.
Their next project will be...
We don't want to create an avatar that will play
as a contestant on Jeopardy!
We want to create the host of Jeopardy!
And they can do it.
WATSON: Good afternoon, Mr. Trebek.
I've been waiting for this moment
for a very, very, very long time.
The exploration continues on NOVA's website,
where you can find a Q&A
with Watson team leader David Ferrucci,
see other smart machines transforming our world,
and hear what top computer scientists have to say
about the future of artificial intelligence.
Dig deeper into technology and engineering
with expert interviews, interactives, teacher resources
and more.
Follow NOVA on Facebook and Twitter
and find us online at pbs.org.
Next time on NOVA...
In stormy skies, more than 300 miles from land...
(alarm bell ringing)
Flight 447 plummets into the sea.
MAN: Almost unheard-of series of failures,
one right behind the other.
NOVA and a team of independent investigators
unravel the mystery of what went wrong.
MAN: We need to understand what happened
that night out over the Atlantic.
"The Crash of Flight 447," next time on NOVA.
Major funding for NOVA is provided by:
And...
And by the Corporation for Public Broadcasting
and by contributions to your PBS station from:
Captioned by Media Access Group at WGa AcH access.wgbh.org
This NOVA program is available on DVD at shopPBS.org.
Or call 1-800-play-pbs.