Althouse: The computer beats the nerds.

February 17, 2011

The computer beats the nerds.

On "Jeopardy!"

56 comments:

Scott M said...: A singularity moment? Do we glimpse emergent complexity? Did the Terminator movies not teach us anything?

I'm not all that worried, though. We don't have enough money left in this teetering nation to fund the research and construction of AI. So, when that groping, metallic claw drags you and your family from the rubble, it will probably be yelling Chinese at you.; 2/17/11, 8:29 AM
Original Mike said...: Watson cheated.; 2/17/11, 8:32 AM
Original Mike said...: Watson has machine like reflexes on the button. Not sure that's fair.; 2/17/11, 8:34 AM
Lincolntf said...: Let's see how Watson does against Sean Connery before we declare the Age of Man over.; 2/17/11, 8:37 AM
X said...: Watson is a threat to our way of life; 2/17/11, 8:39 AM
William T. Sherman said...: I wasn't impressed. The only reason the computer won is because it was able ring in faster than the humans. If three contestants all have about the same level of knowledge the fastest on the buzzer will always win. Machines can always be quicker, it doesn't mean they're better or smarter.

Unless the U.S. has annexed Toronto in the last week, and I haven't heard about it yet. If so, please ignore the above.; 2/17/11, 8:39 AM
Anonymous said...: Ray Kurzweil has been predicting for some time that computer intelligence would exceed human intelligence in, I don't know, somewhere around 2020.

He based his predictions on Moore's Law.

Kurzweil is a genius. He invented the Kurzweil PC2X, a synthesizer keyboard that I own. It has an 88 key, fully-weighted keyboard, and sounds about as close to a real concert grand piano as you can get.

Still, I think that Steve Sailer hit the nail on the head on this one. The crux of Jeopardy is the timing of hitting the response button. So, the computer didn't win on intelligence. It was better able to time the response.; 2/17/11, 8:40 AM
Mary Beth said...: Ken was gracious in welcoming our new computer overlords.; 2/17/11, 8:40 AM
Paddy O said...: I'll admit to embarrassing ignorance here. Could someone say why this is such a big deal?

I don't get it, and not simply because I have a jeopardy game for my computer and it sometimes beats me.

More, I already go to the internet to answer just about any question I have on just about any topic. Shouldn't a computer, then, have the ability to store all kinds of data?

Is the amazing thing here a matter of the computer understanding the questions and responding quickly to topics not predetermined?; 2/17/11, 8:41 AM
Paddy O said...: Maybe Wisconsin can hire Watson to teach its students while the teachers are out sick.; 2/17/11, 8:42 AM
Original Mike said...: Watson also weighs a ton. I'll be more impressed when they get him down to 3 pounds and he still wins.

I do think it's an amazing achievement. Language is damn difficult.; 2/17/11, 8:44 AM
rhhardin said...: Artificial Intelligence was thoroughly debunked by Coleridge in chapters 5-9 of Biographia Literaria, long ago.

Motto: matter has no inwards.; 2/17/11, 8:47 AM
Automatic_Wing said...: Kind of a cool parlor trick, but it's not really artificial intelligence. It's just an enormous database that correlates key words really fast.; 2/17/11, 8:49 AM
Anonymous said...: Now, if the computer can get sex, it'll have yet another advantage over the nerds.

Peter; 2/17/11, 8:49 AM
Hoosier Daddy said...: Watson cheated

I'm not much for conspiracy stuff but I watched that Ken guy for weeks and I honestly believed something wasn't right about that whole thing. I mean the guy knew everything. No one was ever coming close to him. I mean the guy was answering questions about 17th century French lit and then loses on what company's employees work only four months a year and he answers Fed Ex? Seriously?

I still think he was a ratings ploy.; 2/17/11, 8:53 AM
Original Mike said...: @Maguro: How does the brain work?; 2/17/11, 8:53 AM
Anonymous said...: Now, if the computer can get sex, it'll have yet another advantage over the nerds.

Where would you place the public hair on the computer, Peter?; 2/17/11, 8:54 AM
Scott M said...: It's just an enormous database that correlates key words really fast.

It's not really a human rendering machine, it's just an enormous self-aware ex-thresher that's got self-repairing parts and an innate sense of self-preservation, just harvesting feral humans for the power plant.; 2/17/11, 8:54 AM
Hoosier Daddy said...: Watson also weighs a ton

If Trooper were awake he'd say knock it off with the fat jokes.; 2/17/11, 8:54 AM
Anonymous said...: "Computer beats the Nerds"

The modern version of Oedipus. But who is Jocasta?; 2/17/11, 8:56 AM
Hoosier Daddy said...: Did the Terminator movies not teach us anything?

Yep. If your name is Sarah Conner, you might want to change it.; 2/17/11, 8:56 AM
Scott M said...: Yep. If your name is Sarah Conner, you might want to change it.

No good. It has detailed files.; 2/17/11, 9:00 AM
Anonymous said...: shoutingthomas -

Digital merkins?

(Oh, joy! Another reason to use the word merkin on Althouse. Those of you who haven't been around for very long, click here.); 2/17/11, 9:01 AM
SteveR said...: Yeah I don't get it, a huge database knows more facts or can access them quicker than a human. Well I would hope so.

Intelligence is like Paul Krugman or Ezra Klein not the atomic weight of Strontium.; 2/17/11, 9:04 AM
G Joubert said...: They should take a random sample of winning rings by humans, determine the average time elapsed between end-of-question and hitting the button among them, then write an algorithm that gives the computer that speed. Not hard.; 2/17/11, 9:08 AM
Original Mike said...: They wouldn't let a contestant ring in until after Alex had finished reading the question (the buttons were inactive). Is that how Jeopardy usually works?; 2/17/11, 9:12 AM
Hoosier Daddy said...: No good. It has detailed files.

I think it was on IMBD where I read that after T2, Arnold claimed that was the most grueling movie he ever shot and his one line at the end 'I need a vacation' was an ad lib that they kept in.; 2/17/11, 9:20 AM
Automatic_Wing said...: They wouldn't let a contestant ring in until after Alex had finished reading the question (the buttons were inactive). Is that how Jeopardy usually works?

Yeah, it is. Buzzer skills are huge in Jeopardy because a lot of the questions (answers?) are really not that difficult. I would imagine that a large percentage of the questions are known by all 3 contestants and it's simply a matter of who buzzes in first.; 2/17/11, 9:22 AM
Original Mike said...: @Maguro: I wonder why they do that. Seems to me it would be a better game if you could ring in as soon as you knew the answer. Maybe the strategy then would be to ring immediately and bet that you could answer the question. But that could be fixed by making the penalty for a wrong answer higher.; 2/17/11, 9:27 AM
PaulV said...: Mike, also if you rings in too early there is a time delay penalty before your button can work. Humans are not allowed reference material like a computer.; 2/17/11, 9:28 AM
Original Mike said...: "Mike, also if you rings in too early there is a time delay penalty before your button can work."

Interesting. My impression from watching the last three days was that the humans often looked like they knew the answer but couldn't beat Watson to the buzzer. I'm still impressed with IBM's achievement, but to really test if Watson is better than humans, you'd need to restructure the experiment somehow to eliminate the buzzer issue.; 2/17/11, 9:33 AM
jr565 said...: I lost on Jeapordy, baby, ooh ooh ooh ooh...; 2/17/11, 9:37 AM
jr565 said...: Paddy O wrote:
Is the amazing thing here a matter of the computer understanding the questions and responding quickly to topics not predetermined?

That and also that I think it was responding to the spoken question so can "understand" language. Then again, maybe it simply was able to take a screen shot of the spoken question, convert that to text, type that text into a search bar and access the info using a faster than average computer. And was doing all that stuff faster than than it took Alex to read the question.
You're right though. It's not that much more impressive than doing a google search and coming up with search results; 2/17/11, 9:42 AM
Original Mike said...: @jr: My understanding is that Watson was sent the question as a text file.

"It's not that much more impressive than doing a google search and coming up with search results"

I disagree. The program has to "understand" the search results. That is a big deal.; 2/17/11, 9:46 AM
Paul Brinkley said...: Watson gets questions as text, not by recognizing Trebek's speech. (The contestants can see the text too, if I'm not mistaken; dunno how easy it is to read from where they stand though.)

This led to at least one obvious screw-up Watson made. Since Watson doesn't recognize speech, it doesn't recognize answers given by other contestants. A question came, a contestant rang in and asked "What were the 20s?", got it wrong, Watson rang in next, and gave the exact same question.; 2/17/11, 10:19 AM
Paul Brinkley said...: In the whole, this counts as a Great Moment in Singularity, but not the moment. As said above, Watson weighs much more than a human brain, draws much more than the 12 watts of power the brain does, and uses 12 terabytes of RAM. For those of you who aren't computer nerds, this is random access memory, analogous to the brain's short-term memory, as opposed to the long-term memory that takes a lot more time to access, and is much larger. That would be like Watson's hard drives.

For comparison, most desktop computers probably have about 2 gigabytes of RAM - about 1/6000 of what Watson has.; 2/17/11, 10:24 AM
Jim Gust said...: Not shown, Watson crashed so often that it took 4 hours to tape the three half hour shows. Had they not allowed Watson the timeouts for rebooting, it would have lost.

No doubt the buzzer rules favored Watson, and should have been changed. Also, the humans had to split the questions that Watson didn't know the answers to, which amounted to a meaningful number. Had Ken Jennings played against two Watsons (splitting the answers the computer was good at) he would have won.

Finally, Trebeck was very lenient, giving Watson credit for some partial answers that would get no credit in normal Jeopardy.

Still and all, a good show.; 2/17/11, 11:06 AM
Original Mike said...: "This led to at least one obvious screw-up Watson made. Since Watson doesn't recognize speech, it doesn't recognize answers given by other contestants. A question came, a contestant rang in and asked "What were the 20s?", got it wrong, Watson rang in next, and gave the exact same question."

I think that was in practice runs, before this week's contest. It's my understanding that they fixed that limitation, and that in the contest this week, Watson knew the other responses. Not sure how they did it, though.; 2/17/11, 11:12 AM
Original Mike said...: "Not shown, Watson crashed so often that it took 4 hours to tape the three half hour shows."

Really? So I guess Watson won't be replacing airline pilots any time soon.; 2/17/11, 11:13 AM
Original Mike said...: "Also, the humans had to split the questions that Watson didn't know the answers to, which amounted to a meaningful number."

Good point.; 2/17/11, 11:14 AM
traditionalguy said...: But the nerds can figure out Watson's trick and design and build themselves a button pushing electronic aid just like nerds designed and built the one inside Watson.; 2/17/11, 11:21 AM
J said...: Watson's more pseudo-AI hype (and/or advertiser's gimmick). Merely processing data, however rapidly or correctly, does not at all suffice for human intelligence.

Watson appears to be just word-recognition software attached to a massive DB, with some quasi-logic programs attached (optimize!). Were that enough for intelligence any data-base--or say Deep Blue-- would be conscious. Now, had Watson say taken over the show--and network-- and had some automatons seize Trebek and the contestants--; 2/17/11, 11:31 AM
Lex said...: I am frightened that computers access information more quickly than humans! Because that is not at all why I use computers in the first place!; 2/17/11, 11:36 AM
Scott M said...: I am frightened that computers access information more quickly than humans! Because that is not at all why I use computers in the first place!

Good snark. You'll need a sense of humor like that when they burn a barcode into you and ship you off to a gulag with the rest of the damned, dirty humans.; 2/17/11, 11:39 AM
Tibore said...: I really don't see what the big deal is. Computers have always been better than humans at some tasks, the drudgery of accessing randomly presented information stored in a database being one of them. Watson is simply a different front end query interface.

As someone wrote in a CNN.com article, though:

"Even more frustrating is knowing that you will never get to ask the question you really want to ask, as it isn't an option: "If I drive to Newark and board my Flight to Tel Aviv there will you cancel my whole trip, as I haven't started from my ticketed airport of origin, Ithaca?"

A human would immediately understand the question and give you an answer. That's why knowledgeable travelers rush to the nearest airport when they experience a cancellation, so they have a chance to talk to a human agent who can override the computer, rather than rebook by phone (more likely wait on hold and listen to messages about how wonderful a destination Tel Aviv is) or talk to a computer...

... Providing unique answers to questions limited to around 25 words is not the same as dealing with real problems of an emotionally distraught passenger in an open system where there may not be a unique answer.

Watson beating the pants out of us on "Jeopardy!" is fun -- rather like seeing a tractor beat a human tug-of-war team. Machines have always been better than humans at some tasks.

A slide rule could do better than humans at some calculations. It is no big deal."

I work in IT. The ability to leverage computing systems to magnify the work a human does is awesome. But note how I'm phrasing that: Magnify the work a human does. It's true that a computerized system can be developed to actually generate tasks and, to a degree, originate within a set of parameters, but that ability so far will have been originally generated and developed by a human. If we want an example of what a computerized system is best at, read Robert Shaler's "Who they were. Inside the World Trade Center DNA story: The unprecedented effort to identify the missing". While the computerized DNA systems were frustrating and created headaches for them, those systems also allowed them to magnify their abilities at DNA identification of remains in order to tackle an unprecedentedly monumental task. That's what computers are best at.

You want to know what computerized systems are bad at? What they're not specifically designed to tackle. They're possibly the best tool mankind ever created because of the incredible flexibility of application they represent, but ultimately, they're a tool. They agument humanity. At this stage of their development, they obviously cannot replace humans, Second Life and The Sims nonwithstanding ;-). And sure, Watson may be an ass kicker at Jeopardy, but when "he" leaves, does he pick up his "normal" job of serving coffee, teachning schoolchildren, or fixing cars? No. Being an excellent database querier was his "job". Being on Jeopardy was simply a new face on it. Humans still get the last laugh when they go home from the show.; 2/17/11, 12:57 PM
Scott M said...: Humans still get the last laugh when they go home from the show.

Humans will in fact have the last laugh when the last human tells the last funny joke right before being crushed under articulated buckycarbon claws and fed into a biohopper.; 2/17/11, 1:02 PM
Tibore said...: As long as I go out getting that one last fling with the hot & curvy fembot, I won't whimper.

... Well... Ok, maybe I will, just a little bit...; 2/17/11, 1:35 PM
traditionalguy said...: The Krell who built Forbidden Planet's computers are my heroes. And they did not even have nano-bots. But I do worry... the Constitution requires "natural born" Presidents, but openly allows droids on the SCOTUS which as we know controls everything.; 2/17/11, 2:22 PM
Scott M said...: but openly allows droids on the SCOTUS which as we know controls everything.

The Framers were going to include specific language to forbid just such an occurrence, but used the more classical Greek reference of "attendants" (see Hephaestus). Skynet dropped a T1000 into the mix, though, and got them to take it out somehow. I've not been able to pin down which one it "replaced", but my money is on Hamilton.; 2/17/11, 2:26 PM
Tibore said...: This comment has been removed by the author.; 2/17/11, 2:37 PM
Tibore said...: Hamilton my ass. It was Jefferson.

Look at the guy. I mean, just look at him!; 2/17/11, 2:37 PM
Scott M said...: Hamilton my ass. It was Jefferson.

Look at the guy. I mean, just look at him!

Touche. That solves the whole mystery about a T1000 being taken out by a pistol-fired musket ball too. Who are you, Dan Brown?; 2/17/11, 2:53 PM
Tibore said...: Naw. Just an observant student of cyborg history.

Speaking of T-1000's, you notice anything strange about Trebek lately?...; 2/17/11, 3:08 PM
Fred Drinkwater said...: Interesting show, but where the hell did "Toronto" come from?
I'll be REALLY interested when the whole apparatus weighs 160 pounds, is self-powered during the show, and consumes no more than 150 watts.

WV: excesu - what Bill Joy had before he left UC Berkeley.; 2/17/11, 3:25 PM
raptros-v76 said...: The accomplishment here is that they've built a natural language processing system that can generate answers in very short periods of time.
Listen, processing human language computationally is _difficult_. All we've got in the computational linguistics field is a couple of probability tricks and crossed fingers. We have no idea how to get word meaning right now. Parsing syntax doesn't work very well. Watson is backed by a huge pile of english language text. In order to answer a question, it has to look at the statistical correlations between various features of the question and various features of the data it has on its index. The big feat of the Watson system is that it is capable of running a battery of analysis tools (that analyze different features of the texts it is working with) in a very short period of time. Those fast reflexes it has? Yeah, that's amazing to me as a student of computational linguistics. Watson doesn't hit the buzzer until it it comes up with an answer that it has given a confidence value above a certain threshold. It's working with gigabytes of natural language text (including all the text in wikipedia), and it was capable of getting answers in faster than reasonable time. It's not working with pre-built knowledge databases, it's working with very basic indexes. (The knowledge-base approach was abandoned in the late 80's as being pointlessly difficult to construct). All it's got is a bunch of probabilities connecting some basic features of the text data it was loaded with, and it's pulling answers out of that faster than a human. Pardon me for saying this, but holy shit, it's fast.
To sum up: IBM's figured out how to build a question answering system that can answer human language questions by analyzing it against a store of human language text with high speed. You better believe this is a big deal.; 2/17/11, 10:40 PM
raptros-v76 said...: Sorry about the long comment above. BTW, for anyone who's interested in Toronto answer, IBM has an interesting analysis on the Watson site. I can't get a direct link, but if you click on the "What is Toronto" thing on the upper right, it'll get you there.
Also, apparently the Watson system might be useful for legal work.; 2/17/11, 10:49 PM