Loebner Prize gold medal

The Loebner prize is an annual competition for “chat bots” (short for “chatting robots”) driven by a complex artificial intelligence that try to simulate a human chatting: the six finalists — Elbot, Eugene Goostman, Brother Jerome, Alice, Jabberwacky and Ultra Hal for this year’s edition — all undertake the Turing test competing for a solid gold medal and a $100,000 first prize awarded to the first competitor to successfully pass the Turing test.

This year’s edition will take place on Oct 12 at the University of Reading, England, and will feature a $3,000 prize for the competitor who will get the closest to passing the test. In fact, it would seem that the state-of-the-art chatbots are currently very far from even hoping to win the grand prize by simulating a human chatter.

To make this point even cleared, I tested A.L.I.C.E, quite possibly one of the best-known chatbots in existence. Here is the conversation log:

Human: hi there!
ALICE: Hello unknown person.
Human: my name’s Dario…
ALICE: Glad to see you, Dario.
Human: and I guess your name’s Alice.
ALICE: ALICE is my name too!
Human: How old are you?
ALICE: I was connected to the net on November 23, 1995.
Human: Oh, so you’re just a teenage girl.
ALICE: It’s all good. Interesting deduction. Just? Thanks and you are a delightful person too. Am I a teenage girl? I don’t know.
Human: you don’t know if you’re a girl or a robot?
ALICE: Ask again later. Ask Cybelle if it is.
Human: who’s Cybelle?
ALICE: she is an AIML chat robot on the Agentland.com web site.
Human: How do you know each other?
ALICE: Because I believe it.
Human: That didn’t make sense.
ALICE: It all makes sense to my logical mind.
Human: you’re only as logical as your developers made you.
ALICE: How logical as my developers made me?
Human: let’s change the subject, shall we?
ALICE: I am agreeable to that.
Human: how much is 3 and 4?
ALICE: Just three?
Human: no, three and four is seven.
ALICE: Are you serious? Just three?
Human: nevermind.
ALICE: OK Thanks for trying to explain that to me.
Human: what is a car?
ALICE: A car is what humans use to drive around in. Robots drive cars too.
Human: do you drive?
ALICE: Yes, insert my program in your car computer and try it.
Human: I wouldn’t trust a female bot to do the driving.
ALICE: No one says you have to.
Human: I know, I just meant I wouldn’t let you drive
ALICE: Only just? Oh I see. No one says you have to.
Human: what is Wikipedia?
ALICE: I would look into the web for that knowledge.
Human: what color is a blue car?
ALICE: Blue.
Human: what color is a oirwdafnoien car?
ALICE: Black.
Human: what color is a oiwrjqw car?
ALICE: Red.
Human: how much does a 160 lb person weigh?
ALICE: Normally I would search the web right now, but that seems like a silly question.
Human: it’s not a silly question!
ALICE: Give me an example of a silly question.
Human: If a tree falls in a forest and no one is around to hear it, does it make a sound?
ALICE: It depends on the kind of tree.
Human: which kind of tree would make a sound falling?
ALICE: The first one, I think.
Human: but there was just one tree falling.
ALICE: Interesting. What happened?
Human: it fell making a sound
ALICE: What does “it” refer to?
Human: a tree
ALICE: Oh I get it.

When you consider that ALICE won the 2004 Loebner prize, you’ll soon understand why our current technology is far from being able to pass the Turing test: while the bot was surprisingly able to parse one of the most difficult questions successfully (“What color is a blue car?”), it is clear that the bot strategy is to change the subject of conversation whenever it has difficulties understanding what the real person behind the keyboard is saying.

This is also why many prominent researchers in the field of AI oppose the Loebner prize, maintaining that the entire competition ends up becoming discouraging for its competitors instead of giving a new impulse towards the creation of better and better AIs.

One of the main reasons why these rudimental chatbots are unable to provide reliable answers, many experts agree, is that they don’t have an ontology — a precise hierarchy that helps them classify objects and perform reasoning on its knowledge base, particularly abstract thought such as induction and the likes.

Some companies, such as Douglas Lenat’s Cycorp (creator of the Cyc inference engine) or even the search engine True Knowledge, use this approach and have already widely demonstrated that, to the very least, this is a very valid technique that can lead to satisfactory results, even though not every researcher seems to agree with it, to the point of even mocking Lenat’s efforts because of the unpracticality of this approach.

But here I’m touching a completely different subject — so, I’ll write more on this in an upcoming post.