Someone Should Run an Actual Turing Test

I feel like it’s time to point at the sign again. Go read “Computing Machinery and Intelligence” first. I’ll wait. There’s a bit of a flutter on Twitter about something that Sam Altman said. I generally agree with the sentiment he shared there but it bears pointing out that no one has actually bothered to run a real Turing test yet.

Specifically, Turing wanted to pick a task so difficult, so obviously requiring of intelligence that if a computer could do it that it would be obvious to everyone that it IS intelligent. For better or for worse he picked deception for reasons that he explains in his paper. Leaving aside for the moment whether that is a good task to focus on or not, I generally agree that it’s a task that cannot be accomplished AT HUMAN LEVELS without intelligence. However, testing whether or not an AI surpasses human deception capabilities you have to establish the HUMAN baseline for deception.

The control group to a Turing test experiment is ALL HUMAN. Two human participants and one human judge. One of the human participants is attempting to deceive the judge into believing that they have a property which they do not actually have but which the other participant actually has. The judge’s job is to figure out which HUMAN is lying. That establishes the baseline for the Turing test.

Thank you for attending my TED Talk.