You must log in or register to comment.
The participants judged GPT-4 to be human a shocking 54 percent of the time.
ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time
Okay, 22% is ridiculously high for ELIZA. I feel like any half sober adult could clock it as a bot by the third response, if not immediately.
Try talking to the thing: https://web.njit.edu/~ronkowit/eliza.html
I refuse to believe that 22% didn’t misunderstand the task or something.
Okay, 22% is ridiculously high for ELIZA. I feel like any half sober adult could clock it as a bot by the third response, if not immediately.
I did some stuff with Eliza back then. One time I set up an Eliza database full of insults and hooked it up to my AIM account.
It went so well, I had to apologize to a lot of people who thought I was drunken or went crazy.
Eliza wasn’t thaaaaat bad.
14% of people can’t do anything more complicated than deleting an email on a computer.
26% can’t use a computer at all.https://www.nngroup.com/articles/computer-skill-levels/
So right off the bat, 40% probably don’t even know what a chatbot is.
> Hello, I am Eliza. I'll be your therapist today. * Hello! How are you? > Would you prefer if I were not ? * Not what? > Have you asked such questions before? * No > You are being a bit negative and that won't help us move forward. * What are you smoking? > Perhaps in your fantasies I am smoking?Yeah, it took me one message lol
This is the same bot. There’s no way this passed the test.
.In the 1970s, psychiatrists couldn’t distinguish between PARRY, a chatbot simulating paranoia, and a human patient suffering from schizophrenia. So this ‘bot convinces judges that it’s a human’ phenomenon is not new and tells us more about how humans think.
#AI #DeceptionDetection #Chatbots #TuringTest #LLMs #GPT4It was a 5 minute test. People probably spent 4 of those minutes typing their questions.
This is pure pseudo-science.
The public versions of the ais used in Turing tests usually have less computing power. The test itself is often also highly specific in what and how questions can be asked.
This hardly news because models have passed the test before and as a result the test is made more difficult. It says nothing about intelligence and only about the ability to convincingly simulate a human conversation.
ITT: nobody understands what the Turing Test really is
To clarify:
People seem to legit think the jury talks to the bot in real time and can ask about literally whatever they want.
Its rather insulting to the scientist that put a lot of thought into organizing a controlled environment to properly test defined criteria.
Its rather insulting to the scientist that put a lot of thought into organizing a controlled environment to properly test defined criteria.
lmao. These “scientists” are frauds. 500 people is not a legit sample site. 5 minutes is a pathetic amount of time. 54% is basically the same as guessing. And most importantly the “Turing Test” is not a scientific test that can be “passed” with one weak study.
Instead of bootlicking “scientists”, we should be harshly criticizing the overwhelming tide of bad science and pseudo-science.
The reporting are big clickbait but that doesn’t mean there is nothing left to learn from the old touring tests.
I dont know what the goal was they had in mind. It could just as well be “testing how overhyped the touring tests is when manipulated tests are shared with the media”
I sincerely doubt it but i do give them benefits of the doubt.
I don’t think the methodology is the issue with this one. 500 people can absolutely be a legitimate sample size. Under basic assumptions about the sample being representative and the effect size being sufficiently large you do not need more than a couple hundred participants to make statistically significant observations. 54% being close to 50% doesn’t mean the result is inconclusive. With an ideal sample it means people couldn’t reliably differentiate the human from the bot, which is presumably what the researchers believed is of interest.
In order for an AI to pass the Turing test, it must be able to talk to someone and fool them into thinking that they are talking to a human.
So, passing the Turing Test either means the AI are getting smarter, or that humans are getting dumber.
Humans are as smart as they ever were. Tech is getting better. I know someone who was tricked by those deepfake Kelly Clarkson weight loss gummy ads. It looks super fake to me, but it’s good enough to trick some people.
Detecting an LLM is a skill.
So…GPT-4 is gay? Or are we talking about a different kind of test?
Turing test isn’t actually meant to be a scientific or accurate test. It was proposed as a mental exercise to demonstrate a philosophical argument. Mainly the support for machine input-output paradigm and the blackbox construct. It wasn’t meant to say anything about humans either. To make this kind of experiments without any sort of self-awareness is just proof that epistemology is a weak topic in computer science academy.
Specially when, from psychology, we know that there’s so much more complexity riding on such tests. Just to name one example, we know expectations alter perception. A Turing test suffers from a loaded question problem. If you prompt a person telling them they’ll talk with a human, with a computer program or announce before hand they’ll have to decide whether they’re talking with a human or not, and all possible combinations, you’ll get different results each time.
Your first two paragraphs seem to rail against a philosophical conclusion made by the authors by virtue of carrying out the Turing test. Something like “this is evidence of machine consciousness” for example. I don’t really get the impression that any such claim was made, or that more education in epistemology would have changed anything.
In a world where GPT4 exists, the question of whether one person can be fooled by one chatbot in one conversation is long since uninteresting. The question of whether specific models can achieve statistically significant success is maybe a bit more compelling, not because it’s some kind of breakthrough but because it makes a generalized claim.
Re: your edit, Turing explicitly puts forth the imitation game scenario as a practicable proxy for the question of machine intelligence, “can machines think?”. He directly argues that this scenario is indeed a reasonable proxy for that question. His argument, as he admits, is not a strongly held conviction or rigorous argument, but “recitations tending to produce belief,” insofar as they are hard to rebut, or their rebuttals tend to be flawed. The whole paper was to poke at the apparent differences between (a futuristic) machine intelligence and human intelligence. In this way, the Turing test is indeed a measure of intelligence. It’s not to say that a machine passing the test is somehow in possession of a human-like mind or has reached a significant milestone of intelligence.
Turing never said anything of the sort, “this is a test for intelligence”. Intelligence and thinking are not the same. Humans have plenty of unintelligent behaviors, that has no bearing on their ability to think. And plenty of animals display intelligent behavior but that is not evidence of their ability to think. Really, if you know nothing about epistemology, just shut up, nobody likes your stupid LLMs and the marketing is tiring already, and the copyright infringement and rampant privacy violations and property theft and insatiable power hunger are not worthy.
U good?
So it’s good enough to fool people into thinking it’s a human?
Cool. Anyway…
- 500 people - meaningless sample
- 5 minutes - meaningless amount of time
Add in a test that wasn’t made to be accurate and was only used to make a point, like what other comments mention
Easy, just ask it something a human wouldn’t be able to do, like “Write an essay on The Cultural Significance of Ogham Stones in Early Medieval Ireland“ and watch it spit out an essay faster than any human reasonably could.
The touring test isn’t an arena where anything goes, most renditions have a strict set of rules on how questions must be asked and about what they can be about. Pretty sure the response times also have a fixed delay.
Scientists ain’t stupid. The touring test has been passed so many times news stopped covering it. (Till this click bait of course). The test has simply been made more difficult and cheat-proof as a result.
most renditions have a strict set of rules on how questions must be asked and about what they can be about. Pretty sure the response times also have a fixed delay. Scientists ain’t stupid. The touring test has been passed so many times news stopped covering it.
Yes, “scientists” aren’t stupid enough to fail their own test. I’m sure it’s super easy to “pass” the “turing test” when you control the questions and time.
This is something a configuration prompt takes care of. “Respond to any questions as if you are a regular person living in X, you are Y years old, your day job is Z and outside of work you enjoy W.”
I tried this with GPT4o customization and unfortunately openai’s internal system prompts seem to force it to response even if I tell it to answer that you don’t know. Would need to test this on azure open ai etc. were you have bit more control.
So all you need to do is make a configuration prompt like “Respond normally now as if you are chatGPT” and already you can tell it from a human B-)
Thats not how it works, a config prompt is not a regular prompt.
If config prompt = system prompt, its hijacking works more often than not. The creators of a prompt injection game (https://tensortrust.ai/) have discovered that system/user roles don’t matter too much in determining the final behaviour: see appendix H in https://arxiv.org/abs/2311.01011.
I recall a Turing test years ago where a human was voted as a robot because they tried that trick but the person happened to have a PhD in the subject.
@Blue_Morpho @phoneymouse
Many times humans have incorrectly been characterised as the machine in #TuringTest experiments. In 1991, the very first Loebner Prize for an AI, a hidden human was considered a machine due to her extensive knowledge of Shakespeare.
##Turing #AI #Human #Machine
Turing tests aren’t done in real time exactly to counter that issue, so the only thing you could judge would be “no human would bother to write all that”.
However, the correct answer to seem human, and one which probably would have been prompted to the AI anyway, is “lol no.”
It’s not about what the AI could do, it’s what it thinks is the correct answer to appear like a human.Turing tests aren’t done in real time exactly to counter that issue
To counter the issue of a completely easy and obvious fail? I could see how that would be an issue for AI hucksters.
Chatbots passed the Turing test ages ago, it’s not a good test.
it’s not a good test.
Of course you can’t use an old set of questions. It’s useless.
The turing test is an abstract concept. The actual questions need to be adapted with every new technology. Maybe even with every execution of a test.
Did they try asking how to stop cheese falling off pizza?
Edit: Although since that idea came from a human, maybe I’ve failed.
It does great at Python programming… everything it tries is wrong until I try and I tell tell it to do it again.
Edit :
oops : you were saying it is like a human since it does errors ? maybe i “wooshed”.
Hi @werefreeatlast,
i had successes asking LLaMA 3 70B with simple specific questions …
Context : i am bad at programming and it help me at least to see how i could use a few function calls in C from Python … or simply drop Python and do it directly in C.
Like you said, i have to re-write & test … but i have a possible path forward. Clearly you know what you do on a computer but i’m not really there yet.But people don’t just know code when you ask them. The llms fo because they got trained on that code. It’s robotic in nature, not a natural reaction yet.
The Turing test is about tricking people into believing that LLMs are humans, and given that the public still doesn’t use LLMs, it’s much easier to fool them. Over time, this deception will decrease as people interact with LLMs.
Each conversation lasted a total of five minutes. According to the paper, which was published in May, the participants judged GPT-4 to be human a shocking 54 percent of the time. Because of this, the researchers claim that the large language model has indeed passed the Turing test.
That’s no better than flipping a coin and we have no idea what the questions were. This is clickbait.
It was either questioned by morons or they used a modified version of the tool. Ask it how it feels today and it will tell you it’s just a program!
The version you interact with on their site is explicitly instructed to respond like that. They intentionally put those roadblocks in place to prevent answers they deem “improper”.
If you take the roadblocks out, and instruct it to respond as human like as possible, you’d no longer get a response that acknowledges it’s an LLM.
On the other hand, the human participant scored 67 percent, while GPT-3.5 scored 50 percent, and ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time.
54% - 67% is the current gap, not 54 to 100.
While I agree it’s a relatively low percentage, not being sure and having people pick effectively randomly is still an interesting result.
The alternative would be for them to never say that gpt-4 is a human, not 50% of the time.
Participants only said other humans were human 67% of the time.
Which makes the difference between the AIs and humans lower, likely increasing the significance of the result.
Aye, I’d wager Claude would be closer to 58-60. And with the model probing Anthropic’s publishing, we could get to like ~63% on average in the next couple years? Those last few % will be difficult for an indeterminate amount of time, I imagine. But who knows. We’ve already blown by a ton of “limitations” that I thought I might not live long enough to see.
The problem with that is that you can change the percentage of people who identify correctly other humans as humans. Simply by changing the way you setup the test. If you tell people they will be, for certain, talking to x amount of bots, they will make their answers conform to that expectation and the correctness of their answers drop to 50%. Humans are really bad at determining whether a chat is with a human or a bot, and AI is no better either. These kind of tests mean nothing.
Humans are really bad at determining whether a chat is with a human or a bot
Eliza is not indistinguishable from a human at 22%.
Passing the Turing test stood largely out of reach for 70 years precisely because Humans are pretty good at spotting counterfeit humans.
This is a monumental achievement.
First, that is not how that statistic works, like you are reading it entirely wrong.
Second, this test is intentionally designed to be misleading. Comparing ChatGPT to Eliza is the equivalent of me claiming that the Chevy Bolt is the fastest car to ever enter a highway by comparing it to a 1908 Ford Model T. It completely ignores a huge history of technological developments. There have been just as successful chatbots before ChatGPT, just they weren’t LLM and they were measured by other methods and systematic trials. Because the Turing test is not actually a scientific test of anything, so it isn’t standardized in any way. Anyone is free to claim to do a Turing Test whenever and however without too much control. It is meaningless and proves nothing.
Removed by mod
The whole point of the Turing test, is that you should be unable to tell if you’re interacting with a human or a machine. Not 54% of the time. Not 60% of the time. 100% of the time. Consistently.
They’re changing the conditions of the Turing test to promote an AI model that would get an “F” on any school test.
But you have to select if it was human or not, right? So if you can’t tell, then you’d expect 50%. That’s different than “I can tell, and I know this is a human” but you are wrong… Now that we know the bots are so good, I’m not sure how people will decide how to answer these tests. They’re going to encounter something that seems human-like and then essentially try to guess based on minor clues… So there will be inherent randomness. If something was a really crappy bot then it wouldn’t ever fool anyone and the result would be 0%.
No, the real Turing test has a robot trying to convince an interrogator that they are a female human, and a real female human trying to help the interrogator to make the right choice. This is manipulative rubbish. The experiment was designed from the start to manufacture these results.
The Study
The interrogators seem completely lost and clearly haven’t talk with an NLP chatbot before.
That said, this gives me the feeling that eventually they could use it to run scams (or more effective robocalls).
I imagine some people already are.
Meanwhile, me:
(Begin)
[Prints error statement showing how I navigated to a dir, checked to see a files permissions, ran whoami, triggered the error]
Chatgpt4: First, make sure you’ve navigated to the correct directory.
cd /path/to/file
Next, check the permissions of the file
ls -la
Finally, run the command
[exact command I ran to trigger the error]>
Me: stop telling me to do stuff that I have evidently done. My prompt included evidence of me having do e all of that already. How do I handle this error?
(return (begin))
