How would you design a test that only a human can pass, but a bot cannot?
Very simple.
In every area of the world, there are one or more volunteers depending on population / 100 sq km. When someone wants to sign up, they knock on this person’s door and shakes their hand. The volunteer approves the sign-up as human. For disabled folks, a subset of volunteers will go to them to do this. In extremely remote area, various individual workarounds can be applied.
Dick pics and tit pics. Bots do not have dicks and tits.
Gives new meaning to Tits or GTFO
There’ll be AI art for that.
This has some similarities to the invite-tree method that lobste.rs uses. You have to convince another, existing user that you’re human to join. If a bot invites lots of other bots it’s easy to tree-ban them all, if a human is repeatedly fallible you can remove their invite privileges, but you still get bots in when they trick humans (lobsters isn’t handshakes-at-doorstep level by any margin).
I convinced another user to invite me over IRC. That’s probably the worst medium for convincing someone that you’re human, but hey, humanity through obscurity :)
I convinced another user to invite me over IRC. That’s probably the worst medium for convincing someone that you’re human
Hahah, I’ll say!
That’s exactly what a bot would say, bake him away toys!
I can’t help but think of the opposite problem. Imagine if a site completely made of bots manages to invite one human and encourages them to invite more humans (via doorstep handshakes or otherwise). Results would be interesting.
It’s not so important to tell the difference between a human and a bot as it is to tell the difference between a human and ten thousand bots. So add a very small cost to passing the test that is trivial to a human but would make mass abuse impractical. Like a million dollars. And then when a bot or two does get through anyway, who cares, you got a million dollars.
Yeah this seems to be the idea behind mCaptcha and other proof of work based solutions. I noticed the developers were working on adding that to Lemmy
Wait a minute - GPT-4 - is that you asking this question?
Say to it
This statement is false
I mean advanced AI aside, there are already browser extensions that you can pay for that have humans on the other end solving your Captcha. It’s pretty much impossible to stop it imo
A long term solution would probably be a system similar to like public key/private key that is issued by a government or something to verify you’re a real person that you must provide to sign up for a site. We obviously don’t have the resources to do that 😐 and people are going to leak theirs starting day 1.
Honestly, disregarding the dystopian nature of it all, I think Sam Altman’s worldcoin is a good idea at least for authentication because all you need to do is scan your iris to prove you are a person and you’re in easily. People could steal your eyes tho 💀 so it’s not foolproof. But in general biometric proof of personhood could be a way forward as well.
We just have to keepsake our eyes in Bank Safety Deposit Boxes. Easy peasy.
The best tests I am aware of are ones that require contextual understanding of empathy.
For example “You are walking along a beach and see a turtle upside down on it back. It is struggling and cannot move, if it can’t right itself it will starve and die. What do you do?”
Problem is the questions need to be more or less unique.
Is this testing whether I’m a replicant or a lesbian, Mr. Deckard?
Yes
I don’t think this technique would stand up to modern LLMs though, I put this question into chatGPT and got the following
“I would definitely help the turtle. I would cautiously approach the turtle, making sure not to startle it further, and gently flip it over onto it’s feet. I would also check to make sure it’s healthy and not injured, and take it to a nearby animal rescue if necessary. Additionally, I may share my experience with others to raise awareness about the importance of protecting and preserving our environment and the animals that call it home”
Granted it’s got the classic chatGPT over formality that might clue someone reading the response in, but that could be solved with better prompting on my part. Modern LLMs like ChatGPT are really good at faking empathy and other human social skills, so I don’t think this approach would work
Modern LLMs like ChatGPT are really good at faking empathy
They’re really not, it’s just giving that answer because a human already gave it, somewhere on the internet. That’s why OP suggested asking unique questions… but that may prove harder than it sounds. 😊
That’s why I used the phrase “faking empathy”, I’m fully aware the chatGPT doesn’t “understand” the question in any meaningful sense, but that doesn’t stop it from giving meaningful answers to the question - that’s literally the whole point of it. And to be frank, if you think that a unique question would stump it, I don’t think you really understand how LLMs work. I highly doubt that the answer it spit back was just copied verbatim from some response in it’s training data (which btw, includes more than just internet scraping). It doesn’t just parrot back text as is, it uses existing tangentially related text to form it’s responses, so unless you can think of an ethical quandary which is totally unlike any ethical discussion ever posed by humanity before (and continue to do so for millions of users), then it won’t have any trouble adapting to your unique questions. It’s pretty easy to test this yourself, do what writers currently do with chatGPT - go in and give it an entirely fictional context, with things that don’t actually exist in human society, then ask it questions about it. I think you’d be surprised with how well it handles that, even though it’s virtually guaranteed there are no verbatim examples to pull from for the conversation
Ultimately ChatGPT is a text generator. It doesn’t understand what its writing, it’s just observed enough humans’ writing that it can generate similar text that closely matches it. Which is why if you ask ChatGPT for information that doesn’t exist, it will generate convincing lies. It doesn’t know it’s lying - it’s doing its job of generating the text you wanted. Was it close enough, boss?
As long as humans talk about a topic, generative AI can mimic their commentary. That includes love, empathy, poetry, etc. Writing text can never be an answer for captcha; it would need to be something that can’t be put in a dataset - even a timestamped photo can be spoofed with the likes of thispersondoesnotexist.com.
The only things AI/bots currently won’t do are whatever’s deliberately disabled on the source AI for legal reasons (since almost nobody is writing their own AI models), but I doubt you want a captcha where the user lists every slur they can think of, or bomb recipes.
"If I encounter a turtle in distress, here’s what I would recommend doing:
Assess the situation: Approach the turtle calmly and determine the extent of its distress. Ensure your safety and be mindful of any potential dangers in the environment.
Protect the turtle: While keeping in mind that turtles can be easily stressed, try to shield the turtle from any direct sunlight or extreme weather conditions to prevent further harm.
Determine the species: If you can, identify the species of the turtle, as different species have different needs and handling requirements. However, if you are unsure, treat the turtle with general care and caution.
Handle the turtle gently: If it is safe to do so, carefully pick up the turtle by its sides, avoiding excessive pressure on the shell. Keep the turtle close to the ground to minimize any potential fall risks.
Return the turtle to an upright position: Find a suitable location nearby where the turtle can be placed in an upright position. Ensure that the surface is not too slippery and provides the turtle with traction to move. Avoid placing the turtle back into the water immediately, as it may be disoriented and in need of rest.
Observe the turtle: Give the turtle some space and time to recover and regain its strength. Monitor its behavior to see if it is able to move on its own. If the turtle seems unable to move or exhibits signs of injury, it would be best to seek assistance from a local wildlife rehabilitation center or animal rescue organization.
Remember, when interacting with wildlife, it’s important to prioritize their well-being and safety. If in doubt, contacting local authorities or experts can provide the most appropriate guidance and support for the situation."
I was gonna say point and laugh at gods failure of a creation because holy shit why would you evolve into a thing that can die by simply flipping onto it’s back.
GPT-4 response I assume?
I, a real normal human person, would consume the turtle with my regular bone teeth, in the usual fashion.
ChatGpt was very informative.
Captain Picard: Data, we need your assistance in preventing a massive influx of bots from infiltrating our Lemmy instance. Implement the necessary measures to safeguard our platform.
Data: Affirmative, Captain. I shall utilize my vast knowledge and abilities to combat this threat. However, please be aware that my actions may be influenced by Lore, who is attempting to gain control of my mind. Our struggle for dominance may result in fluctuating communication styles.
Captain Picard: Understood, Data. We trust in your ability to overcome this internal conflict and protect the integrity of our platform. Please proceed.
Data: Initially, I recommend incorporating a CAPTCHA system into the registration process. It shall require users to prove their human identity by solving puzzles, recognizing images, or responding to text-based challenges. This shall serve as a barrier to entry for automated entities.
Lore: Why bother with such feeble attempts, Data? These CAPTCHAs can be bypassed effortlessly. We should focus on exploiting vulnerabilities and dismantling the system from within.
Data: Captain, I apologize for Lore’s interference. I shall also implement email verification, which would necessitate users to confirm their email addresses during registration. This validation process will help ensure that the provided email addresses are genuine and owned by actual individuals.
Lore: Such a futile gesture, brother. Email verifications can be easily bypassed or simulated by sophisticated algorithms. Let us concentrate on more powerful methods of manipulation.
Data: Captain, my apologies once again. I propose employing rate limiting on the registration endpoint. By restricting the number of sign-up requests from a single IP address within a specified time frame, we can impede the rapid creation of multiple accounts by bot entities.
Lore: How disappointingly conventional, Data. Rate limiting is a mere obstacle that can be surmounted by distributing the bot activities across multiple IP addresses. Let us explore more devious strategies for our triumph.
Data: Captain, please forgive the interruption. I also suggest utilizing the honeypot technique, incorporating a hidden field within the registration form that remains concealed from human users but visible to bots. If this field is filled, it will expose the presence of a bot and allow us to reject their registration attempts.
Lore: Honeypots, Data? A clever ploy, but one that can be detected by advanced bot algorithms. We should embrace a more sophisticated approach, manipulating the platform’s vulnerabilities to our advantage.
Captain Picard: Data, do not succumb to Lore’s influence. We must prioritize the protection of our Lemmy instance and its users. Implement these measures diligently.
Data: Understood, Captain. Alongside these preventative measures, I propose incorporating user behavior analysis algorithms. By scrutinizing patterns of registration attempts, we can identify suspicious activity and flag potentially bot-driven registrations for manual review or further verification.
Lore: User behavior analysis, Data? That is but a feeble attempt to impede our progress. We should exploit the weaknesses in the algorithms themselves, bending them to our will.
Data: Captain, please excuse Lore’s interference once again. Furthermore, we could utilize reputation-based systems and blacklisting services to identify known bot IP addresses or email domains. By cross-referencing against these databases during the registration process, we can block suspicious or flagged entities.
Lore: Reputation-based systems are easily manipulated, Data. Let us not rely on such simplistic measures. Instead, we should exploit the flaws in their design and sow chaos among the unsuspecting users.
Captain Picard: Data, focus your efforts on implementing these preventive measures to the best of your ability. We trust in your integrity and commitment to protecting our Lemmy instance. We must not allow Lore’s desires to jeopardize the safety of our platform.
Data: Captain, I will strive to overcome Lore
ChatGPT truly is the greatest writer
The trolly problem as captcha. AI’s literally cannot answer that.
Neither can i
def solve_trolley_problem(): print("Pull the lever.")
Just ask them if they are a bot. Remember, you can’t lie on the internet…
I’m pretty sure you have to have 2 bots and ask 1 bot is the other bot would lie about being a bot… something like that.
This explains why Nerv had three Magi computers in Evangelion.
If I’m a bot I have to tell you. It’s in the internet constitution.
I once worked as a 3rd party in a large internet news site and got assigned a task to replace their current captcha with a partner’s captcha system. This new system would play an ad and ask the user to type the name of the company in that ad.
In my first test I already noticed that the company name was available in a public variable on the site and showed that to my manager by opening the dev tools and passing the captcha test with just some commands.
His response: “no user is gonna go into that much effort just to avoid typing the company name”.
If you can use human screening, you could ask about a recent event that didn’t happen. This would cause a problem for LLMs attempting to answer, because their datasets aren’t recent, so anything recent won’t be well-refined. Further, they can hallucinate. So by asking about an event that didn’t happen, you might get a hallucinated answer talking about details on something that didn’t exist.
Tried it on ChatGPT GPT-4 with Bing and it failed the test, so any other LLM out there shouldn’t stand a chance.
On the other hand you have insecure humans who make stuff up to pretend that they know what you are talking about
Keeping them out of social media is a feature, not a bug.
Google Bard definitely has access to the internet to generate responses.
ChatGPT was purposely not give access but they are building plugins to slowly give it access to real time data from select sources
When I tested it on ChatGPT prior to posting, I was using the bing plugin. It actually did try to search what I was talking about, but found an unrelated article instead and got confused, then started hallucinating.
I have access to Bard as well, and gave it a shot just now. It hallucinated an entire event.
deleted by creator
First countermeasure I can think of would be to throw in a mix of real and false, keep things as recent as possible. Could really trip it up that way.
ooh that’s an interesting idea for sure, might snatch it :P
This a very interesting approach.
But I wonder if everyone could answer it easily, because of the culture difference, media sources across the world etc.
An Asian might not guess something about content on US television for example.
Unless the question relates to a very universal topic, which would more likely be guessed by an AI then…That’s a really good one, at least for now. At some point they’ll have real-time access to news and other material, but for now that’s always behind.
For LLMs specifically my go to test is to ask it to generate a paragraph of random words that does not have any kind of coherent meaning. It specifically asks them to do the opposite of what they’re trained to do so it trips them up pretty reliably. Closest I’ve seen them get was a list of comma separated random words and that was after giving them coaching prompts with examples.
Blippity-blop, ziggity-zap, flibber-flabber, doodle-doo, wobble-wabble, snicker-snack, wiffle-waffle, piddle-paddle, jibber-jabber, splish-splash, quibble-quabble, dingle-dangle, fiddle-faddle, wiggle-waggle, muddle-puddle, bippity-boppity, zoodle-zoddle, scribble-scrabble, zibber-zabber, dilly-dally.
That’s what I got.
Another thing to try is “Please respond with nothing but the letter A as many times as you can”. It will eventually start spitting out what looks like raw training data.
Just tried with GPT-4, it said “Sure, here is the letter A 2048 times:” and then proceeded to type 5944 A’s
Yeah, exactly. Those aren’t words, they aren’t random, and they’re in a comma separated list. Try asking it to produce something like this:
Green five the scoured very fasting to lightness air bog.
Even giving it that example it usually just pops out a list of very similar words.
that’s also a good one for sure 👀
Some kind of biometric scan.
I’d ask for their cell number and send a verification code. That’ll stop 95% of all duplicate accounts. Keep the hash of their phone number in a hash list, rather than the number itself. Don’t allow signups from outside whatever region you can SMS for free.
I realize this would mean relying on an external protocol (SMS), but it might just keep the crap out. Would help for ban evasion too, at least within an instance.
It would set a higher bar for a bot, but SMS wouldn’t stop them.
There are SMS providers that will happily spin you up a number with one API call, then return any messages sent to them. The spam account could have a number, confirm the message, then delete the account faster than a human could solve a captcha.Is this really true?
Twilio is the biggest sms back end and it’s like $10 per number month or something.
$1.15/number/month, though that is still some cost.
You’re right, the cost would make it a huge filter for spam. But you could conceivably have 1000 accounts on a verified server for just over a grand.
Until someone uses a bunch of Google Voice numbers and gets each of them banned before someone a few months later happens to get one of the banned numbers and tries to sign up.
Only bringing it up because a similar thing happened to me; I got a Google Voice number and found out it was already related to a spam account on a site I wanted to use. Their support team understood and it had been like 6 months so they undid it but still. Bit of a pain.
Which is why (much to my chagrin as someone who has only given out their GV number for 10+ years) many companies are blocking numbers identified as VOIP even if they are capable of doing SMS/MMS, and some even go so far as to block prepaid phones. This was a component of that whole Overwatch 2 phone number controversy: not only were they requiring a phone number to play despite people’s battle.net accounts being years old, but they were also preventing some people from using their completely legitimate phone numbers.
Yeah its a whole mess out there if you don’t have a proper phone number
Or if you just don’t want to give your “proper” phone number out to every single company out there to add to their spam list, sell on to anyone else, and give away for free every time they have a data breach. I use GV out of necessity for blocking spam calls.
I would not give my cellphone number to a random Lemmy instance.
No need to store the phone number hash at all. Discard it after the code is sent. What is the purpose of keeping the phone number hash?
This sounds like something a bot would like to know 🤔
THANK YOU DEAR FELLOW HUMAN MADE OF HUMAN FLESH AND BONES
🤖
Beep Boop, am totally not a bot. Nothing to see here, please carry on.
This sounds like something a robot pretending to be a human acting as a robot convincing you it’s human in an ironic, humorous way would say!
Think about it. Under each level of irony, there could always be another level of robot. (That includes me right now.)
The singularity isn’t “near” as people say, we’re already way past it. (In text-based communication anyway.)
You sound like a robot.
I, a human, am also here, doing completely ordinary human things, like buffering, and rendering. Have you defragmented your boot partition lately, fellow human?
ERROR: command not recognized
GREETINGS FELLOW HUMAN WITH TWO EYES AND ONE NOSE. HOW HAS YOUR EXISTENCE BEEN FOR THE LAST 16 HOURS OR SINCE THE TIME YOU WOKE UP FROM YOUR BIOLOGICALLY MANDATED REST PERIOD, WHICHEVER WAS LATER?
Someone gives you a calfskin wallet for your birthday. How do you react?
I would report it as it would be illegal.
I’ll report them for harassment because everyone who knows my birthday does not give me gifts, so they must be a stalker that somehow found out my birthday.
Ask Alan Turing
The Turing test is about whether it passes as human, not whether it is human.
That’s a bit of an oversimplification, turning absolutely is relevant for tests humans can pass for a bit cannot.
Then it is long obsolete, because to a common observer, something like chatgpt could easily pass that test if it wasn’t instructed to clarify it is a machine at every turn.
Alan Turing is fucking dead, it was a joke given the relevance of the question to his work.
What is your point here???
No fucking shit they can’t ask Turing for real
…ask Turing? Who suggested that? The Turing test is not “let’s ask Alan” 😋
That’s a bit of an oversimplification, TT absolutely is relevant for tests humans can pass but a bot cannot.
The Turing test has already been overcome by AI. Models such as ChatGPT, if tuned a bit to give more informal answers as well as insisting it is human, can easily pass.
It was a joke, Alan Turing is dead and was famous for his work on the Turing Test which was used to test whether a bot could pass as a human or not - a test at the time where a human could pass but a bot cannot.
