Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.
Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.
The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).
In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.
Huffman said:
Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.
“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.
Honestly, my biggest issue with LLMs is how they source their training data to create “their own” stuff. A meme calling it a plagiarism machine struck a chord with me. Almost anyone else I’d sympathize with, but fuck Spez.
What resonated with me is people calling LLMs and Stable Diffusion “copyright laundering”. If copyright ever swung in AI’s favor it would be super easy to train an AI on stuff you want to steal, add in some generic training, and now you have a “new” piece of art.
LLMs and Stable Diffusion are just compression algorithms for abstract patterns, only one level above data.
The real takeaway of all of this is that copyright law is massively out of date and not fit for purpose in the 21st century or frankly the late 20th.
The current state of copyright law cannot deal with the internet, let alone AI
Yep they now get paid for the data we have them. I have no sympathy lol. At least these models can’t actually store it all losslessly by any stretch of the imagination. The compression factors would have to be like 100-200X+ anything we’ve ever been able to achieve before. The numbers don’t work out. The models do encode a lot though and some of it is going to include actual full text data etc but it’ll still be kinda fuzzy.
I think we do need ALL OPEN SOURCE. Not just for AI, but I know on that point I’m preaching to the choir here lol
Fuck Spez
Gross.
No thanks.
profiting off of user generated content 😒
Reddit is dying anyway.
In all the ways that matter, it’s already dead. Once something enshittifies beyond a certain point, is its zombified, shambling corpse really considered “alive” anymore?
It’s Digg v…5 6? v5 would be when they inflated all post scores and stopped showing upvotes and downvotes separately.
v6 is… gestures to the all of it
It’s easy to say this as someone who is “on the other side”. But the data doesn’t really back up that statement.
I don’t have data, but the quality of the content certainly seemed to be declining, even as the quality went up.
I think autocorrect might’ve gotten you: You posted “quality” twice in a contradictory way.
Fixed.
It’s awful. Politics is unavoidable at this point, and the amount of general anger on the platform is crazy.
People love watching their videos of people getting TBIs… Or getting too excited about a “justice served” post where a woman gets hit.
It’s kinda nice to see someone get their comeuppance, but then you look in the comments and there are just weirdos saying stuff like “glad that bitch got hit”, like… wtf?
And yet reddit is happy to make money off our content for free.
Or at least it did. Personally I overwrote and deleted all my content a while back.
They aren’t making money off my content anymore/in the future
All you did was screw over people looking for potentially useful content while searching online. Reddit still has all your content.
Sure, but if they’re not displaying it what good is it doing them or their advertisers?
They’re not selling the ability to scrape their publicly viewable site, they’re selling API access to their database. Deleted posts aren’t removed from that database.
You think that Reddit didn’t already have the previous content saved?
Bingo, the only winning move is not to play at all and stop using Reddit.
Yep that’s how it works. Older content past a certain date is cached which is why you can’t comment or post on some old posts.
Everyone always says this like it’s some kind of gotcha, but all of my nuked posts still have my “fuck you, reddit” content and haven’t been reverted. It’s been nearly exactly a year.
Maybe reddit has an offline copy of my old content and that of others somewhere, but if so they’d be handing that directly over to whoever under some kind of agreement – that certainly wouldn’t be the subject of any kind of site crawling which is the crux of the issue here.
You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet
Or that they could start feeding your archived (not cached) data directly to the AI companies anyway for a price
IMO, you can win by jamming your “transmissions” with noise. It’s easier to hide in noise as noise than it Is to be silent IMO. Muddy the waters as it were
Removed by mod
Content is absolutely archived and they have financial incentive to restore the quality of their “knowledge base”
That’s a fair amount of circumstance and motivation to support my idea, regardless of tangible evidence
Removed by mod
Alright well I guess evidence is needed before we can have ideas - crazy
I certainly wasn’t implying that they were going to revert your comments.
it never was deleted, all that happened is that an extra line was added to a database that said “comment 65432426542654 now should be displayed as “fuck you, reddit” rather than the original text”. The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.
Removed by mod
It seemed to happen to some people but I wouldn’t be surprised it it was just some sort of coincidental database fuck-up
Removed by mod
I’d read some claims that posts appeared to be deleted but then later came back. Could’ve even been some sort of caching shenanigans with their local browser though I guess.
Removed by mod
Removed by mod
You are assuming edits overwrite existing content. Instead of overwriting, they could just store the edited post as a new entry in the database with a higher version number. Then, you only show the latest version of each post to the end users while keeping the older versions available die Reddit’s own use.
In fact, it is extremely likely they do this. It is basically a necessity if you want to be able to properly moderate a site like Reddit. Otherwise you could simply post spam or unsavory content, and then overwrite it with something benign an hour or so later, before there were enough reports and a moderator would have gotten a chance to review it.
Removed by mod
The fact that they managed to restore overwritten posts after people started to delete their history.
Removed by mod
Reddit used to be open source. There is still a copy of that source available on github. It’s 7 years old so it’s probably significantly different from what they are running now. Still, it gives some insight into the design.
For example, deleted comments aren’t deleted, it just sets a deleted flag. Example code that shows this.
I haven’t dug around the code enough to figure out how editing works, it’s Python code so an unreadable mess. The database design also seems very strange. It’s like they built a database system on top of a database.
As I said in another comment, I was not suggesting that Reddit would restore your comments to public view.
Removed by mod
There’s no evidence of the contrary either.
Except for incidents like This
On the other side of the same coin: When I mass edited my comments before quitting Reddit, I got site-banned. Basically, my first account’s automated edit got me auto-banned from several subs with pro-spez mods. Some subs had set their automod to detect when people were using the more popular methods of auto-editing, and set the automod to ban for using them. Then when I did the same with my second (and third, and fourth, and fifth, etc…) account, it almost immediately got site-banned for ban evasion.
Basically, account 1 was banned from a sub, so when account 2 started doing the same thing on the same IP address, it was flagged as ban evasion. And ban evasion is one of the few things that will get you banned site-wide instead of just from a specific sub.
I went back and checked a few months ago, and all of those site bans were lifted and the edits were undone. Likely because a site ban prevents the comments from showing up (which hurts Reddit’s bottom line, because they show up as a bunch of [removed] comments instead,) but also prevented any of the edits from actually being published. So when they lifted the site ban (to get those old comments to show back up again) it was as if I had never edited them at all. I had probably a million karma spread across my various accounts. I was extremely active at one point, so Reddit had a direct incentive to unban those accounts with literal thousands of comments.
What if I had an agreement with MS that they can scrape my data and anything I post online?
What if Microsoft updated their Windows EULA to state that all users agree to allow MS to scrape their online data (if they haven’t already), and then take that to court against reddit? It would certainly be an interesting court case to watch, especially if they could get actual users to stand up in court and confirm that they did indeed approve of this. And it might settle the issue once and for all regarding companies trying to block freely-visible internet content just because someone scraped the info.
Robots.txt isn’t a binding agreement, this isn’t stopping anyone for who their drive for profit outweighs their ethics.
Also, Fuck Spez.
Fuck Spez. He’s probably editing the comments anyway, he literally can’t help himself.
How about starting a company that gathers people’s CAD design…grabCAD!.. Oh can’t scrape out design work Microsoft, you gotta pay!..or how about a company that stores people’s records or drawings or movies… Adobe! Oh Microsoft, you can’t scrape our data! It’s our data!
What if I had an agreement with MS that they can scrape my data and anything I post online?
Late stage capitalism
deleted by creator
Why? He’s a piece of shit who made a bad deal and is now upset about it.
deleted by creator
ITT people who don’t know what sarcasm is without a literal “/s”
We all learned a long time ago to assume someone is serious until they indicate otherwise. If the joke is indistinguishable from a serious reaction then it’s a bad joke due to ambiguity.
I guess you’re not wrong, but that does mean that deadpan delivery is dead.
Deadpan delivery on a strictly text medium has always been a difficult needle to thread. It only works in literature because the writer can describe how it was stated, which would look odd in forums/texts/emails/etc. he added sincerely
You’re not wrong there either, but I do recall a time in which (at least in threads/forums/chat) it worked fairly well.
The issue is that the main way to detect it is to look for an opinion that was wildly diverged from the consensus, but… yeah, I guess people just broadcast those seriously now.
Fuck.
Yeah sadly right wing fuckwits thrive on ambiguity and ruined it for the rest of us. Comedy is a powerful shield.
Minimum royalty laws should exist!
Because if there’s one thing this world needs more its more rights for property.
I’m sure plenty of others join me in the sentiment of thinking “Who the fuck are you to restrict MY free content that I contributed?”
God, fuck reddit so fucking hard
If you’re still donating your content to the Facebook, Twitter and Reddit data stores, then I don’t know what to tell you man, you’re literally enriching the worst people, who will do the worst things, with your information, your stories, your answers, your comments, your labor, your effort. You are giving yourselves to them. Literally.
