The one-liner:
dd if=/dev/zero bs=1G count=10 | gzip -c > 10GB.gz
This is brilliant.
Have you ever heard of sparse files, and how Linux and Windows deal with zips of it? You’ll love this.
I want to know he they built that visualization
I’ve been thinking about making an nginx plugin that randomizes words on a page to poison AI scrapers.
There are “AI mazes” that do that.
I remember reading and article about this but haven’t found it yet
The one below, named Anubis. Is the one I heard about. Come back to the thread and check the link.
That is a very interesting git repo. Is this just a web view into the actual git folder?
If you have the time, I think it’s a great idea.
This is why I use things like Docusaurus to generate static sites. Vulnerability injections are pretty hard when there’s no code to inject into.
Probably only works for dumb bots and I’m guessing the big ones are resilient to this sort of thing.
Judging from recent stories the big threat is bots scraping for AIs and I wonder if there is a way to poison content so any AI ingesting it becomes dumber. e.g. text which is nonsensical or filled with counter information, trap phrases that reveal any AIs that ingested it, garbage pictures that purport to show something they don’t etc.
I don’t know as to poisoning AI, but one thing that I used to do was to redirect any suspicious bots or ones that were hitting their server too much to a simple html page with no JS or CSS or forward links. Then they used to go away.
When it comes to attacks on the Internet, doing simple things to get rid of the stupid bots means kicking 90% of attacks out. No, it won’t work against a determined foe, but it does something useful.
Same goes for setting SSH to a random port. Logs are so much cleaner after doing that.
Setting a random SSH port and limiting it to 3/min saw failed login attempts fall by 99% and jailed IPs fall to 0.
I’ve found great success using a hardened ssh config with a limited set of supported
Cyphers/MACs/KexAlgorithms. Nothing ever gets far enough to even triggerfail2ban. Then of course it’s key only login from there.
There have been some attempts in that regard, I don’t remember the names of the projects, but there were one or two that’d basically generate a crapton of nonsense to do just that. No idea how well that works.
❤️
I’d be amazed if this works, since these sorts of tricks have been around since dinosaurs ruled the Earth, and most bots will use pretty modern zip libraries which will just return “nope” or throw an exception, which will be treated exactly the same way any corrupt file is - for example a site saying it’s serving a zip file but the contents are a generic 404 html file, which is not uncommon.
Also, be careful because you could destroy your own device? What the hell? No. Unless you’re using dd backwards and as root, you can’t do anything bad, and even then it’s the drive contents you overwrite, not the device you “destroy”.
On the other hand, there are lots of bots scraping Wikipedia even though it’s easy to download the entire website as a single archive.
So they’re not really that smart…
Yeah, this article came across as if written by a complete beginner. They mention having their WordPress hacked, but failed to admit it was because they didn’t upgrade the install.
Sadly about the only thing that reliably helps against malicious crawlers is Anubis
I don’t really like this approach, not just because I was flagged as a bot, but because I don’t really like captchas. I swear I’m not a bot guys!
Neat
That URL is telling me “Invalid response”. Am I a bot?
Now you know why your mom spent so much time with the Amiga
https://anubis.techaro.lol/docs/user/known-broken-extensions
If you have JShelter installed, it breaks the proof of work from anubis
You’re using a VPN, right?
Nope. Just using Vivaldi on my Android device.
Im not and it gave an invalid response. I am just chilling on my home wifi.
I’m sorry you had to find out this way.
This reminds me of shitty FTP sites with ratios when I was on dial-up. I used to push them files full of null characters with filenames that looked like actual content. The modem would compress the upload as it transmitted it which allowed me to upload the junk files at several times the rate of a normal file.
that is pretty darn clever
I use a torrent client that will lie on the upload (x10 or x11, or a myriad of other options) so as to satisfy the upload ratio requirement of many members only torrent communities
At least in germany having one of these on your system is illegal
Out of curiosity, what is illegal about it, exactly?
I mean i am not a lawyer.
In germany we have § 303 b StGB. In short it says if you hinder someone elses dataprocessing through physical means or malicous data you can go to jail for up to 3 years . If it is a major process for someone you can get up to 5 and in major cases up to 10 years.
So if you have a zipbomb on your system and a crawler reads and unpacks it you did two crimes. 1. You hindered that crawlers dataprocessing 2. Some isp nodes look into it and can crash too. If the isp is pissed of enough you can go to jail for 5 years. This applies even if you didnt crash them die to them having protection agsinst it, because trying it is also against the law.
Having a zipbomb is part of a gray area. Because trying to disrupt dataprocessing is illegal, having a zipbomb can be considered trying, however i am not aware of any judgement in this regard
Edit: btw if you password protect your zipbomb, everything is fine
TL;DR: It’s illegal to have publically available or share.
Making it illegal to create one for research purposes on your own hardware is not illegal as far as I know. And if it is, I wouldn’t mind seeing someone challenge that with the EU.
I wonder if having a robots.txt file that said to ignore the file/path would help.
I’m assuming a bad bot would ignore the robots.txt file. So you could argue that you put up a clear sign and they chose to ignore it.
Good question i dont know tbh. Would be an interesting question for a lawyer influencer
Severely disrupting other people’s data processing of significant import to them. By submitting malicious data requires intent to cause harm, physical destruction, deletion, etc, doesn’t. This is about crashing people’s payroll systems, ddosing, etc. Not burning some cpu cycles and having a crawler subprocess crash with OOM.
Why the hell would an ISP have a look at this. And even if, they’re professional enough to detect zip bombs. Which btw is why this whole thing is pointless anyway: If you class requests as malicious, just don’t serve them. If that’s not enough it’s much more sensible to go the anubis route and demand proof of work as that catches crawlers which come from a gazillion IPs with different user agents etc.
Telecom for example does Deep PackageInspection. That is rather well kown. Derec made a statement years ago that it is normal for other european isps too. Here is a secondary source for it, i cant find the primary source anymore https://netzpolitik.org/2012/berec-studie-dpi-bei-vielen-providern-bereits-im-einsatz/
If you are succesful in disrupting some dataprocessing doesnt matter, trying to do it is illigal. If you put it there to disrupt crawlers you are trying to disrupt an entities dataprocessing.
If your isp does dpi an archive bomb is able to crash their server. Even if they have measures againt it, it is still illigal because trying it is illigal.
The intent is to get rid of crawlers which are disrupting the operation of your servers. That’s not intent of doing harm to the crawler’s operator, or their business. It’s analogous to telling a hawker to fuck off: Polite, no, but them being able to profit off you is not your responsibly, you do not have to accede to that. And intent to harm the ISP is even less reasonable to assume.
cant find the primary source anymore https://netzpolitik.org/2012/berec-studie-dpi-bei-vielen-providern-bereits-im-einsatz/
That’s out of date anyway. How about this one. DPI is limited to OSI level 5 and only allowed to resolve network issues – and a crawler crashing is not a network issue.
That’s out of date anyway. How about this one.
Good to know
A crawler is a data processing machine, nothing more. therefor you are disrupting dataprocessing through data. If you think its not thats ok too. I would still advise to contact your lawyer in germany if you are thinking about hosting a zipbomb
A crawler is a data processing machine, nothing more. therefor you are disrupting dataprocessing through data. If you think its not thats ok too.
Nah it’s definitely disrupting data processing, even though at a very low-key level – you’re not causing any data to become invalid or such. It’s the intent to harm the operator that’s the linchpin: “Jemandem einen Nachteil zufügen”. “Jemand” needs to be a person, natural or legal. And by stopping a crawler you don’t want to inflict a disadvantage on the operator you want to, at most, stop them from gaining an advantage. “Inflict disadvantage” and “prevent advantage” are two different things.
I would still advise to contact your lawyer in germany if you are thinking about hosting a zipbomb
Good idea, but as already said before: First, you should contact a sysadmin. Who will tell you it’s a stupid idea.
Illegal to publically serve or distribute.
Maybe bots shouldn’t be trying to install malicious code? Sucks to suck.
Still illegal. Not immoral, but a lot of our laws aren’t built on morality.
Funny part is many of us crusty old sysadmins were using derivatives of this decades ago to test RAID-5/6 sequencial reads and write speeds.
macOS compresses its memory. Does this mean we’ll see bots running on macOS now?
Linux and Windows compress it too, for 10 years or more. And that’s not how you avoid zip bombs, just limit how much you uncompress and abort if it’s over that limit.
I was going to say the same thing.
Is it immune to zip bombs?
All I know is it compresses memory. The mechanism mentioned here for ZIP bombs to crash bots is to fill up memory fast with repeating zeroes.
I thought it was to fill all available storage. Maybe it’s both?
No, but that’s an interesting question. Ultimately it probably comes down to hardware specs. Or depending on the particular bot and it’s env the spec of the container it’s running in
Even with macos’s style of compressing inactive memory pages you’ll still have a hard cap that can be reached with the same technique (just with a larger uncompressed file)
How long would it take to be considered an inactive memory page? Does OOM conditions immediately trigger compression, or would the process die first?
First off, be very careful with
bs=1Gas it may overload the RAM. You will want to setcountaccordinglyYup, use something sensible like 10M or so.
I would normally go much lower,
bs=4K count=262144which creates 1G with 4K block size
How I read that code:
“If the dev folder’s bullshit is equal to 1 gram…”
The article writer kind of complains that they’re having to serve a 10MB file, which is the result of the gzip compression. If that’s a problem, they could switch to bzip2. It’s available pretty much everywhere that gzip is available and it packs the 10GB down to 7506 bytes.
That’s not a typo. bzip2 is way better with highly redundant data.
zstd is a significantly better option than anything else available unless you need something specific for a specific reason: https://github.com/facebook/zstd?tab=readme-ov-file#benchmarks
LZ4 is likely better than zstd, but it doesn’t have wide usability yet.
You might be thinking of lzip rather than lz4. Both compress, but the former is meant for high compression whereas the latter is meant for speed. Neither are particularly good at dealing with highly redundant data though, if my testing is anything to go by.
Either way, none of those are installed as standard in my distro. xz (which is lzma based) is installed as standard but, like lzip, is slow, and zstd is still pretty new to some distros, so the recipient could conceivably not have that installed either.
bzip2 is ancient and almost always available at this point, which is why I figured it would be the best option to stand in for gzip.
As it turns out, the question was one of data streams not files, and as at least one other person pointed out, brotli is often available for streams where bzip2 isn’t. That’s also not installed by default as a command line tool, but it may well be that the recipient, while attempting to emulate a browser, might have actually installed it.
Brotli gets it to 8.3K, and is supported in most browsers, so there’s a chance scrapers also support it.
Gzip encoding has been part of the HTTP protocol for a long time and every server-side HTTP library out there supports it, and phishing/scrapper bots will be done with server-side libraries, not using browser engines.
Further, judging by the guy’s example in his article he’s not using gzip with maximum compression when generating the zip bomb files: he needs to add -9 to the gzip command line to get the best compression (but it will be slower).(I tested this and it made no difference at all).You can make multiple files with different encodings and select based on the
Accept-Encodingheader.
TIL why I’m gonna start learning more about bzip2. Thanks!
I believe he’s returning a gzip HTTP reaponse stream, not just a file payload that the requester then downloads and decompresses.
Bzip isn’t used in HTTP compression.
For scrapers that not just implementing HTTP, but are trying to extract zip files, you can possibly drive them insane with zip quines: https://github.com/ruvmello/zip-quine-generator or otherwise compressed files that contain themselves at some level of nesting, possibly with other data so that they recursively expand to an unbounded (“infinite”) size.
Brotli is an option, and it’s comparable to Bzip.
