- cross-posted to:
- sysadmin@lemmy.ml
- sysadmin@lemmy.world
- cross-posted to:
- sysadmin@lemmy.ml
- sysadmin@lemmy.world
All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.
Apparently caused by a bad CrowdStrike update.
You must log in or register to comment.
Huh, so that’s why the office couldn’t order pizza last night lmfao
Meanwhile Kaspersky: *thinks if so incompetent people can even make antivirus at all*
Why do people run windows servers when Linux exists, it’s literally a no brainer.
lol
too bad me posting this will bump the comment count though. maybe we should try to keep the vote count to 404
CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas.
Never trust a texan
I’m so exhausted… This is madness. As a Linux user I’ve busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.
This is a better article. It’s a CrowdStrike issue with an update (security software)
I agree that’s a better article, thanks for sharing
If these affected systems are boot looping, how will they be fixed? Reinstall?
It is possible to edit a folder name in windows drivers. But for IT departments that could be more work than a reimage
It’s just one file to delete.
Having had to fix >100 machines today, I’m not sure how a reimage would be less work. Restoring from backups maybe, but reimage and reconfig is so painful
There is a fix people have found which requires manual booting into safe mode and removal of a file causing the BSODs. No clue if/how they are going to implement a fix remotely when the affected machines can’t even boot.
Probably have to go old-skool and actually be at the machine.
And hope you are not using BitLocker cause then you are screwed since BitLocker is tied to CS.
Exactly, and super fun when all your systems are remote!!!
It’s not super awful as long as everything is virtual. It’s annoying, but not painful like it would be for physical systems.
Really don’t envy physical/desk side support folks today…
You just need console access. Which if any of the affected servers are VMs, you’ll have.
Yes, VMs will be more manageable.
Do you have any source on this?
It seems like it’s in like half of the news stories.
I can confirm it works after applying it to >100 servers :/
Nice work, friend. 🤝 [back pat]
If you have an account you can view the support thread here: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19
Workaround Steps:
-
Boot Windows into Safe Mode or the Windows Recovery Environment
-
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
-
Locate the file matching “C-00000291*.sys”, and delete it.
-
Boot the host normally.
-
Some intern is getting their ass beat right now, never release into prod without extensive test.
Stop running production services on M$. There is a better backend OS.
This is why you create restore points if using windows.
A lot of people I work with were affected, I wasn’t one of them. I had assumed it was because I put my machine to sleep yesterday (and every other day this week) and just woke it up after booting it. I assumed it was an on startup thing and that’s why I didn’t have it.
Our IT provider already broke EVERYTHING earlier this month when they remote installed" Nexthink Collector" which forced a 30+ minute CHKDSK on every boot for EVERYONE, until they rolled out a fix (which they were at least able to do remotely), and I didn’t want to have to deal with that the week before I go in leave.
But it sounds like it even happened to running systems so now I don’t know why I wasn’t affected, unless it’s a windows 10 only thing?
Our IT have had some grief lately, but at least they specified Intel 12th gen on our latest CAD machines, rather than 13th or 14th, so they’ve got at least one win.
My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I’ll be working from home for a few days before anything is fixed.
play stupid games win stupid prizes
Buy the dip!
deleted by creator
But probably not immediately, probably slowly over time as contracts come due.
