Live: Global IT outage affecting banks, airlines, media outlets across the world
www.abc.net.au
There are reports of IT outages affecting major institutions in Australia and internationally.

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

  • ramble81@lemm.eeEnglish
    0·
    10 months ago

    We had a bad CrowdStrike update years ago where their network scanning portion couldn’t handle a load of DNS queries on start up. When asked how we could switch to manual updates we were told that wasn’t possible. So we had to black hole the update endpoint via our firewall, which luckily was separate from their telemetry endpoint. When we were ready to update, we’d have FW rules allowing groups to update in batches. They since changed that but a lot of companies just hand control over to them. They have both a file system and network shim so it can basically intercept **everything **

  • Encrypt-Keeper@lemmy.worldEnglish
    0·
    10 months ago

    Yeah my plans of going to sleep last night were thoroughly dashed as every single windows server across every datacenter I manage between two countries all cried out at the same time lmao

      • Mjpasta710@midwest.socialEnglish
        0·
        10 months ago

        This is a crowdstrike issue specifically related to the falcon sensor. Happens to affect only windows hosts.

      • marcos@lemmy.worldEnglish
        0·
        10 months ago

        It’s only marginal for running custom code. Every large organization has at least a few of them running important out-of-the-box services.

      • Pringles@lemm.eeEnglish
        0·
        10 months ago

        Marginal? You must be joking. A vast amount of servers run on Windows Server. Where I work alone we have several hundred and many companies have a similar setup. Statista put the Windows Server OS market share over 70% in 2019. While I find it hard to believe it would be that high, it does clearly indicate it’s most certainly not a marginal percentage.

        • jj4211@lemmy.worldEnglish
          0·
          10 months ago

          I’m not getting an account on Statista, and I agree that its marketshare isn’t “marginal” in practice, but something is up with those figures, since overwhelmingly internet hosted services are on top of Linux. Internal servers may be a bit different, but “servers” I’d expect to count internet servers…

      • rottingleaf@lemmy.worldEnglish
        0·
        10 months ago

        Well, I’ve seen some, but they usually don’t have automatic updates and generally do not have access to the Internet.

      • Eril@feddit.orgEnglish
        0·
        10 months ago

        My current company does and I hate it so much. Who even got that idea in the first place? Linux always dominated server-side stuff, no?

      • Delta_V@lemmy.worldEnglish
        0·
        10 months ago

        Not too long ago, a lot of Customer Relationship Management (CRM) software ran on MS SQL Server. Businesses made significant investments in software and training, and some of them don’t have the technical, financial, or logistical resources to adapt - momentum keeps them using Windows Server.

        For example, small businesses that are physically located in rural areas can’t use cloud based services because rural internet is too slow and unreliable. Its not quite the case that there’s no amount of money you can pay for a good internet connection in rural America, but last time I looked into it, Verizon wanted to charge me $20,000 per mile to run a fiber optic cable from the nearest town to my client’s farm.

      • Encrypt-Keeper@lemmy.worldEnglish
        0·
        10 months ago

        There was a point where words lost all meaning and I think my heart was one continuous beat for a good hour.

        • jj4211@lemmy.worldEnglish
          0·
          10 months ago

          I work in a datacenter, but no Windows. I slept so well.

          Though a couple years back some ransomware that also impacted Linux ran through, but I got to sleep well because it only bit people with easily guessed root passwords. It bit a lot of other departments at the company though.

          This time even the Windows folks were spared, because CrowdStrike wasn’t the solution they infested themselves with (they use other providers, who I fully expect to screw up the same way one day).

  • BurnSquirrel@lemmy.worldEnglish
    0·
    10 months ago

    I’m so exhausted… This is madness. As a Linux user I’ve busy all day telling people with bricked PCs that Linux is better but there are just so many. It never ends. I think this is outage is going to keep me busy all weekend.

  • Victor@lemmy.worldEnglish
    0·
    10 months ago

    If these affected systems are boot looping, how will they be fixed? Reinstall?

    • CanadaPlus@lemmy.sdf.orgEnglish
      0·
      10 months ago

      Honestly my philosophy these days, when it comes to anything proprietary. They just can’t keep their grubby little fingers off of working software.

      At least this time it was an accident.

    • Hotzilla@sopuli.xyzEnglish
      0·
      10 months ago

      There is nothing unsafer than local networks.

      AV/XDR is not optional even in offline networks. If you don’t have visibility on your network, you are totally screwed.

  • CanadaPlus@lemmy.sdf.orgEnglish
    0·
    10 months ago

    Yep, this is the stupid timeline. Y2K happening to to the nuances of calendar systems might have sounded dumb at the time, but it doesn’t now. Y2K happening because of some unknown contractor’s YOLO Friday update definitely is.

  • ari_verse@lemm.eeEnglish
    0·
    10 months ago

    It’s a fair point but I would rather diversify and also use something that is open / less opaque

  • foremanguy@lemmy.mlEnglish
    0·
    10 months ago

    Bahaha 😂😂 continue using proprietary software, that’s all you are going to get in addition to privacy issues… Switch to Linux.

  • Cornelius_Wangenheim@lemmy.worldEnglish
    0·
    10 months ago

    A bunch of shitty sysadmins/cybersec people just learned why you don’t blindly deploy new updates to production without testing them first.

    I’ve used crowd strike before. It has support for deploying version N to a pilot group, N-1 to the test environment and N-2 to production.

  • Spaceinv8er@sh.itjust.worksEnglish
    0·
    10 months ago

    So that’s why my work laptop is down for the count today. I’m even getting that same error as the thumbnail picture

  • catch22@programming.devEnglish
    0·
    10 months ago

    Interesting how ARPA net (the internet) was build to with stand these issues, but companies like Microsoft and Amazon (and no regulation) have completely reversed it’s original intent. I actually didn’t even notice this since I use Lemmy, and have my own internal network running home assistant, synology, emby, ect…