Originally Posted by ancman

United had the same problem. They’re WAY ahead of us with their recovery.

The continued abysmal recovery lies squarely on Delta. We lack system redundancy, IT personnel, OCC personnel.

None of that matters to management, as they believe that our customers will continue to pay a premium to fly on us if we wear hats and stand in the way saying goodbye during deplaning.

Originally Posted by Transit

This part is not correct. Crowdstrike is a corporate MDR solution (think antivirus on steriods). They have nothing to do with Microsoft or Windows updates. Corporate security/IT chose this particular piece of software because they are the market leader, somewhere around 25%. The problem was cause by Crowstrike themselves who pushed out a faulty update. How this happened we won't know until an offical post mortem report is written. Modern MDR/Anti-Virus solutions run in Ring 0 (Kernel) level of the operating system which is the highest level of access for Windows and thus the most dangerous when things go wrong. It needs to however, live here to operate sufficently and do the things it needs to do to stop threat actors.

The preliminary information that has come out so far is the file that Crowstrike pushed contained nothing but zeros. Because this file lived in such low level in the operating system, it caused Windows to boot loop. How this file came to be containing no information and why it was not caught in QA before being pushed is yet to be determined.

The reason why this is so devestating is because it requires physical access to each machine if the machine does not have out of band mangement (Intel vPRO/IPMI comes to mind but this has vunerability issues of it's own) in order to remove the faulty file. Bitlocker (Which is a microsoft feature for windows) comes into place in all this as a device encryption feature in windows. This prevents someone from removing the drive and placing it into another computer OR booting from a USB drive and manipulating the underlying Windows OS (Remove Passwords, etc) However bitlocker is actually doing what it's designed to do in this case and isn't a culperate. The bitlocker recovery key is needed to get to a command prompt in recovery mode in order to remove the fault Crowdstrike update.

TLDR: This is entirely on Crowdstrike. They are a third party vendor. Nothing to do with Microsoft for a change.



Backend systems are different. Rumor on the block is Crew360 is the cause for our pain. I don't know much about the software or it's backend Database but apparently this is our archillis heel at the moment.

Originally Posted by ancman

What does it accomplish? The OCC is already aware if there’s an unassigned crew position on a flight. They have all of the data that they need in front of them. There simply aren’t enough people there to process it.

I’m not opposed to sending a message over Flight Family if you think there’s something unique to your situation that they’re unaware of. But more than likely, they’re just going to add it to the pile.

Originally Posted by Hotel Kilo

No United didn't have our issues because they are not as fully infested with Microsoft as we are

Originally Posted by Hotel Kilo

They are not as fully infested with Microsoft as we are. That's why they got up and running quicker

Originally Posted by Hotel Kilo

You're absolutely wrong. The update is pushed directly from Crowd strike into Microsoft software. It is not a windows update per se from Microsoft. Microsoft contracts with crowd strike and they are allowed to deploy their software directly. It's been like this for years

Up until Friday morning there was never an issue.

Originally Posted by Viper25

you keep saying this. They are NOT aware. They have no idea where people are. A FF or ACARS message saying that XYZ is here to work the flight (that Tracking didn’t know about or forgot about) can be a huge help. Heard numerous anecdotes proving exactly that from both pilots and FAs.

Originally Posted by ancman

That’s true of American. United has a more comparable level of Microsoft deployment. Their impact on day one was very similar to ours.

United has cut their cancellations roughly in half every day since Friday. They’re down to only 17 cancellations today, vs our 697 (and growing). Our cancellations numbers have remained constant since Friday, with no sign of improvement at all. On Sunday we actually saw MORE cancellations than the previous two days.

Issues like this will always arise. System redundancy and adequate support staff is vital. Management is doing us a disservice if they continue to try to pass the buck to a vendor.

Originally Posted by biigD

Source?

(filler)