Search

Notices

IT outage

Thread Tools
 
Search this Thread
 
Old 07-22-2024, 05:37 PM
  #211  
Gets Weekends Off
 
Joined APC: Mar 2007
Position: Petting Zoo
Posts: 2,090
Default

Originally Posted by velosnow
Yep, we'll get the usual mea culpa and empty promises to do better. Meanwhile, we'll keep doing Doing Everything Like The Amish.
Do the Amish cancel 1000 quilts four days after a vendor doesn't deliver a needle?

​​​​Comparing our IT to the Amish is...slanderous to Amish
Sputnik is offline  
Old 07-22-2024, 05:46 PM
  #212  
In a land of unicorns
 
Joined APC: Apr 2014
Position: Whale FO
Posts: 6,593
Default

Originally Posted by Transit
The only part correct here is Crowdstrike pushed an update that affected Microsoft Software (Windows). It did not affect Linux, MacOS for a variety of reason but mostly because those channel updates were not full of zeros like the Windows version. Microsoft does not contract with Crowdstrike to install their software. This is chosen by the corporate enduser. Computers without Crowdstrike installed were not affected for this reason.
Linux/MacOS don't have the threat the channel file #291 is used for, so that's why they weren't affected. I also don't think a Linux kernel module would crash because of a page fault like this. Nerd hat on for a second (It's been almost 2 decades since I was a dev so some terminology might be a bit off). The issue was that the corrupt channel file messed up an array that's used to look up memory addresses. Either creating a null pointer or just a corrupt memory address. The driver that crashed, csagent.sys tries to move a value to an address pulled from the array (I think it's mov r9d to r8, r8 being the corrupt memory address), and in this case this address was outside paged memory, that causes Windows to crash. Windows has to crash in a situation like this, because moving things to unallocated memory can cause your entire computer to get all effed up.

This is interesting in many ways. How the heck did they code a kernel-level driver so poorly that a simple corrupt setting file causes an unhandled error. This is software testing 101-level stuff. Feeding a null and/or corrupt setting file is one of the first things you should test for when testing a driver, and when it's a boot-start driver file, they should test that 10 times longer than a normal user mode driver. The fact that the driver does not verify any data in the channel files makes one wonder how secure that Falcon software is in itself.

Interesting to see how Crowdstrike manages to stay afloat after this. The damages are easily in billions, this is the biggest IT outage in history. I think they have had one barely profitable year in their entire history.

As you said, this is not a Microsoft issue, this is 100% on Crowdstrike. And based on the forensics people have been doing, it seems like this is a junior dev-level error in the heart of their software.

Last edited by dera; 07-22-2024 at 06:09 PM.
dera is offline  
Old 07-22-2024, 05:55 PM
  #213  
Gets Weekends Off
 
DeltaboundRedux's Avatar
 
Joined APC: Nov 2020
Position: Enoch Powell Enthusiast
Posts: 2,283
Default

Always found the Flight Aware Cancellations page interesting.

Delta data seems slow to update. No idea why.

Best way to use it is wait 48 hours then look backwards.

(Looks backwards 48 hours)

My god.

Crowd Strike? I have questions.
DeltaboundRedux is offline  
Old 07-22-2024, 07:14 PM
  #214  
On Reserve
 
Joined APC: Jul 2024
Posts: 12
Default

4 days in a row of at least 1,000 cancellations and 1,500 delays.

Thanks for stealing thousands of dollars out of all our profit sharing checks Ed.

Already over 200 cancellations for tomorrow...
CloudMonkey is offline  
Old 07-22-2024, 07:27 PM
  #215  
Gets Weekends Off
 
Joined APC: Apr 2018
Posts: 3,238
Default

Originally Posted by CloudMonkey
4 days in a row of at least 1,000 cancellations and 1,500 delays.

Thanks for stealing thousands of dollars out of all our profit sharing checks Ed.

Already over 200 cancellations for tomorrow...
When is this going to end? I've never seen a doom loop this bad...
m3113n1a1 is offline  
Old 07-22-2024, 07:39 PM
  #216  
Gets Weekends Off
 
Joined APC: Jun 2022
Posts: 1,437
Default

Originally Posted by m3113n1a1
When is this going to end? I've never seen a doom loop this bad...
They just need to shut everything down for 24-48 hours. If you’re not on a trip stay off the airwaves. Find out where everyone is thats out on the line and then put them in rest at location. 24 hours to rebuild to 50%…48 hours to ops normal.

i did (not) stay at a holiday inn last night

They should have done this Saturday night as soon as they realized that folks are not where icrew thinks they are. Its over at that point. Might as well take the loss and reset.
Hubcapped is offline  
Old 07-22-2024, 07:42 PM
  #217  
Gets Weekends Off
 
Joined APC: Jun 2015
Posts: 1,757
Default

Originally Posted by dera
Linux/MacOS don't have the threat the channel file #291….

This is interesting in many ways. How the heck did they code a kernel-level driver so poorly that a simple corrupt setting file causes an unhandled error. This is software testing 101-level stuff. Feeding a null and/or corrupt setting file is one of the first things you should test for when testing a driver, and when it's a boot-start driver file, they should test that 10 times longer than a normal user mode driver. The fact that the driver does not verify any data in the channel files makes one wonder how secure that Falcon software is in itself.

They just posted some security footage from crowdstrike. Let me see if I can re-post:


Last edited by Planetrain; 07-22-2024 at 08:10 PM.
Planetrain is offline  
Old 07-22-2024, 07:54 PM
  #218  
Gets Weekends Off
 
madmax757's Avatar
 
Joined APC: Jun 2010
Position: seated - facing forward
Posts: 1,070
Default

Originally Posted by CloudMonkey
4 days in a row of at least 1,000 cancellations and 1,500 delays.

Thanks for stealing thousands of dollars out of all our profit sharing checks Ed.

Already over 200 cancellations for tomorrow...
Im sure Ed’s profit sharing / bonus which is usually stock options will be millions less. I doubt it’s something he wanted either.

Imagine the phone calls he is getting from investors, He probably needs eyes in the back of his head right now.
madmax757 is offline  
Old 07-22-2024, 11:42 PM
  #219  
Gets Weekends Off
 
Joined APC: Jan 2019
Posts: 872
Default

Originally Posted by CloudMonkey
4 days in a row of at least 1,000 cancellations and 1,500 delays.

Thanks for stealing thousands of dollars out of all our profit sharing checks Ed.

Already over 200 cancellations for tomorrow...
There will be no profit sharing checks for mainline . In fact everyday this continues , when Valentine's Day comes around next year the employees will owe Delta money , LOL 💀
overqualified52 is offline  
Old 07-23-2024, 01:37 AM
  #220  
Gets Weekends Off
 
Joined APC: Jan 2008
Posts: 1,341
Default

Originally Posted by overqualified52
There will be no profit sharing checks for mainline . In fact everyday this continues , when Valentine's Day comes around next year the employees will owe Delta money , LOL 💀
Hey finally something you can be excited about.

perhaps now you don’t regret turning down Delta.

cencal83406 is offline  
Related Topics
Thread
Thread Starter
Forum
Replies
Last Post
DALFA
Delta
13
08-15-2016 06:19 PM
Makanakis
Delta
79
08-11-2016 10:25 PM
bottoms up
United
2
02-05-2015 12:54 PM
LNL76
Major
3
01-11-2014 04:45 PM
LeoSV
Hangar Talk
6
09-28-2007 05:17 AM

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



Your Privacy Choices