IT outage
07-24-2024, 07:16 PM
#312
Gets Weekends Off
Joined APC: Apr 2007
Posts: 4,613
EB has read it.
Listen, when 100% of your front line operational computer systems all take a dump at the same exact time - the magnitude of this is lost on many here - no book by Taleb is gonna help.
This was akin to a cyber attack, albeit an "innocent" one. The very group hired to protect our systems from such events, ironically was the very source of this system wide crash
I've seen the magnitude at how far this affected our systems, which are pretty robust compared to many other places. Rebuilding the operation from basic scratch in a little under 4 days was pretty impressive to watch.
As in the past our ability to communicate between credws and Mecca and back must be fixed. We seen this many, many times before in previous melts, let's hope they get the message - finally.
Listen, when 100% of your front line operational computer systems all take a dump at the same exact time - the magnitude of this is lost on many here - no book by Taleb is gonna help.
This was akin to a cyber attack, albeit an "innocent" one. The very group hired to protect our systems from such events, ironically was the very source of this system wide crash
I've seen the magnitude at how far this affected our systems, which are pretty robust compared to many other places. Rebuilding the operation from basic scratch in a little under 4 days was pretty impressive to watch.
As in the past our ability to communicate between credws and Mecca and back must be fixed. We seen this many, many times before in previous melts, let's hope they get the message - finally.
07-24-2024, 09:10 PM
#313
Gets Weekends Off
Joined APC: Mar 2019
Posts: 1,013
EB has read it.
Listen, when 100% of your front line operational computer systems all take a dump at the same exact time - the magnitude of this is lost on many here - no book by Taleb is gonna help.
This was akin to a cyber attack, albeit an "innocent" one. The very group hired to protect our systems from such events, ironically was the very source of this system wide crash
I've seen the magnitude at how far this affected our systems, which are pretty robust compared to many other places. Rebuilding the operation from basic scratch in a little under 4 days was pretty impressive to watch.
As in the past our ability to communicate between credws and Mecca and back must be fixed. We seen this many, many times before in previous melts, let's hope they get the message - finally.
Listen, when 100% of your front line operational computer systems all take a dump at the same exact time - the magnitude of this is lost on many here - no book by Taleb is gonna help.
This was akin to a cyber attack, albeit an "innocent" one. The very group hired to protect our systems from such events, ironically was the very source of this system wide crash
I've seen the magnitude at how far this affected our systems, which are pretty robust compared to many other places. Rebuilding the operation from basic scratch in a little under 4 days was pretty impressive to watch.
As in the past our ability to communicate between credws and Mecca and back must be fixed. We seen this many, many times before in previous melts, let's hope they get the message - finally.
07-25-2024, 03:40 AM
#314
Gets Weekends Off
Joined APC: Feb 2008
Posts: 19,588
No. I don't have to acknowledge anything because none of those were points I was making or arguing.
Everything I need to know about our crew planning software/IT resiliency is in comparing outcomes of 3 airlines with the exact same starting condition. Crowdstrike can be blamed for Friday and Saturday. Maybe even startup Sunday. Anything beyond that is a Delta problem and it exposed significant shortcomings in our proprietary software suite. You can quote answers from AI bots answering questions no one asked all you want. But it took between 150-200% longer than our competition to recover. One of the biggest reasons people choose us over others was reliability. And we just gave them all a measuring stick to show that maybe we aren't as superior as we claim
Everything I need to know about our crew planning software/IT resiliency is in comparing outcomes of 3 airlines with the exact same starting condition. Crowdstrike can be blamed for Friday and Saturday. Maybe even startup Sunday. Anything beyond that is a Delta problem and it exposed significant shortcomings in our proprietary software suite. You can quote answers from AI bots answering questions no one asked all you want. But it took between 150-200% longer than our competition to recover. One of the biggest reasons people choose us over others was reliability. And we just gave them all a measuring stick to show that maybe we aren't as superior as we claim
07-25-2024, 04:53 AM
#316
Gets Weekends Off
Joined APC: Jan 2019
Posts: 871
Running an airline on bank hours during the biggest IROP in history did not help. Crazy to see CP offices empty after 6pm and on the weekends. And then no word from management until late Monday/Tuesday. If they had taken this problem seriously on Saturday it would have been cleaned up by Monday. It needed to be all hands on deck over the weekend, not on Tuesday.
07-25-2024, 06:38 AM
#317
Gets Weekends Off
Joined APC: Jan 2023
Posts: 1,510
It would have been simpler for you to say you've never read the book. The concept of anti-fragile is protecting against a single point of failure created by a "robust" system. Other airlines are better at exception processing, probably because they don't run as smooth during normal ops. It's good to admit we failed miserabley and grossly underperformed our peers. Making excuses because of the magnitude is how we set ourselves up for an even bigger failure next time. Read the book it's interesting, even if you aren't running the company.
The book? Yes I am familiar.
Thanks for your contribution
07-25-2024, 07:05 AM
#318
Gets Weekends Off
Joined APC: Mar 2024
Posts: 219
It would have been simpler for you to say you've never read the book. The concept of anti-fragile is protecting against a single point of failure created by a "robust" system. Other airlines are better at exception processing, probably because they don't run as smooth during normal ops. It's good to admit we failed miserabley and grossly underperformed our peers. Making excuses because of the magnitude is how we set ourselves up for an even bigger failure next time. Read the book it's interesting, even if you aren't running the company.
07-25-2024, 07:23 AM
#319
Can’t find crew pickup
Joined APC: Jun 2021
Posts: 2,231
I’m not saying mistakes weren’t made. And we HAVE to do better.
07-25-2024, 07:37 AM
#320
Gets Weekends Off
Joined APC: Apr 2018
Posts: 3,237
Yeah, but even during a normal IROP caused by, I don't know, a small storm in Atlanta or the Northeast it takes hours to get through to tracking or accomodations to get a hotel or rotation updated.
Thread
Thread Starter
Forum
Replies
Last Post
