C-00000291*.sys

July 19th – Like most days, I’m up early – take care of some personals, make coffee, and look for the latest in cyber news aka I hop on Twitter (X? we calling it that yet?) and look for a rabbit hole. Today I caught a text from my Dad first thing – sidebar my Dad introduces me as his son who works on computers.

I laugh to myself, my Dad’s general lack of trust and his keen instinct to question all information has him running this mainstream-hearsay “global outage” by his professional computer-fixing son.

On the heels of the previous shifts Microsoft outages involving their Azure cloud computing platform and M365 services was something a little more interesting. He was referring to US cybersecurity software company, CrowdStrike – whose Falcon platform, a next-generation antivirus (abbr. NGAV) and EDR vendor – whose Friday update did not go well. What’s an EDR? – glad you asked – EDR stands for Endpoint Detection and Response, which is what the industry deems next-generation security software that takes antivirus to another level – modern problems require modern solutions and this one is so good, all the ‘big dawgs’ use it.

With this early breaking news and no alarms yet from work I drop an article into Teams and proceed with my otherwise routine morning. A bike commute to work with a stop off at a local coffee shop, plus a Strava picture for my 4 followers. Meanwhile the world is seeing something like a billion Microsoft computers and servers with the infamous blue screen.

CrowdStrike is going to catch plenty of flack across blogs and on social media posts so I don’t need to throw more shade their way. I started with my anecdote, because I wanted to remind you, dear readers, that us IT folks are pretty regular people. Aside from our proclivity to technology, most of us feel that if we are doing our job you don’t even know we exist. Today many of us came in to a show and most of us had to manually visit users, enter a 48-digit BitLocker key, boot to safe mode, remove a file, and reboot. There’s a good chance that some of my colleagues are still out fighting the good fight tonight and this weekend. I’ll take this time to point out that it’s the guys/gals not in charge of departments, systems or patching going desk to desk listening to the same end-user comments and you are the real MVPs this week. Big props and thanks to the technicians being hands-on today!

Where does this leave us, what’s my message? First let’s talk a little bit more about the BSoD – Dave Plummer, Microsoft Operating System programmer aka the guy who created the Task Manager and other features in Microsoft you know, gives a solid (and with permission to share) presentation. Watch and drop a like/comment/subscribe etc. to boost his engagement. Let’s just say we should be calling this bug check Cerulean Screen of Death.

Second, we need to understand the wonderful wild world (www) we live in of connected and internet facing (IoT) everything requires immediate updating of signatures and patching of vulnerabilities. Some security updates have earned the esteemed “auto-patch” stamp of approval and receive multiple updates per day. We need to remember that even in this fast-paced, understaffed, under-funded world we technologists live in, that we exercise caution to the extent that acceptable risk allows. Find the balance, create a patch cycles to allow those less-than-critical machines to catch patches first (especially if you can’t test before release) and give yourself some hours before you patch the mission critical servers. A lot of us would have had a better day if we had any amount of buffer. It’s a huge problem to be dealing with servers and user PC getting ‘borked’ in the same patch window.

Finally, I think it’s worth mentioning that CrowdStrike found and removed the issue relatively quickly. While we saw a Y2K level event (Troy Hunt with the credit here @troyhunt on Twitter) it could have (always) been worse. If we IT Pros can leverage events like this help explain to our overstimulated managers that we still need to make time for due diligence and take precautions with even our most trusted vendors, we can find the equilibrium. Damned if you patch and damned if you don’t. Let’s all get back to no changes on Friday and please show some much needed love to your favorite IT professionals. Our vendors are not infallible and we need to protect ourselves from the excessive privileges and power granted, as well as our over-reliance on them.

0010