Two Truths and a Dare

A fun little walk down memory late as it relates to the complexity of running a mission critical IT environment to end the year on a light note.

Truth #1

My first “tech job” was a disk drive company for unix workstations (at Box Hill Systems), the kind that was common on Wall Street trading desks in the 90’s.

I’ll never forget the “our drive failed” call from Standard & Poors, when I grabbed a big multi-drive unit and humped it down to their office on the subway… where I then wait for hours for them to find the machine connected to the failed drive!

I remember being surprised that they didn’t know where all their machines and equipment were. If you think about it, there were a lot of computers in any given building (even in 1990)!

Truth #2

If you think there’s a lot of physical equipment, try to count software services.

About 17 years after the incident at S&P, and before Microservices went mainstream, I was working for a company with a magical piece of software that could automagically discover apps running in a web app server and the apps communicating with them.

Since that’s a bit of a confusing sentence, let me just say, we installed our software at an e-commerce shop and they told us that it turned out an old version of their store was still running (and a processing orders) somewhere. They had no idea!

Imagine that? Having no idea that software generating revenue is running and selling your product? I have a million points to make about what could go wrong, but if you’re read this far you can probably imagine a few of your own, so I’ll spare you mine.

Another company, a major financial institution had an HR service they created for one specific use case. Our software discovered about 30 different applications using that service! The funny thing (if the story wasn’t already funny enough)… the service was still in development and most of those 30 applications were in production.

Talk about risk.

The Dare

If you’re any good at math, you realize my last story was from about 2007. And, 15 years have passed since then, so a lot has changed.

One of those things of course is the complexity of our IT environments.

Every router, switch, load balancer, IDS, and Firewall gets configured, and they’re (obviously) not all replicas of each other even if they are all from the same vendor (which I’d bet they’re not). They each get “touched” on a regular basis by a variety of well meaning, but stressed, human administrators.

Humans who make mistakes.

So here’s the dare:

I dare you to state with 100% confidence that:

1. you know that all of your network devices are still configured the way you deployed them,
2. each and every machine can be quickly restored or rebuilt from backup, and
3. your inventory system, even if it’s a spreadsheet, reflects reality.

Bonus: Hot Off the Presses

It’s not a network related but McGraw Hill’s exposed 100k students’ grades, their own source code, and digital keys between 2015 and now as a result of misconfigured Amazon S3 buckets.

That’s a headline that speaks for itself.