A blog series that shines a spotlight on real-world moments where network engineers use BackBox security-centric automation to save their organizations from costly downtime and surprises.

Synopsis

All firewall vendors require that their customers keep their certificates up to date. It’s a mandatory but time-consuming and tedious administrative task that’s error-prone and usually happens at an inconvenient time, making it ideally suited to be solved with automation. Here, we use a recent real-world example with Palo Alto to demonstrate what can happen without the benefit of automation and the value automation delivers.

On November 10, 2023 Palo Alto released a knowledge article and customer advisory informing customers that the PAN-OS root and default certificates would expire on December 31. This was just seven weeks-notice, five if you exclude the holiday weeks in between.

Using the BackBox Network Automation Platform, customers were able to save hundreds of hours of tedious overtime work by tracking and updating certificates proactively and well in advance of the deadline.

Problem

Palo Alto describes the problem best:

“On December 31, 2023, the root certificate and default certificate for Palo Alto Networks firewalls and appliances running PAN-OS software will expire. If you do not renew your certificates before they expire, your firewalls and Panorama appliances will no longer establish new connections to Palo Alto Networks cloud services, which will impact network traffic and potentially cause a network outage when existing connections terminate and attempt to reconnect due to network changes, configuration changes, or unforeseen events.”

Judging by the thread on the customer community board, the impending deadline certainly caused some panic, reaching 35 pages of discussion within a few days before it was closed for comments.

Impact

There are two primary ways to assess the impact of this.

1. Mitigation is not quick enough, and certificates expire.

Expired root and default certificates can have devastating effects; cloud services, browsers, and operating systems will no longer trust your firewalls or management servers. This will lead to significant disruption, almost certainly a network outage, and serious problems for services like threat protection systems.

At best your business might grind to a halt because your perimeter is down, at worst the integrity of your entire network could be at risk.

2. You have enough time and resources to mitigate before the deadline, but the logistics entailed are draining.

While this issue might not take down your business, it is still significant for a lot of companies. In most cases an upgrade is required which, while a nuisance, is probably manageable within a five-week window for a small to medium enterprise. Every network engineer has to cater to emergency actions and planned maintenance with very short notice.

However, consider the large enterprises and service providers who manage hundreds or perhaps even thousands of devices. For very good reason, they will have a significant amount of process to follow for any change – testing, documenting, change approval, scheduling with impacted business units or end customers, etc. Even if we disregard the time it takes to prepare for this work which can vary widely, it’s estimated to take around three hours to upgrade each HA pair.

Think also about the time of year and the impact on customers in the retail industry during their core revenue period. Ecommerce sites are typically on lockdown until the end of the year, so special permission is required for any change and the pressure to minimize errors is enormous.

Mitigation and Avoidance

I don’t think there is anyone working in NetSecOps who would disagree that avoidance is the best solution to any given problem. In this case, the Palo Alto customers who track certificate expiry would have had a lot more than seven weeks-notice to get ahead of this.

One BackBox customer has around 250 HA pairs of Palo Alto firewalls in their estate. Thanks to BackBox, they worked through this potential disaster in a calm and controlled manner with months to spare. Here’s how:

1. A regularly run IntelliCheck told them three months in advance that the certificates would be expiring.
2. They planned and scheduled upgrades, engaging early with their customers and internal stakeholders.
3. They used BackBox to automate the upgrade of these devices. Instead of one engineer working nights to upgrade one or two HA pairs at a time, they automated through BackBox and had that same engineer oversee 20-25 HA pairs per maintenance window.

Outcomes

In this instance, Palo Alto could have alerted their customers sooner. But to be fair to them, they pulled out the stops and supported their customers as best they could under challenging circumstances. Vendors are always measured on how they respond to such circumstances. So, while it’s uncomfortable at the time, it was positive to see support teams shining.

However, the real winners were the NetSecOps teams that used the BackBox Network Automation Platform. They avoided the issue completely and didn’t even realize a potentially catastrophic event was looming for the business.

BackBox customers were able to:

• Understand early on when those certificates would expire.
• Had the time to communicate and plan for the upgrade.
• Leverage automation to go from 750 hours to 30 hours to upgrade 250 HA pairs (in this example). That’s a mere 7 minutes per HA pair vs. 3 hours.

Conclusion

For an MSSP or internal NetSecOps team, having the right tools to head off events such as this is critical to not only delivering a world-class service but also avoiding situations that build stress and lead to burn out.

The mantra Happy Teams = Happy Customers says it all.

See for yourself how to use BackBox to automate certificate management. Take 20 minutes and watch the webinar on-demand.