This blog series spotlights real-world moments in which network engineers use BackBox security-centric automation to save their organizations from costly downtime and surprises.

Synopsis

All firewall vendors require that their customers keep their certificates up to date. This mandatory but time-consuming and tedious administrative task is error-prone and usually occurs at an inconvenient time, making it ideally suited to automation. Here, we use a real-world example with Palo Alto to demonstrate what can happen without automation and the value it delivers.

On November 10, 2023, Palo Alto released a knowledge article and customer advisory informing customers that the PAN-OS root and default certificates would expire on December 31. This was just seven weeks’ notice, five if you exclude the holiday weeks in between.

By utilizing the BackBox Network Cyber Resilience Platform, customers saved hundreds of hours of tedious overtime work by proactively tracking and updating certificates before the deadline.

Problem

Palo Alto describes the problem best:

“On December 31, 2023, the root certificate and default certificate for Palo Alto Networks firewalls and appliances running PAN-OS software will expire. If you do not renew your certificates before they expire, your firewalls and Panorama appliances will no longer establish new connections to Palo Alto Networks cloud services, which will impact network traffic and potentially cause a network outage when existing connections terminate and attempt to reconnect due to network changes, configuration changes, or unforeseen events.”

Judging by the thread on the customer community board, the impending deadline caused some panic, reaching 35 pages of discussion within a few days before it was closed for comments.

Impact

There are two primary ways to evaluate its impact.

1. Mitigation is not quick enough, and certificates expire.

Expired root and default certificates can have devastating consequences. Cloud services, browsers, and operating systems will no longer trust your firewalls or management servers, leading to significant disruptions, network outages, and serious issues for services such as threat protection systems.

At best, your business might grind to a halt because your perimeter is down, and at worst, the integrity of your entire network could be at risk.

2. You have sufficient time and resources to address the issues before the deadline, but the logistics involved are exhausting.

While this issue may not cripple your business, it is still important for many companies. In most instances, an upgrade is needed, which, though inconvenient, is likely manageable within a five-week timeframe for small to medium-sized enterprises. Every network engineer must prepare for emergency actions and scheduled maintenance at very short notice.

However, consider the large enterprises and service providers that manage hundreds or even thousands of devices. For very good reason, they will have significant processes to follow for any change—testing, documenting, change approval, scheduling with impacted business units or end customers, etc. Even if we set aside the time required to prepare for this work, which can vary widely, it’s estimated to take about three hours to upgrade each HA pair.

Consider the time of year and its impact on customers in the retail industry during their peak revenue season. Ecommerce sites are usually on lockdown until year-end, requiring special permission for any changes, and the pressure to minimize errors is significant.

Mitigation and Avoidance

I don’t think anyone working in NetSecOps would disagree that avoidance is the best solution to any given problem. In this case, the Palo Alto customers who track certificate expiry would have had much more than seven weeks’ notice to get ahead of this.

One BackBox customer has approximately 250 HA pairs of Palo Alto firewalls in their infrastructure. Thanks to BackBox, they navigated this potential disaster calmly and with months to spare. Here’s how:

1. A regularly scheduled compliance check informed them three months in advance that the certificates would expire.
2. They organized and arranged upgrades, collaborating early with their customers and internal stakeholders.
3. They used BackBox to automate the upgrades of these devices. Instead of requiring one engineer to work nights upgrading one or two HA pairs at a time, they automated the process with BackBox, allowing that same engineer to oversee 20-25 HA pairs during each maintenance window.

Outcomes

In this instance, Palo Alto could have informed their customers sooner. However, to be fair to them, they did everything possible to support their customers under challenging circumstances. Vendors are always evaluated on how they respond to such situations. So, while it’s uncomfortable at the moment, it was encouraging to see support teams excel.

Nevertheless, the true winners were the NetSecOps teams utilizing BackBox. They completely sidestepped the issue and were unaware that a potentially catastrophic event awaited the business.

BackBox customers were able to:

• Understand early on when those certificates would expire.
• Had the time to communicate and plan for the upgrade.
• Utilize automation to reduce the time from 750 hours to 30 hours for upgrading 250 HA pairs (in this example). That’s only 7 minutes per HA pair compared to 3 hours.

Conclusion

For an MSSP or internal NetSecOps team, having the right tools to prevent events such as this is critical to delivering a world-class service and avoiding situations that build stress and lead to burnout. The mantra “Happy Teams = Happy Customers” says it all.

See for yourself how BackBox automates certificate management. Schedule a 30-minute demo for an interactive tour of the BackBox platform.