Southern Company Builds SBOM for Electric Power Substation

The utilitys software bill of materials (SBOM) experiment aims to establish stronger supply chain security — and tighter defenses against potential cyberattacks.

S4x24 – Miami –
Energy giant Southern Company in the past year set out to inventory all of the hardware, software, and firmware in equipment running in one of its Mississippi substations in an effort to create a
software bill of materials (SBOM)
for the operational technology (OT) site.
The SBOM experiment began with Southern Companys cybersecurity team traveling to the Mississippi Power substation to physically catalog the equipment there, taking photos and gathering data from network sensors. Then came the most daunting — and at times, frustrating — part: acquiring
software supply chain
details from the 17 vendors whose 38 devices the utility identified in the substation during its reconnaissance mission.
Alex Waitkus, principal cybersecurity architect at Southern Company and head of the SBOM project, said collecting and correlating information on all of the hardware, software, firmware, and interdependencies in the systems for the
SBOM
supplied the energy company with a deeper understanding of all of the software components in the substation and their potential exploitable vulnerabilities. Prior to the project, Southern had visibility into its OT network assets there via its Dragos platform, but software details were an enigma.
We had no idea what the different versions of software we were running before launching the SBOM project, he said in a presentation here this week at the S4x24 ICS/OT security conference. We had multiple business partners who managed different parts of the substation.
That meant consulting with physical security and maintenance teams, for example, to assist in the utilitys gaining the full picture of the substations software supply chain. Southern cataloged a total of 18 control network system devices from two vendors; eight physical security devices from three vendors; four cybersecurity and telecommunications devices from four vendors; eight OT monitoring systems from six vendors; and one OT fault system device from one vendor.
The next step in the project was gathering SBOMs from each of its vendors represented in the substation. But there were roadblocks, as nearly 60% of the vendors Southern contacted for their products SBOM information declined to provide the utility with the information. And on average, it took 60 days and a dozen meetings for Southern to actually receive in hand SBOMs from the cooperating vendors.
So we were updating firmware [in some cases] by the time we received the SBOM, Waitkus said. That meant an outdated SBOM, added Waitkus, who co-presented Southerns findings here at S4 with Matt Wyckhouse, CEO of software supply chain risk vendor Finite State. The project spun out of an OT SBOM roundtable organized by Wyckhouse at last years S4 conference.
SBOMs are a sort of list of the ingredients in a software program — the components and their versions — aimed at providing transparency in the software and its makeup to identify possible security weaknesses and vulnerabilities. Southerns project was especially challenging because creating an SBOM for an OT environment comes with more hurdles than for an IT network, mainly because industrial networks often have older software in legacy equipment critical to supporting uptime for industrial processes. And getting information on that old code is not easy.
Supply chain transparency is an interesting issue that I never thought Id get into, Waitkus told Dark Reading in an interview. But Southerns SBOM experiment, he said, illuminated three major security benefits for the power company: NERC CIP (North American Electric Reliability Corporations Critical Infrastructure Protection) compliance management, vulnerability management, and software patching prioritization.
When you have a lot of substation sites, a lot of people arent always comfortable rolling out firmware updates remotely … so you have a lot of people going to sites to do work. With proper vulnerability management integration, you can prioritize patching through vulnerability and exploitability analysis — and potentially drive lower overhead for those updates, he explained.
SBOMs also can provide visibility into a threat before it escalates, he noted. Think Log4j: it took almost two years to get a full readout on which software was vulnerable to the flaw, for example, he said. SBOMs can help identify that and make it easier for them [software suppliers] to know if they have a Log4j in the future.
Waitkus said he envisions SBOMs as useful tools in the utilitys procurement process as well, allowing Southern to gain deeper visibility into software products during the kicking-the-tires phase: Whats your third party code? Whats your copyright license?
In their presentation, Waitkus and Wyckhouse admitted the project didnt acquire all of the data they had hoped it would, and because of the lack of information from vendors, the overall quality of the resulting SBOM was not ideal.
Some vendors were eager to assist Southern in identifying their software, but most were resistant. For example, one vendor told Southern in an email that the SBOM information on its particular product was not available because it predates SBOM requirements under the time frame of the Biden administrations 2021
executive order
for critical infrastructure security.
Southern didnt just take the vendor SBOMs they received at face value, however. Waitkus and his team created scripts to analyze the SBOMs for their accuracy to confirm they had correct component and code dependency data. In one case, Southern found in its firmware analysis that the vendors SBOM was out-of-date, missing 72 components. Several other vendor SBOMs also were missing component and dependency data.
Software vulnerability information provided by Southerns vendors also required vetting to ensure it was accurate and timely, so Waitkus and his team mapped the SBOMs to vulnerability databases. Waitkus and his team ran vulnerability assessment and analysis tools to bolster the verification process and brought in independent vulnerability testers to weed through the bugs.
The goal was to ferret out vulns that were exploitable in the substation systems. In some cases, that meant that a vendors own vulnerability exploitability analysis included with the SBOM didnt show the true picture of threats to Southern.
We started to understand that the sensational number of vulns [in some vendor reports] really wasnt that big, Waitkus told attendees. In one open source package, for example, there were 3,000 vulns listed in the vendor-supplied SBOM, but Southern whittled that down to just seven that were actually exploitable, he said.
In another case, a vendor had pegged a vuln as nonexploitable, but Southern found otherwise in its own analysis. There were public exploits on the interwebs for us to use to prove it, he said in his presentation. Trust but verify.
Southern Company plans to operationalize the SBOM program but still has some work to do to before it can go to production. One challenge: It will require restructuring some vendor contracts to include SBOM requirements, Waitkus told Dark Reading.
SBOM sharing is clunky right now, Finite States Wyckhouse noted in the presentation. If youre just collecting SBOMs and you cant do anything with the data, they are just JSON documents in a folder.
Meanwhile, a next phase of the OT SBOM project is about to kick off. Schneider Electric, MITRE, Ameren, EPRI, and Scythe, will join Southern and Finite State in a US Department of Energy-funded effort to automate some elements of the Southern SBOM project — including inventory, SBOM collection, verification, vulnerability and exploit analysis.

Last News
▸ Researchers create BlackForest to gather, link threat data. ◂ Discovered: 23/12/2024 Category: security	▸ Travel agency fined £150,000 for breaking Data Protection Act. ◂ Discovered: 23/12/2024 Category: security	▸ 7 arrested, 3 more charged in StubHub cyber fraud ring. ◂ Discovered: 23/12/2024 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
Southern Company Builds SBOM for Electric Power Substation