Creative Commons
Supply chain – it’s a term and topic that is now discussed around dinner tables as families and friends discuss and debate COVID-19’s spotlight on the US dependence on other countries to provide the essential products and materials we need in time of crisis. The other dinner table discussion seems to focus around cybersecurity. With a precipitous increase in attacks and exploits in this new remote work environment, IT pros and those working from the home office are equally concerned.
While the focus of supply chain discussions has largely been around medicines and emergency supplies, there is another conversation that has been simmering in the tech sector for a long time. Flare ups occur every time the press reports on a suspected exploit found in infrastructure. How dependent can supply chain infrastructures become on foreign suppliers (or suppliers with factories based in other countries) before they become overly dependent? And how can an IT infrastructure manufacturer (in this case server) assure that the components being used in a bill of materials (BOM) are genuine and contain no microcode or other components that can be used to exploit that equipment over time? We will address this over the next few paragraphs.
Supply chain concerns are legitimate
The supply chain can be used as another attack vector for bad actors to exploit data, be it hackers looking to hold IP and sensitive information for ransom, or nation states looking to wreak havoc or disable critical functions of our companies or government. Such actors can insert motherboard implants that can go overlooked or insert malicious microcode that can create a backdoor once a platform is in production. Additionally, components such as a baseboard management controller (BMC), the control plane of the server, can have built-in vulnerabilities. These are all exploits that are not just theoretically possible – they have, in fact, already happened.
We often talk about cybersecurity in terms of perimeter defenses such as firewalls, or access control from companies like Aruba. IT organizations that are more serious about security look to technologies like HPE’s designed Silicon Root of Trust (SiROT) as the starting point where cybersecurity starts. While SiROT is a critical and fundamental element of a cybersecurity strategy, the reality is that cybersecurity starts in the supply chain –ordering the parts and components that go into the server. From storage and memory to the CPU, to the inductors, capacitors and resistors that go on to the motherboard.
Strangely enough, securing the supply chain is not just about security. It’s also about ensuring quality that is assured through authenticity. One of the challenges that exist today is ensuring the components that populate a server at the time it is stored at in a datacenter are the same components that were in that server at time of assembly. And that those parts are genuine manufacturers’ parts.
Supply chain is really complex
Per John Grosso, Vice President of Global Operations Engineering, Global Supply Chain at HPE, the average 1U or 2U ProLiant rack server has between 3,500–4,000 components. That is, 3,500–4,000 components that have to be tracked across hundreds of suppliers around the world—checked for security and for quality purposes.
The HPE supply chain is complex.
Moor Insights & Strategy
Consider the very simplified graphic above. The team at HPE (or any manufacturer) must ensure quality and integrity from left to right. Meaning, every component coming from every supplier is authentic and untainted as it leaves the suppliers factory and arrives at HPE’s manufacturing facility. The team then must ensure the servers are assembled with those very same authentic components and the integrity of the server is intact. After assembly, every server must be tested prior to shipping out to customers or distributors and resellers (in the case of the indirect sale, HPE must ensure that these servers are not modified or compromised in any way as they sit on warehouse floors, ready to fulfill orders). Upon arrival at a customer’s datacenter, HPE must ensure that server boots up with the hardware, firmware and software components that were installed when the server left the factory floor.
But, how does this happen? Grosso described his team’s approach to driving integrity across the process, and it’s quite comprehensive. He uses a term called roving cyber validation, whereby team members embedded with suppliers perform regular audits and informal spot checks on a regular basis to ensure the genuineness of components. As components are shipped to HPE factories, random x-raying takes place to ensure no tampering took place during shipment.
As servers are assembled in HPE or partner facilities, a cryptographic manifest is built after assembly and validation of components is complete. This manifest is attested at the customer site at first boot at the customer’s datacenter, through HPE’s Silicon Root of Trust (SiROT).
Supply chain management does not end at first boot. HPE (and other server vendors) must be able to ensure the integrity and quality of that server throughout its lifecycle. Through maintenance, upgrades and repairs, Grosso’s team is charged with ensuring the integrity of that server.
In the case of HPE, SiRoT and other utilities built into the company’s integrated lights out (iLO) management platform can immediately detect, remove and recover a server from malware and ransomware (to learn more about this, check out my coverage here and here).
How does HPE do it?
During a time where “buy American” is starting to gain steam, it’s unrealistic to believe that the server supply chain can be brought on the shores of the US to guarantee security and quality. This may be an unpopular view, but it’s realistic. Given this reality, infrastructure companies need to be vigilant about these things. And this commitment to securing the supply chain must be viewed as a pillar of a company’s strategy.
Securing the supply chain for HPE appears to be equal parts organizational, technical and cultural. And based on conversations I was able to have with Grosso and Security CTO Gary Campbell, this is nothing new. Campbell says the seeds of today’s focus on supply chain management were sewn about 10 years ago in a briefing he and Antonio Neri (now CEO) had with a large government customer.
In 2014, HPE started to develop a holistic security architecture that could help the company in its fight with the counterfeiting of products and overall security. Out of this effort, SiRoT was developed and implemented across the HPE portfolio.
Upon Neri’s appointment to CEO, one of his top priorities was to embed a “security first” mindset across HPE, understanding this could be a real value to companies of all sizes and a true differentiator in the market. It feels as though this message has permeated the company. Security is a key messaging pillar around every product the company introduces to the market, and its PointNext services has a very healthy consulting practice focused on cybersecurity.
One of the interesting things I learned from speaking with Campbell was the fact that HPE is the only server company to design and develop its own BMC. Why is this important? Think of the BMC as the control plane of the server. It is the lowest level management interface and provides the basis for all of the physical monitoring of a server’s condition. A compromised BMC can lead to a compromised server, and as previously mentioned, there are many news articles about this very thing happening. By developing its own BMC, HPE not only ensures the security of its servers, it has the ability to enable greater controls through its iLO management technology.
Bringing in Grosso and centralizing the management of the product lifecycle under his direction was a smart move by HPE. This enabled a single view of the product, spanning design, NPI (new product introduction), supplier quality, factory output and customer quality. Why does this matter? It enables a critical input to the product requirements and development process, ensuring security is fleshed out and given appropriate consideration across all stages of product life.
To ensure the team was being complete in its thinking and efforts, a Supply Chain Center of Excellence (COE) was built with representation from across the company. Its charter included three areas–capturing the needs (and feedback) of customers and the market, sharing of best practices across the various teams and ensuring consistency of security practices across all product lines.
Finally, to make sure product and supply chain security remains a priority to HPE, its board of directors (BoD) has a committee headed by Mary Agnes Wilderotter that receives quarterly reports on the status of end-to-end product security, including the supply chain.
What’s next?
Considering the typical server has 3,500–4,000 components (upwards of 7,000 components for converged infrastructure), it is hard to envision shifting the supply chain entirely to domestic suppliers. However, companies like HPE continue to work on reducing their dependence on suppliers who may not be able (or willing) to deliver in a time of need or crisis. As Grosso says, his team never stands still.
Given the scrutiny the government has put on foreign suppliers over the last couple years and the bright spotlight COVID has put on supply chain, I do expect to see further development from companies like HPE in ensuring these risks around dependency are not only mitigated, but minimized or removed.
Closing thoughts
As an analyst who has experience as an IT executive, I can fully appreciate the approach HPE takes to supply chain security. I never considered the integrity of the servers coming into my datacenter (or the quality of their performance), because I never had to worry. The upfront work of companies like HPE simplified my life and allowed me to deploy and run infrastructure with one less thing to worry about.
While securing the supply chain may not be as cool to talk about as edge computing, data analytics or cloud-native application development, it is arguably the most important consideration in choosing infrastructure to enable these environments. It’s something we should all thinking about, even after this COVID craziness passes.
Disclosure: My firm, Moor Insights & Strategy, like all research and analyst firms, provides or has provided research, analysis, advising, and/or consulting to many high-tech companies and consortia in the industry, including HPE. I do not hold any equity positions with any companies or organizations cited in this column.
