Spotlight
Thought Provoking Leaders
JIM ROUTH
A Security Hall of Famer
Background of an Innovator
Jim Routh has been recognized for Cyber Security Leadership including the CSO Hall of Fame award, Shared Assessments Lifetime Achievement award, SINET Impact award, and others. He regularly publishes articles on innovative practices to improve enterprise resilience.
Mr. Routh has developed deep experience during his career as the CIO/CISO for companies including CVS Health, Aetna, JP Morgan Chase, KPMG, DTCC, and American Express. At Aetna, he developed one of the most mature converged security programs in the private sector.
He serves on the board for several organizations including Syn Ventures Advisory Board, Cyber Starts Advisory Board, and the Global Resilience Federation. He is the former Chair of the Health Information Sharing & Analysis Center (H-ISAC) and former board member of the FS-ISAC. He serves on the company boards of Acceptto, GrammaTech , Supply Wisdom and ZeroNorth.
Based on Mr. Routh’s experience building a continuous risk management program, Hybrid Pathways asked him to share which practical approaches and tools that he recommends to help companies move forward in this area.
Based on Mr. Routh’s experience building a continuous risk management program, Hybrid Pathways asked him to share which practical approaches and tools that he recommends to help companies move forward in this area.
How has third-party governance (supply chain resilience) evolved over the past few decades?
"This is a great place to start. There is a foundation of practices, approaches, and techniques that have been codified and baked into third-party risk management framework control objectives and requirements. These practices have been reaffirmed and established by AICPA certified public accountants that do 3rd party assessments. It is essentially an ecosystem itself that is based on one fundamental premise, which is that you look at the risk of a 3rd party to the enterprise at a point in time. The process of recertifying your vendor’s risk is conducted on an annual basis. This annual, point in time process has been firmly established for the past several decades and re-affirmed by regulators and industry regulations. It does not consider the explosion of growth in cloud computing for third parties and enterprises. It also does not take into account the dependence and inter-dependence of global 3rd party suppliers and services. Many of those 3rd party products and services are running on 3rd party cloud platforms, creating 4th and 5th party risks. Companies are dealing with a dated process that only looks at their universe of 3rd party partners at a point in time once a year. Contrast this approach with how enterprise companies evaluate operational risk. Data inputs for operational risk are considered on a continuous basis from multiple internal and external sources. So why don’t enterprises do that continuous risk assessment for 3rd parties?"
The inherited process of point-in-time annual approach to 3rd party risk management does not make sense in the world of technology, globalization, and supply chain dependence that we have today. We need to break the inertia that we’ve faced for the past two decades and apply some innovation and work towards a continuous 3rd party assessment process where changes in risk trigger a response not a calendar date.
What are the drivers of change for any enterprise related to supply chain risk management?
"Businesses are more dependent on their global network of suppliers than ever before. There are many important business drivers for these relationships and they come with increased risk and uncertainty. Technology changes, impacts from climate change, a pandemic, economic and political uncertainty, infrastructure disruptions, and many other factors can all affect your supply chain. We started our discussion today talking about the heat wave we are experiencing. Global warming is causing more frequent disruptions such as hurricanes, fires, earthquakes, and flooding that impact our partners and providers across the globe. We’re still working our way through the supply chain impacts from the COVID pandemic. For example, new cars can’t get chips in their supply chain so we are buying more used cars than ever before. Major business disruptions caused by software supply chain poisoning (NotPetya, Solar Winds, etc.) are more common."
Given the risks associated with this expanding security perimeter, it is not surprising that a majority of (if not all) enterprise companies have experienced a third-party incident. If your company has not experienced a third-party incident yet, it is only a matter of time.
It is a clear time to move towards continuous risk management processes for our supply chain. A continuous risk management model is actually more efficient for an enterprise and more cost-effective. There are investments that need to be made to make the changes, but once you have automated and built a foundation of data science for 3rd party risk management – this process will pay for itself over time through labor productivity and improvements in risk management.
What is the challenge of building continuous risk management using near real time data feeds for third party governance?
"The biggest challenge is weaning yourself away from the annual assessment process. What we must do is shed the baggage that comes along with this dated approach. Instead of responding to high-risk vendors based on an annual assessment, we need to apply our resources when risk changes for any of our vendors. To do this, we need to build on a foundation of data science. Initially, we will want to organize our suppliers into similar domains and more specific sub-domains. We can then establish the basic control requirements appropriate for that domain and more detailed controls based on the risk profiles of the sub-domains. If, for example, there are software vendors in a domain, a sub-domain might be those software vendors that perform authentication. This sub-domain would have additional control requirements beyond those set for software vendors that do not perform authentication. I’ve now customized my control requirements based on the vendor’s risk to my environment. The vendor is then assessed against the domain-specific requirements. I score my vendors on how well they meet the specific requirements I’ve established based on the impact on my business. This scoring can be used in many ways, from focusing on important vendor risk areas to help mitigate risks, to negotiating and even choosing the vendors that the enterprise wants to work with. A vendor’s risk score is not fixed, it can change as the vendor’s environment changes. To stay on top of our vendor’s risk, we will use data feeds to monitor the vendors regularly. This near-real-time information will influence and could change, the overall risk score of the vendor at any point in time. For example, if there is a supply chain poisoning event, like SolarWinds, data feeds can inform me of my suppliers that might have been impacted by this event. I need to monitor these incident databases because they will alert me if one of my vendors has publicly filed a breach notification. Using data sources that are monitoring real-time information about resilience and disruptions, I can better identify and focus my attention on the risks that are important to my business. The ultimate model is when the event will trigger an automated workflow response and I don’t even have to be involved. For example, if a vendor score declines due to a breach notification, an automated email can be sent to competing vendors issuing a request for a quote to replace the original vendor. Using scripts to monitor and respond to the trigger is the desired end game since actions are taken as risks change resulting in more accurate risk scores and better risk management with less labor cost."
"Once we have categorized and scored our vendors, and are using real-time data to monitor any risks, then we can implement the capability to automatically scan all 3rd party software regularly. This automatic binary scanning gives us more opportunities to surface, analyze, and respond to issues more proactively. For example, there are several software packages that any enterprise can buy today that run a binary vulnerability scan of any 3rd party software. Say your company uses Zoom, using Zoom software bill of materials, if I knew where there were specific vulnerabilities, I could set up triggers to monitor that risk. These scans can be applied to all software to surface risks. The administrator then becomes a data analyst. The more risk events (or thresholds) that we can set up to trigger an automated response, the more effective, efficient, and secure the enterprise becomes."
We are no longer spending time doing annual recertifications and assessments. The more automated process takes fewer people and it is improving the entire risk management profile of the enterprise. This automated process should be able to apply to all 4th and 5th party suppliers in the chain as well."
How do you get suppliers to be more open and share the information you need to build a successful continuous monitoring program?
"I consider the cybersecurity practitioners that work for our suppliers as my peers. If I have good relationships with them, our business transactions will be much easier with less friction. As an industry, we have a history of “beating up our suppliers” using the size of the enterprise wallet. We must change our behavior and stop using our wallets as a bludgeon, this is not a way to build community. As an example of peer-sharing, if I worked at a bank, I’m going to the FS-ISAC and sharing cybersecurity information with my peers to help the entire community. At Aetna/CVS and MassMutual we would work closely with the vendor security practitioners and share information on how to put good controls in place for their benefit which also benefits the enterprise. The goal of this peer-to-peer information sharing is to work together to solve problems and make all our enterprises more resilient. This effort takes years to achieve system results so you have to get started today."
"The enterprise has an opportunity to offer suppliers a carrot if those suppliers better meet the control requirements and are more responsive to changes in risk. Allowing logos use, participating in case of studies, making a vendor a preferred provider, are “carrots” that can positively motivate a supplier to improve security while building good rapport and community. These actions can help to incentivize the behavior that you want to see in your ecosystem. Those suppliers that work with the enterprise to meet a specified low-risk profile are the suppliers I might choose to switch to. By having redundant suppliers and choosing those with lower risk profiles, the enterprise now has redundancy and added resiliency along with the associated benefits."
"My belief is that if enterprises have person-to-person relationships with the cybersecurity professionals in the third-party community, then everyone benefits from the improvements in resilience across the entire community. "
"Another approach and this is very controversial so you may need to check with your legal department, but I recommend that you avoid putting security controls in your contract terms & conditions. We used to put control requirements into the contracts as Standard Terms and Conditions. This becomes the starting point that companies negotiate against, and then you end up losing control through the vendor negotiation process. Security control requirements change as threat actors adjust tactics. Contract terms & conditions don’t change given the labor it takes to re-negotiate annually. Rather than putting security controls in contract T&Cs, I recommend establishing the domain/sub-domain pre-determined essential controls. The enterprise should minimize the number of controls to what’s essential for that sub-domain. I share that information in a descriptive way (documented) with the vendor so that they understand the control objectives. I am open to different ways for the vendor to meet that control objective and I try to help them overcome any obstacles to accomplishing the control. As a customer, I can help to elevate the importance of investing in meeting a control with the vendor’s leadership. I’ve done this hundreds of times. Sometimes hearing the recommendations from the customer is more powerful and motivating than hearing them from your employees. Resilience in the relationship with a supplier does not originate from the contract terms and conditions. I’ve experienced a high level of responsiveness to the changing threat landscape from third parties where there were minimal [if any] control requirements in the contract terms and conditions."
"We need to treat vendors and suppliers as a community. The benefit of this approach is that our company and the whole community can then achieve higher resilience.
How do you identify 4th and 5th party suppliers?
"I don’t want to make this sound too easy because it is not. Our goal is to design the capability to monitor the multiple levels of suppliers. Conceptually, using the data aggregation services we can identify the business concentration risk for the enterprise. Let’s assume that you are working with a software developer that is writing code on behalf of the enterprise. If that software developer is using cloud computing infrastructure for the hosting of their development environment, that IaaS vendor is now a 4th party that is integral to the resilience of your developer. Some of these data services will not only identify the 4th party, in this case, the cloud infrastructure provider, but they can tell me what the business concentration risk is between my 3rd party and their IaaS provider, and the risk between the 3rd party and the enterprise. If this software provider is just creating a proof of concept that is not tied to any production, if they experience a cloud infrastructure outage, the risk to my enterprise is low. However, if that software development firm is doing a multi-million-dollar contract to build a production capability in an automated continuous pipeline, and has a significantly high concentration with one cloud service provider, then all of a sudden the business concentration risk of that 4thparty is now amplified. But if you don’t know any of this, you are only hammering on the 3rd party and missing the full risks across all parties along with the impact to the enterprise."
"In our previous example, the cloud provider was a 4th level supplier. However, in large organizations, there are multi-prong relationships and inter-relationships among vendors that can be very complex. The repository management capabilities may come from another entity (GitHub, GitLab, etc.). For example, AWS could be a 3rd party, 4th party, or 5th party provider within the same company. Large entities can have hundreds of vendor relationships, somewhere the enterprise itself is the 3rd party and must provide risk information. When you add cloud infrastructures into the mix, it gets really complex to sort out enterprise risk.
When we consider that software development today can be assembled with bits of code from developers on GitHub, that exposes another source of inter-related parties in our ecosystem. Millions of developers and millions of bits of code can be assembled like LEGOs to help get to market quickly. This large and growing attack surface will continue to be exploited."
"Our previous example focused on a software developer vendor, but we need to consider all parties in the supply chain, like our utility providers. Every enterprise depends on electricity. An exploited utility will put a strain on our cloud providers, our suppliers, our enterprise, and our customers. We are highly dependent on utility services, but this does not show up on our radar until we consider 4th party providers. None of our vendors can currently run without electricity. If a utility is targeted by a nation-state, which has happened many times, that’s going to put resilience strain on our cloud providers, our infrastructure, and ultimately our customers. This is something we have to consider."
"Enterprises today, at least need to understand and design for the 4th and 5th party risks. Today the data aggregation tools and services cover two categories: general business resilience information and cybersecurity vulnerability information. Going forward, a good enterprise 3rd party governance program will include both sources of risk data."
Seriously, Don’t Wait.
"Here’s an important confession, my enterprise could have been better prepared for SolarWinds. SolarWinds was not the first or most significant supply chain poisoning event, it was probably the 5th or 6th event that was significant. The most significant event occurred three years earlier in 2017. The methodology for SolarWinds was exactly the same as NotPetya/External Blue. NotPetya was the exact same attack vector, the malware was different, the backdoor was different, but the methodology was essentially the same."
"I experienced NotPetya in a material way as the head of the Health ISAC. I had to talk to the companies that were impacted by NotPetya to understand how the global threat was spread, and how to protect against this type of breach in the future. My confession and my error are that I did not apply all the learnings from this incident in 2017 before SolarWinds in 2020. I started to apply some of the recommendations, but I did not apply all the learnings that I could have and evolving control design."
"Supply chain poisoning is increasing because it is successful from a threat actor perspective. We need to better understand and continuously monitor all of our vendors in our extended ecosystem and specifically how their tactics are evolving. Software Supply Chain Poisoning is increasing as a threat vector and the attack surface for enterprises is increasing."
3 Key Measures
If you haven’t implemented these 3 things for your enterprise yet, then learn from my error and do them now to protect your enterprise against supply chain poisoning
Apply IAM requirements to manage the access to code repositories. Make sure you have a sub-domain for SaaS services, and within that for code repositories. You want to ensure that any DevOps team member that is opening an account in GitHub has MFA established and complex password requirements.
01/
"The password compromised in SolarWinds was SolarWinds123. There are a couple of different ways to monitor this, either reactively or proactively. One way to monitor reactively is to parse emails for welcome emails from code repositories. Proactively, we can set up master accounts with pre-determined MFA. This is an easy step that we can all do. GitHub recently decided to mandate MFA."
02/
Third-party governance must include SaaS services, and a subset of the SaaS services are the code repositories. These code repositories must have control requirements.
"If there is an outage or business disruption for your code repo provider it could have a big business impact for you. This is another straightforward action that we can all do."
Learn how to implement workload run time protection capability.
03/
"This is an emerging category of software that is installed and running in any workload in any environment, but potentially in your code repository. It is not RASP technology, that is an earlier generation. This is a newer class using behavioral attributes that will provide alerts or blocks for workload runtime protection without impacting the code itself. There are several options in the enterprise market today that are worth considering. Try one, run it in specific workloads, then broaden it out to cover all your workloads on-premise and hosted in the cloud. This is a backstop to a software supply chain poisoning incident having an impact on your enterprise. If all your other controls are bypassed, this one will save your bacon and protect the enterprise from business disruption."
Inspired?
We are.
We are grateful to work closely with these thought leaders. Sharing our experiences helps everyone to grow.