by Doug Clare, FICO
Most cybersecurity experts agree it is no longer a matter of if, but simply a matter of when your organization will become the victim of a cyber attack. While security has long been a matter of closing every possible avenue for breach, most people have realized that perfect security is not attainable. It’s now a matter of managing relative risk and having sound information that helps you to deploy limited resources in the most effective way possible. And with more companies outsourcing, risk not only applies to an organization’s own processes and practices but also to its partners (and their partners).
This explains the increase in demand for cyber risk scoring solutions. The idea that companies can quantify risk before someone tries to exploit them (directly or through their supply chain) has CISOs and CROs breathing a sigh of relief. Imagine not only being able to proactively deploy solutions to secure the network and address inconsistent user behaviors, but also identify risks posed by third parties with access to an organization’s systems.
But buyers beware because not all cyber risk scores are created equal. Here are the top things to consider.
- Forward-Looking View
You’re managing your business for the future, so a forward-looking view of cyber risk is important. Many of the security scores and ratings available today are engineered as current-state assessments or are simply reflective of yesterday’s issues. Make sure you ask the following question: What is the objective outcome of the underlying algorithm? What, specifically, is it trying to measure? If the provider has no answer to that question, you should keep shopping. Choose a solution that is designed to measure forward-looking risk. Those solutions that are based on predictive analytics will provide you with a score that reflects the likelihood of problems in the future, which is really what you’re most interested to know.
- Historical Data
Select a technology partner that has the capability to assess historical information, to gain a comprehensive view of the network hygiene practices of the organizations you’re looking to measure. If the security rating provider only starts looking at the organization when you ask for a score, the resulting rating will be based only on point-in-time evidence and won’t be reflective of long term trends or historic performance. Often the best way to predict future performance is by assessing past performance, and a snapshot of current network conditions isn’t going to get you there.
- Inferred Behavior
As a direct follow-on to the previous point, confirm information is continuously collected to assess not only if there are issues or vulnerabilities in the network, but how effectively the management practices and policies of an organization deal with them. Does the security rating take into account how long problems persist and how frequently they re-occur? Is the organization consistently capable of identifying and shutting down new issues, or do they linger and recur? This will help provide a holistic view of the effectiveness of an organization’s practices and policies and correlate these behaviors to risk and the likelihood of future trouble.
- Asset Curation
With the evolution of the workplace and the digital transformation, businesses are undergoing, accurately measuring a company’s Internet footprint is one of the most challenging barriers to overcome when scoring cyber risk. This includes the proper identification of network assets used by a specific enterprise, or division of an enterprise. Make sure any solution you evaluate allows organizations to curate assets being evaluated quickly and easily, to ensure there are no gaps in what is being actively reviewed.
- Model Performance
Assuming you’ve found a forward-looking cyber risk score based on an appropriate objective outcome, the next question is model performance. As with any predictive model, the goal is to reliably forecast that outcome with the fewest number of false positives. Two measures you may want to check are the Area Under the Curve, or AUC, and the dynamic range (or odds range) of the model output. AUC is an indication of how well the model can detect true positives while avoiding false positives (higher is better). The dynamic range of a model, on the other hand, indicates how well the model can identify and separate goods from bads (low-risk from high-risk). Models with a higher dynamic range will help you better discern comparative risk (e.g. between prospective vendors), and make better decisions at the margin.
- Model Governance
Governance and transparency are becoming increasingly important, especially in regulated industries, and are consequently an important topic not only for those organizations but for their entire supply chain. Organizations concerned with governance need to know the objective outcome of models being used to manage risk, and specifically how they perform. They need to ask questions like:
- What is the score-to-odds relationship?
- What data is the score based on?
- How stable are the underlying characteristics?
- How well does the model perform over time?
- As a user, will I be able to assess impacts to model updates before I put those changes into my production workflow?
- Can I choose the timing of the model upgrade, such that I am able to fully absorb the changes into my risk decision workflow?
If there aren’t good answers to these questions, there are likely gaps in best-practice model governance.
As cyber attacks escalate worldwide, every business needs to understand how good its defenses are – as well as those of partners or customers with access to its systems. When it comes to accessing enterprise risk indicators, CISOs and CROs find it can become a proverbial rabbit hole. But choosing the right cyber risk score provider ensures companies have the machine learning capabilities to harness massive amounts of data – which their systems collect every day – and turn that into actionable insight to thwart risk inside and outside the organization.
About the Author