Organisations increasingly quantify the expected benefit of cyber investment when seeking funding. What I have seen organisations find more difficult is returning later to evidence whether that benefit was realised.
For organisations starting their cyber risk quantification (CRQ) journey, understanding their risk exposure in financial terms is an important first step. But I’ve found the more important question is what happens next. What decisions does this information help you make?
When building a business case for cyber investment, organisations might naturally look to strengthen their weakest capability or respond to an emerging threat. Risk quantification adds another useful perspective. The weakest capability may not offer the greatest opportunity to reduce exposure. Instead, improving a relatively mature capability could have a greater impact if it addresses an important driver of risk.
Modelling the expected impact of different investments gives decision-makers better evidence on where their spend could make the greatest difference. Importantly, it also provides a measurable expectation for how an investment should improve capability and change risk exposure.
It’s this expectation that creates the basis for returning later to test what actually happened.
Once a decision has been made, you now have something to measure. Did the investment deliver the capability improvement you modelled, and did that improvement reduce risk as expected?
.png)
In my experience, proving the first part isn’t always straightforward. Investment in a capability doesn’t automatically mean the intended improvement has been realised. A new control may need time to be rolled out across the organisation or embedded into existing processes and tested to establish whether it is operating as intended.
Depending on the capability, evidence of improvement could come from control testing, operational metrics or even red team exercises. Only once the improvement has been evidenced, should it be reflected in the reassessment and compared against the original modelling.
The example above illustrates this distinction. Some capabilities have improved in line with expectations, while others have improved by less than originally forecast. The investment has still materially reduced risk exposure, although it has not delivered the full reduction that was modelled.
Interestingly, Jack Jones, creator of FAIR, recently made a similar point in What I’d Change About FAIR, Twenty Years Later. One of the changes he proposes is adding a feedback loop, recognising that risk analysis produces forecasts but rarely goes back to see how they turned out.
The same principle applies here. If quantification was used to model the expected reduction in risk before the investment was made, reassessment gives you an opportunity to test that expectation against what actually happened.
It’s also important to think about when the reassessment takes place. Some investments will take longer to deploy and operationalise than others, so there isn’t a set cadence that works for every investment. Reassessment could follow key milestones in the improvement programme or align with existing risk reporting cycles, updating the baseline capability assessment only once there is enough evidence to support it.
There is also a question of who owns the follow-up. The people responsible for delivering the investment, operating the control and managing the risk aren’t always the same. They may also be looking for different evidence of whether the investment has worked. The investment owner may be focused on whether the improvement has been delivered, the control owner on whether it is operating effectively, and risk committees on whether there has been a material change in risk exposure. Bringing those views together is important when assessing whether the investment delivered the benefit originally modelled.
If exposure hasn’t reduced as expected, that is useful information too. Implementation may be behind plan, the control may not have had the effect originally assumed, or the risk itself may have changed. Tracking risk reduction against what was originally modelled gives decision-makers an understanding of whether the investment is having the intended impact and decide what to do next. That could mean continuing as planned, making adjustments or changing where future investment is focused.
So, when assessing whether a cyber investment has worked, I would keep three questions in mind:
Risk quantification gives organisations an expectation of how an investment should reduce risk. Its real value comes from returning to that expectation, measuring whether the reduction was achieved and using what you learn to decide what happens next.
