KPIs. Key Performance Indicators. HR departments around the world require that every department and individual have KPIs (or OKRs) to assess how well employees are contributing to the overall organization based on their specific areas of expertise and output. Put more simply: KPIs are meant to show leadership in an organization that people are contributing what they are supposed to be. However, when we look at the KPIs common to SOC (Security Operations Center) Teams, we find that many of the KPIs are not true indicators of how a SOC Team performs, both as a team or as individual contributors. In the coming series of posts written by Cyberbit SOC Experts, we will dive deep into SOC Team KPIs, how they are routinely cheated, and how SOC KPIs should transform to ensure true performance reporting, allowing leaders to optimize and make their SOC Team more effective.
Common SOC Team KPIs
The focus of a KPI should be around optimization of a team to ensure they are performing to their maximum potential. Thus, when looking at the KPIs common to SOC Teams according to the Common and Best Practices for Security Operations Centers: 2019 SOC Survey, the theme should hopefully be around how well a team performs, viewing the SOCs contribution to an organization, measuring the SOCs impact and performance with an eye towards further improvement of SOC activities.
As you can see from the chart above, the most common metrics rely on the quantity of incidents handled and MTTR. While these are very nice metrics to have, they do not give a true picture of how the SOC is performing. Incident quantity is a lot less impressive if the incidents being handled are minor and do not require much work from higher tier analyst to mitigate the incident.
The same logic applies to MTTR as a metric. Mean Time to Response metrics, such as “Time from detection to containment to eradication” do not account for incident complexity. The more complex an attack, the longer it will take between steps leading to a longer response time. Thus, a team that handles only minor incidents will appear to be a vastly superior SOC while the SOC team that handles fewer, more complex, incidents will appear to be struggling, despite facing far superior attackers.
As we continue in this series, we will begin to suggest new SOC metrics, more aligned to organizational and individual performance. The goal is to focus the SOC as a part of the larger organization, viewing their impact on definable and measurable outcomes that relate directly to their performance in everyday activities. Having information that is definable, measurable, and relatable, and actionable will allow SOC leaders to build new processes, find new tools, and add more training that is directly related to their SOC activities on a day to day basis.
Correlating Data for KPIs
Most companies rely on their SIEM (security information and event management) to merge event data with other ancillary security data to measure SOC Team activities. However, as per the aforementioned report, many security professionals are unhappy with using the SIEM alone. The SIEM is viewed as a technical tool from which much of the data for metrics can be acquired or derived.
Acquiring a tool like Splunk that aggregates information across the teams affected by security incidents, including Finance, Marketing, Data, IT, and many others will allow you to see the ripple effect that the SOC team has when mitigating incidents. This data can also be used to optimize performance and incident handling processes to ensure that the incidents with the largest organizational impact are being handled by the appropriate resources inside the SOC.
Are you measuring Soft Skills?
The most unpredictable element in any SOC is also the element that is the least computer controlled: the human element. SOC Team members are subject to all the unpredictable behaviors that humans are. Elements such as team communication skills, working dynamics, leadership skills, analytical skills, ability to perform under pressure, and many other elements have a profound impact on both individual and team performance.
Analysts who are more irritable may have trouble working with others leading to less team communication. Lower levels of team communication can obstruct the incident response cycle giving an attacker more time on your network to encrypt, disrupt, or steal data. The longer an attacker has on your network the more likely they are to find sensitive information that has more value leading to higher incident costs and increased work disruption in other departments.
Conversely, an analyst who shows strong leadership, teamwork, creativity, and communication skills can take your entire SOC Team to the next level of performance. Inspiring people within a SOC will not only improve individual incident response processes but will foster an environment of collaboration that will ensure skill sharing amongst the team and information sharing during critical moments of incident response, having a positive effect across the entire organization. Understanding the larger impact of incident response cycles and how an individual contribution can have a positive effect is key to ensuring the success of your SOC Team.
To ensure a diverse set of soft skills, LinkedIn looks to hire SOC team members from a variety of backgrounds to introduce diverse thinking into their SOC according to Dark Reading. “One of the biggest things I consider when hiring talent is gathering a diversity of perspectives,” wrote Geoff Belknap, CISO of LinkedIn. “Many different types of people interact daily with the products we’re working to secure, which means our team needs to be able to understand and consider needs, work habits, and challenges from several points of view.”
What are the right KPIs for a SOC Team?
KPIs should reflect the larger organizations needs as well as the individual contribution of the SOC and its members. While this is not an easy task, it is imperative to the long term success of your SOC Team, providing you with as many indicators of success or improvement as possible to allow you to make informed decisions for the SOC using the entire picture, not just a snapshot.
Organizational and SOC Team KPIs
Breaking out incident response vectors by organizational impact will give you the full picture of your team’s effect on other departments. For example, attacks focused on theft of private data will have a large external impact on your organization, drawing in marketing, finance, legal, and many other departments. However, attacks focused on business disruption like ransomware, pulls in fewer resources and generally has a lower financial impact on the business. Create your own internal categorization based on business outcomes that are specific to your needs and begin to put incidents into those categories, measuring response times by incident complexity. The chart we propose looks something like this:
The focus here is to show cost vectors directly in relation to attack complexity and type. This will allow you to see where your team is succeeding and where your team is struggling on an organizational level. Costs should include manpower costs as well as business costs from other teams as the longer an attack takes, the higher costs will be. Ancillary costs are costs that come as a direct outcome of the attack that are focused externally such as lawsuit settlements. Manpower plus ancillary costs should be equal to the total cost to the business giving you a look into the breakdown of costs by attack type.
Individual SOC KPIs
Individual KPIs for SOC Team members should provide insight into the individuals performance and impact on the team. Focus on the tasks that each contributing team member performs to see when or where they are a bottleneck or where they are succeeding. This will not only inform you of how an individual is performing, but also show where you need to focus their training efforts to help them reach peak performance. KPIs for individuals should also include soft skills including a scale for the following individual non-task related KPIs:
These skills will let you know who is a leader, who is a follower (each have their place on a team!), and how likely they are to understand the needs of their teammates, ultimately leading to a better team dynamic and better SOC team performance.
Remember, the goal of a KPI is not to get your bonus but to get a realistic picture of where your team stands. Don’t make improvement metrics too easy to reach or your team will not be inspired to improve. Conversely, don’t make metrics too difficult or your team will be discouraged before they even begin the quarter. The more insight your KPIs provide into the COMPLETE PICTURE, the better they will be for you in both the short and the long term. Failing to reach a KPI is not a death sentence to good executives; it provides a roadmap for a path forward, revealing pitfalls, skills gaps on the team, and ultimately where you will focus your training in the coming quarter.