Defining and Tracking Security Metrics: A Guide for Security Product Managers
5/8/20244 min read
Introduction: The Challenge of Defining Metrics in Security Products
Security product managers (builders) are tasked with the dual responsibility of ensuring the security of their products while also creating solutions that enable business operations. However, one of the most significant challenges they face is defining meaningful and measurable security metrics. In traditional software development, metrics are typically centered around product adoption, user engagement, and revenue performance. Security products, however, cannot be measured by these same standards. The goal of security solutions is not simply adoption, but their ability to reduce risk, improve compliance, and enhance the user experience. Furthermore, security often plays a supportive role rather than being a core feature, which makes it difficult to quantify success and measure impact.
For example, measuring the effectiveness of security training programs is a recurring issue. While metrics like training module completion rates are easy to track, quantifying how well developers apply security best practices in their real-world workflows is more complex. Moreover, security tools often operate behind the scenes, meaning their benefits are not always immediately visible to users or stakeholders. This complexity makes it vital for security product managers to understand the right types of metrics to track in order to accurately measure their tools' effectiveness.
Some common challenges faced by security product managers include:
Aligning Metrics with Business Objectives: Security solutions should support broader business goals, rather than acting as a bottleneck to operations or innovation.
Balancing Security and Usability: Security tools that are overly restrictive or intrusive can hinder development speed, leading to friction between security teams and developers.
Defining Measurable KPIs: Because the impact of security solutions is often preventive, it can be difficult to demonstrate a direct, measurable ROI from security investments.
Addressing False Positives and Negatives: High false positive rates can lead to frustration for developers, while false negatives can create hidden risks, making it harder to demonstrate the efficacy of security tools.
These challenges underscore the need for a strategic approach to defining and tracking security metrics, which can provide valuable insights into the performance and impact of security products.
How to Think About Security Metrics and KPIs
Security product managers can overcome these challenges by developing a structured approach to metrics. This involves:
Understanding Stakeholder Needs: Security product managers must engage with various stakeholders — including developers, security engineers, executives, and compliance officers — to understand what success looks like from each group's perspective. Different stakeholders may prioritize different aspects of security, and understanding these perspectives is critical for developing meaningful metrics.
Defining Leading and Lagging Indicators: Metrics can be categorized into two types:
Leading Indicators: These are predictive metrics that provide early warning signs or signals of potential issues. For instance, tracking MFA adoption rates, the usage frequency of security scanning tools (like DAST or SAST), or security training completion rates can serve as leading indicators of security maturity.
Lagging Indicators: These metrics measure past performance and outcomes. For example, the number of security incidents that occurred within a specific time frame, or the time it took to remediate vulnerabilities, are examples of lagging indicators that can provide insights into how well security measures have performed in the past.
Establishing a Baseline and Improving Over Time: Security metrics should not only track performance but should also help guide continuous improvement. Establishing baseline metrics allows security product managers to measure progress over time and understand where changes in strategy or execution are needed. Regular tracking and trend analysis can lead to data-driven improvements in both security practices and the development lifecycle.
Security Metrics That Matter
For security product managers, focusing on the right metrics is essential. These metrics should address both the adoption of security tools and their ability to reduce risk, improve developer experience, and enhance security resilience. Below are three major categories of security metrics that can help drive meaningful insights:
Product Utilization: This category measures how widely and effectively a security product is being adopted across an organization. It is one of the most crucial leading indicators of security maturity, as higher adoption rates generally signal greater organizational focus on security.
For example:
MFA Adoption Rate: This measures the percentage of users or applications that have implemented multi-factor authentication. A higher rate of MFA adoption indicates that users are taking proactive steps to secure their accounts, reducing the potential for credential-based attacks.
DAST and SAST Scan Platform Utilization: These tools are critical for identifying vulnerabilities during the development process. By measuring how often these scanning tools are used, security product managers can gauge whether developers are actively scanning their code for vulnerabilities.
Coverage by Business Segment or Service Type: This metric helps identify areas where security adoption is strong or lacking. It can highlight which business units or service types are more or less secure, helping security teams target improvements where they are most needed.
Seamless Customer Experience: Security should act as an enabler for developers rather than a blocker. These metrics assess how well security integrates into the development workflow, ensuring that security does not hinder development speed or productivity.
Number of Blocked Releases Due to Security Tools Errors: This metric tracks disruptions caused by false positives or misconfigurations in scanning tools, highlighting areas where the security tools are creating unnecessary friction in the development pipeline.
Mean Time to Resolution (MTTR) for Errors: MTTR measures how quickly security issues are identified and resolved, providing insights into the efficiency of the security team in mitigating risks.
Onboarding SLA: This measures how quickly new developers are able to start effectively using security tools within their workflows. A longer onboarding process can lead to delays in adoption and lower security maturity.
Efficacy of the Product or Service: Ultimately, the effectiveness of security products is measured by their ability to reduce risk and improve resilience. These metrics focus on whether security controls are functioning as intended to protect the organization.
Number of Blocked Releases Due to Critical Vulnerabilities: This metric indicates how well security tools are preventing risky deployments by blocking releases that contain high-severity vulnerabilities.
Vulnerability Remediation Rate: The faster vulnerabilities are addressed and fixed after detection, the more resilient the organization becomes. This metric tracks the time it takes to remediate vulnerabilities once they have been identified.
False Positive Rate from Scanning Tools: A high rate of false positives can frustrate developers and reduce the perceived value of security tools. By ensuring that scanning tools have low false positive rates, security product managers can improve both the effectiveness and developer experience of the security tools.
Conclusion:
The key takeaway for security product managers is to approach metrics as a means to continuously improve both security practices and the tools that support them. By focusing on metrics that are not just leading indicators of adoption but also provide insights into real-world outcomes such as reduced vulnerabilities, faster remediation, and enhanced developer satisfaction, security product managers can ensure that their products are contributing to the organization's broader goals of reducing risk and ensuring compliance.