Learning from Failure - What our candidates who don’t pass can teach us about our exams and programs

Survivorship bias is the logical error of focusing on a visible successful subgroup rather than a group as a whole. In the certification industry, there is a strong tendency to focus on high-performing candidates and to infer that if candidates don’t pass exams, it’s because of a lack of preparation or knowledge on their part. While in some cases it is true, there is also an opportunity to learn from these failures, both by taking a closer look at exam data and by gathering feedback directly from this group.

In this presentation, we will share real-world examples of survivorship bias, including its origin and development, and examples of how it can be detrimental to certification program goals, even when entirely unintentional. We will provide eight data-driven case studies on how to learn from failing candidates, and offer tips on building awareness of survivorship bias to your leadership and stakeholders. We will specifically cover how the analysis of failure data improves exam quality, and touch on the potential to identify security breaches.