Back around 2014 I started helping with the Clinical Laboratory Standards Institute (CLSI) with a guidance document, Evaluation of Qualitative, Binary Output Examination Performance (EP-12).
In my world of clinical diagnostic development, we call these “assays.” Most everyone else knows them as the “tests” or “blood work” that your doctor wants to see before your physical or when you have a health problem. They are also known as “in-vitro diagnostics” or IVDs, and these particular IVDs give a “binary” result that is, “yes or no” or “positive or negative.”
A lot of testing is done using this approach since in certain situations it’s not important how much of the “analyte” (the thing we are looking for) is found, but whether it is above a certain concentration (called a cutoff or a decision threshold). For example, if the cutoff was 2 for a particular assay, it would not matter if we found 5 or 50 or 75, just that it was at 2 or above, then the reported result would be “yes” or “positive.”
So you can imagine how important these things are. In 2008 a second version of this document was published, and I became lucky enough to get to know one of the authors of that version. When I griped about it being written for statisticians and not development scientists, she made me do something about that, and I signed up to be part of the working group to fix it.
I learned a lot from working on that team and we found out just how big this class of IVDs was throughout the world. This version (ED3) of the CLSI document goes beyond what the previous version explained and I’m very thankful that CLSI agreed to let us present important information for the development of these assays as well as the evaluation of their performance.
We tried to make the ED3 version as comprehensive as possible so that developers/designers (in either clinical labs or IVD industry) would find useful information as well as helping clinical labs by presenting a framework on how to make sure any new examination/assay works for them once the product is released.
It’s important to note that for examinations/assays with a cutoff, evaluation of the performance of the assay is about assessing performance metrics at analyte levels around or related to that cutoff. The information on the concepts of C5 and C95 for these examinations/assays has been expanded to better help developers understand how variability and or bias relates to the overall “accuracy” of the product, and I hope will give everyone the same insight on how this terminology can be used going forward in discussing performance.
Also in this version, we tried to make it more clear that if the examination/assay uses what is called an “internal continuous response” (ICR), that is, a signal or concentration where the cutoff resides to make the qualitative binary call, then data from that ICR should be used to determine quantitatively (e.g. a precision profile approach) the C5 and C95.
These documents are written for the clinical labs and the developers; however, they also describe best practice or good science and as a result are also considered by regulators.
A lot of hard working folks contributed to this document and I’m happy to congratulate each of them for the effort. I know this document will not have everything that everyone wanted or thought would be in it as this is the nature of consensus documents. While this version of EP12 may not change the world, I do believe that it will go a long way to helping us do our best in the development and evaluation of these devices and I’m grateful to have been part of that endeavor.
Perhaps EP-12 will not set the world alight; however, good science enhances patient safety and that’s something worthwhile.