An Argument for Multiple Blinded Readers

Joseph Pierro MD and David Raunig PhD |

Blinded independent central review (BICR) models have been discussed in several past blog posts. This post will present background on the advantage of using multiple readers over the use of single readers from multiple clinical sites and draws on the work of many scientists who have published on this very point.

Gregory Sica’s paper entitled “Bias in Research Studies, Radiology” explains that there are more than 30 types of biases and errors that can affect investigations and should be considered in research study designs.1 Petter Eldevik’s paper discusses observer bias when computed tomography radiologists have access to clinical information and the impact on diagnostic accuracy, false positives and false negatives reporting.2 In fact, numerous research publications dating from 1947 to the present describe in detail that radiologists make errors in medical imaging assessments at about the same rate as medical doctors in all other disciplines.

It is, perhaps, primarily for this reason that it is a nearly universal practice for doctors, when viewing the results of a patient’s image scans, to have two set of eyes on the assessment.  Though Amit’s paper reported “in general discordance between the local evaluation (LE) and BICR can be a function of measurement error,” the rates of disagreement remain constant over the decades, regardless of any advances in imaging. 3 Therefore, to mimic the use of two opinions in the clinic, and to ensure a more accurate final assessed endpoint, two readers are used in pivotal trials.

Colin Miller explains that the use of multiple reviewers during BICR is based on the application of the basic principles of Good Clinical Practice (GCP), Good Research Clinical Research Practice (GCRP) and methodologies defined by the Food and Drug Administration (FDA) and European Medicines Agency (EMA). He goes on to say that “the overriding philosophy is that all data are reviewed by two pairs of eyes, e.g., a double data entry or source data verification.”4 Our post, “Clinical Trial Imaging: A Focus on Reader Agreement,” takes a closer look at the advantages of BICR over LE for use in clinical trials with centralized review and multiple readers, and considerations when selecting read paradigms and the number of independent reviewers.

Robert Ford in his BICR review paper reports that “independent central review (ICR) is advocated by regulatory authorities as a means of independent verification of clinical trial end-points dependent on medical imaging” when submitting to regulatory bodies.5 These points are reinforced in guidance documents from the FDA and EMA, e.g., FDA’s industry guidance pertaining to clinical trial endpoints for the approval of cancer and biologics, and Floquet points out that regulatory bodies commonly require BICR analyses to confirm study results as reported by LE.6

Further support for multiple readers is provided by Dr. Ford’s summary that, for industry sponsored Phase 2 and 3 oncology trials, the most common BICR reader paradigm uses two primary reviewers and an adjudicator. In our post, “Understanding the Blinded Read: Clinical Trial Imaging Series, Part 1,” we covered the various reader paradigms, but as a reminder, the 2+1 model uses two primary readers who review all of the subject’s images and the adjudicator is used when the two primary readers’ assessments disagree. The use of an adjudicator to resolve discordance is an essential component of the multiple reader system and it is easily shown that even an average adjudicator can increase the overall accuracy from a nominal 70% for a single reader in a difficult indication to 80%, and an expert adjudicator to 85% accuracy. (Simply add the probabilities.)

The regulatory goals of using data generated by BICR is to reduce potential bias based on site processes.7 The use of a multiple reader pool used to draw two-reader teams increases the generalizability of the results and can reduce the impact of a single, poorly performing reader or the time and cost of replacing a reader for any reason including vacation – a real possibility on long trials. However, this does come with the necessity of increased attention to training and monitoring in order to reduce the chance of poor performers. Some consideration should also be given to using multiple readers on early phase and underpowered studies that could greatly benefit from the increased accuracy.


References

  1. Gregory T. Sica, MD, MPH, Bias in Research Studies, Radiology: Volume 238: Number 3, March 2006
  2. The Effect of Clinical Bias on the Interpretation of Myelography and Spinal Computed Tomography; Petter Eldevik, M.D., Gunnar Dugstad, M.D., William W. Orrison, M.D., Victor M. Haughton, M.D., Radiology 145: 85-89, October 19821
  3. Amit a,*, F. Mannino a, A.M. Stone b, W. Bushnell a, J. Denne c, J. Helterbrand d, H.U. Burger EUROPEANJOURNALOFCANCER 47 (2011) 1772– 177
  4. Colin G. Miller, Joel Krasnow, Lawrence H. Schwartz, Editors, Medical Imaging in Clinical Trials, Springer-Verlag London 2014
  5. Ford, L. et al., Lessons learned from independent central review, European J of Cancer, 45 (2009) 268–274
  6. Floquet A, et al, Progression-free survival by local investigator versus independent central review: Comparative analysis of the AGO-OVAR16 Trial. Gynecologic oncology. 2015 Jan 1;136(1):37-42
  7. (April 12, 2011). FDA Briefing Document Oncologic Drugs Advisory Committee Meeting-ucm250378. Retrieved from Available at: http://www.fda.gov/AdvisoryCommittees/CommitteesMeeting-Materials/Drugs/OncologicDrugsAdvisoryCommittee/ucm250378.pdf
Share