A large number of factors contribute to reader discordance or variability in Clinical Imaging Trials. These factors include but aren’t limited to:
- The experts’ training and clinical experience
- The experts’ therapeutic area of expertise
- Reader workload
- Study indication including tumor appearance
- The overall complexity of the tumor response evaluation criteria
- Image quality/display
- Overall number of images for review
When using multiple readers in an imaging clinical trial, one of the most important assumptions is that the readers are exchangeable. This implies that each reader, on their own, would provide the same statistical result when evaluating the same subjects. The concept of exchangeability is related to, but not the same as, agreement. Disagreement allows us to exchange readers and not disrupt the statistical results without requiring readers to produce identical results.
ERT reviews readers’ experience (i.e. CVs) and provides training on the protocol and display systems to help achieve exchangeability. We also monitor performance throughout the trial to ensure exchangeability. However, unforeseen circumstances can and do affect the availability of readers throughout a long trial, and it’s common to have to replace readers during a trial. In those cases, it’s important that any disruption of trial operations and data integrity is minimized while the new reader is integrated. In other words, new readers must be as exchangeable as possible with existing readers. To comply with FDA guidelines, sponsors should proactively define these processes in the trial’s related imaging documents.
To achieve these objectives, we recommend the following:
- Training: The new reader must receive training similar to what the other readers received pre-study. While it’s not necessary that all of the readers be present, it’s also not a bad idea. The new reader training should use completed cases instead of, or in addition to, the early training cases provided to the other readers during their pre-study training. This ensures that any differences between the training images and the on-study imaging population are resolved. In ICH E9(R1) guideline terms, it ensures that the estimand is being evaluated 
- Testing: The new reader should be tested on completed reads to determine agreement with the other readers and resolve any systematic differences that could impose a bias on the subsequent reads. This will also help to get the new reader up to speed and ensure exchangeability. Note that this does NOT mean that the new reader must agree completely with the other reader(s) but that disagreement appears to be random at the same rate as the other readers and due more to the ambiguities of the images than any bias by the reader.
- Reader monitoring: The evaluations by the new reader should be monitored for the first 5-10 reads to verify exchangeability.
Occasionally, though not often, reader performance monitoring will identify that a reader will be demonstrably different than the other readers and remedial attempts to correct that reader’s performance do not work. On those thankfully rare occasions, that reader is jeopardizing the integrity of the trial and must be replaced. Aside from the obvious awkwardness of removing an expert reader from a trial for performance reasons, it’s important to have a pre-specified methodology for re-reading any affected subjects.
By following these suggestions the clinical team can determine that the new reader(s), to the highest extent possible, be exchangeable with the other readers.