The Impact of Expanding Reader Pools in Clinical Trial Imaging

Joseph Pierro and David Raunig | |

We previously presented a number of different image read designs that may be used in clinical trial assessments of safety and efficacy data, and that decisions are often made a priori based on the appropriateness of the design to meet trial objectives.

Here, we discuss the option of selecting a fixed versus rolling reader pool for the most common reader paradigm deployed in trials. We’re referring to a paired reader model with disagreements adjudicated by a third reader, commonly referred to as the 2+1 reader model.

In this model, a pair of readers [R1 and R2] will read all of the study images and when their assessments are discordant these cases are reviewed by the adjudicator [R3]. The adjudicator uses either a forced adjudication [adjudicator selects the assessment they agree with the most] or open adjudication [R3 provides their own final assessment resulting in 3 sets of results in the database].

As trials become larger (e.g., >500 subjects) using only 3 readers places excessive demands on the reader’s availability to review study images, which jeopardizes timelines.  One robust and valid solution to this hurdle is to expand the pool of independent readers.  This can be done by using multiple fixed pairs of readers, e.g., R1/R2; R3/R4,…; with a fixed adjudicator R(N) where each of the reader pairs read a fixed proportion of the images and discordant results are reviewed by a single adjudicator.  Since the adjudicator is common to both reader pairs, evaluation bias may be more easily detected than in other reader models. However, there is an increased risk of unwanted adjudicator influence on the endpoint(s).

Another time-saving option is to deploy an expanded rolling reader pool in which cases are assigned based on availability. In this design, the first two available readers make up the primary read team.  The adjudicator can be fixed or can be assigned from the remaining readers in the pool.  Operationally this provides flexibility, balances out the influence of any single reader and allows the adjudication to be balanced between all of the expert readers when assigned from the remaining readers in the pool.

This design should result in a balanced assignment of cases to readers, but not always.  So, it would be wise to monitor the number of cases assigned to each reader in order to identify imbalances that could affect the quality of the evaluations and to ensure a manageable number of fixed reader pair combinations for reader performance analysis.

In medium-size trials, the operational team will need to more closely monitor reader assignments and ideally pre-define a minimum number of cases that each reader or reader pair should assess. This is very important when monitoring reader performance and deciding if any of the readers is consistently associated with higher adjudication rates.  The advantage of this design is that readers will “average” each other out and, importantly, a biased adjudicator is not overly influential.

In each of these enhanced reader pool designs, there is an assumption that the readers are well- and equally trained and that reader performance monitoring validates this assumption.  There is a common belief that more readers necessarily mean more variability.  While a single reader means no variability, more readers do not necessarily give rise to more discordance and, conceivably, actually reduce disagreement.

As stated earlier, more readers help reduce the impact of any single aberrant reader, allow a more efficient review of reader performance and, if the assumption of well- and equally trained is valid, there is no loss of power in the endpoint analysis.  In statistical terms, this is known as exchangeability and diligently works to meet the goals of the study – to provide evidence of safety and efficacy which leads to the delivery of new therapies for patients.

Joseph Pierro is the Medical Director of Imaging at ERT and David Raunig is the Senior Principal Imaging Statistician at ERT.