This is Part 2 of a series on endpoint adjudication in clinical trials. Click here to review the first installment, “Clinical Trial Imaging – Multiple Endpoint and Adjudication Strategies.”
In previous blogs we explained the concept of the clinical site assessment bias (over or under estimates of effect) together with the issue of site versus central reviewer discordance as observed across a broad range of therapeutic indications. These issues serve as the basis for the Food and Drug Administration (FDA) as well as the European Medicine Agency’s (EMA) recommendation to perform safety or efficacy endpoint adjudication in multicenter registration trials especially for endpoints which involve subjective assessments1. “The importance of such committees has been outlined in several studies showing that the classification of events changed in about 20%-30% of cases after assessment by an adjudication committee.”2 We have found lower discordance when significant effort is spent aligning or standardizing the assessments performed at the clinical site and central lab.
To provide more accurate data and reduce assessor bias, centralized adjudication is performed in a blinded fashion and may occur in a variety of shapes and forms. Adjudication may be as simple as using one additional independent assessor adjudicating the clinical site assessors, whose assessments may be performed in an unblinded manner, and or may be as complex as convening adjudication committees made up of multiple therapeutic clinical area specialists.
Perhaps the most well-known example was the everolimus study for kidney cancer decision by the FDA to revise the protocol at the interim analysis to provide for a central adjudicator to resolve site-central discordance, primarily due to the FDA conclusion that it was highly likely that site bias existed3.
Special care should be taken when group reviews are utilized to avoid the loudest voice influence which raises the agreement (coerces) the agreement of the individual members. While typically only two readers and a single adjudicator are used in multiple reader studies, increasing the number of readers in a pool of readers has the effect of reducing the chance of bias due to a single reader. It is a common misconception that adding readers will increase variability and trained and experienced readers could actually reduce overall variability, again, caused by a single reader.
Additionally, in highly variable disease assessments, hepatocellular cancer (HCC) comes to mind, more than 2 readers may significantly increase the accuracy needed to conserve study power. However, more than 2 readers per subject requires a more complex adjudication paradigm and places higher demands on operational system demands, may have longer timelines and higher overall costs.
Regulatory and Statistical Requirements
The regulatory (and statistical) requirement when using multiple readers is that there must be a predefined method to resolve discordance between two assessments. The most common method is a single adjudicator who renders judgment between the two primary readers. This method is carried down from resolving diagnoses discordance where the assessment is binary – Disease/No Disease. However, other methods for other endpoints may be just as good if not better.
The study protocol and imaging charter documents should define:
- The adjudication model which is based on the rate of expected disagreement amongst experts and may include some variation of the following (listing is not exhaustive): site reader and one central independent reader; multiple individual central readers; panel or group of readers reading independently or in a group setting,
- Careful definition of the study endpoint(s) to be adjudicated
- The selected cases for adjudication (i.e. all cases or a predefined subset of cases for example those with a suspected safety event or those cases considered to have progressive disease) and the clinical information (i.e., blinded or un blinded) that will be shared with the adjudicators should be justified (e.g., clinical history, clinical laboratory, biomarkers, imaging, pathology or death reports, etc.)
- Composition and expertise of the adjudicator(s)
- Software and systems to be used including training
- Adjudication work flow/process including decision rules and required meetings
- Data handling, quality processes and data reporting
Short of oncology trials, where regulatory bodies recommend using two independent readers with one adjudicator (referred to as the 2+1 reader model which could be expanded to using a larger groups of readers using an odd number of readers), there is no optimal adjudication strategy that works in all therapeutic indications.
- Dechartres A. et al. Inadequate planning and reporting of adjudication committees in clinical trials: Recommendation proposal. Journal of Clinical Epidemiology, 62 (2009), 695-702
- FDA Oncologic Drugs Advisory Committee Meeting April 12, 2011