Improved Dorfman-Berbaum-Metz Receiver Operating Characteristic Methods for Diagnostic Radiology

This work addresses a fundamental methodological problem in diagnostic radiology research: how to evaluate the diagnostic performance of imaging modalities. Receiver operating characteristic (ROC) analysis is an accepted procedure for measuring how well imaging modalities allow physicians to detect disease. Traditional statistical comparison of different imaging modalities only analyzes ROC measures of a single radiologist from each modality. Dorfman, Berbaum, and Metz (1992) developed a statistical methodology that overcomes this limitation and provides a comprehensive framework within which to analyze ROC studies with multiple readers, taking into account bother reader-sample and case-sample variation. The Dorman-Berbaum-Metz (DMN) methodology allows greater statistical power to detect clinically meaningful differences in a diagnostic performance than single-radiologist approaches. Its validity has been tested extensively using computer simulations and has wide acceptance in diagnostic radiology experiments. Because DBM methodolodgy relies on ROC analysis for its dependent measures, the robustness and accuracy of the ROC analysis affect the statistical precision and power of DBM methodology. DBM analysis of some experiments fails because of flaws in traditional ROC measures. Since the DBM methodology was developed, there have been fundamental advances in ROC analysis, which provide more interpretable measures. DBM could be more robust and statistically powerful if it took advantage of these advances. To accomplish this, we propose building an integrated public-domain software resource to bring together methodological advantages previously only available in separate programs.

Our specific aims are:

• Develop a new modular DBM software architecture that is freely accessible by other software and allows greater flexibility in ROC analysis Develop “proper” ROC analysis modules that provide more interpretable ROC results and study the resulting DBM statistical power;
• Develop modules for types of ROC analysis that take into account location of reported disease and multiple reports of disease, features that have been shown to improve statistical power; Develop modules and procedures to merge DBM methodology with standard statistical software to answer more complex experimental questions;
• Use DBM re-analysis of published data to provide estimates of minimum reader and case sample size combinations needed to detect various performance differences with power for various imaging areas.

Completing these aims will enhance the effectiveness of the primary method used to assess the diagnostic performance of new imaging systems. This will lead to better radiology research, which will raise the quality and lower the cost of health care.