Is empirical medical research doomed? Generalizability of predictions and treatment effect estimates
Ewout Steyerberg, "Assessment of heterogeneity in clinical prediction models"
Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived in the context of an individual patient data meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. Illustration is in a case study with patients suffering from traumatic brain injury, where we aim to predict six-month mortality based on individual patient data using meta-analytic techniques (15 studies, $n=11,022$ patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.
Erik van Zwet, "Using the Cochrane database to interpret a typical RCT"
Randomized Controlled Trials (RCTs) are key to estimation of treatment effects. We can capture the essence of an RCT as a set of three numbers: (beta,b,s). Here beta is the true effect, and b is a normally distributed, unbiased estimate of beta with standard error s. We have collected 45,955 pairs (b,s) from the Cochrane Database of Systematic Reviews (CDSR). We have used these pairs to estimate the joint distribution of the z-value (z = b/s) and the signal-to-noise ratio (SNR = beta/s). This joint distribution is very useful because many important statistical concepts such as power, coverage and relative bias depend on (beta,b,s) only through z and SNR. We can also use the distribution as prior information to interpret any trial that may be considered to be "exchangeable" with the CDSR.