Decision curve analysis
| Net benefit | |
|---|---|
|
|
|
|
|
|
| Net benefit is calculated as a weighted combination of true and false positives, where is the threshold probability, true and false positives are count variables and N is the total number of observations. |
Decision curve analysis evaluates a predictor for an event as a probability threshold is varied, typically by showing a graphical plot of net benefit against threshold probability. By convention, the default strategies of assuming that all or no observations are positive are also plotted.
Decision curve analysis is distinguished from other statistical methods like receiver operating characteristic (ROC) curves by the ability to assess the clinical value of a predictor. Applying decision curve analysis can determine whether using a predictor to make clinical decisions like performing biopsy will provide benefit over alternative decision criteria, given a specified threshold probability.
Threshold probability is defined as the minimum probability of an event at which a decision-maker would take a given action, for instance, the probability of cancer at which a doctor would order a biopsy. A lower threshold probability implies a greater concern about the event (e.g. a patient worried about cancer), while a higher threshold implies greater concern about the action to be taken (e.g. a patient averse to the biopsy procedure). Net benefit is a weighted combination of true and false positives, where the weight is derived from the threshold probability. The predictor could be a binary classifier, or a percentage risk from a prediction model, in which case a positive classification is defined by whether predicted probability is at least as great as the threshold probability.
Theory
[edit]The threshold probability compares the relative harm of unnecessary treatment (false positives) to the benefit of indicated treatment (true positives). The use of threshold probability to weight true and false positives derives from decision theory, in which the expected value of a decision can be calculated from the utilities and probabilities associated with decision outcomes. In the case of predicting an event, there are four possible outcomes: true positive, true negative, false positive and false negative. This means that to conduct a decision analysis, the analyst must specify four different utilities, which is often challenging. In decision curve analysis, the strategy of considering all observations as negative is defined as having a value of zero. This means that only true positives (event identified and appropriately managed) and false positives (unnecessary action) are considered.[1] Furthermore, it is easily shown that the ratio of the utility of a true positive vs. the utility of avoiding a false positive is the odds at the threshold probability.[2] For instance, a doctor whose threshold probability to order a biopsy for cancer is 10% believes that the utility of finding cancer early is 9 times greater than that of avoiding the harm of unnecessary biopsy. Similarly to the calculation of expected value, weighting false positive outcomes by the threshold probability yields an estimate of net benefit that incorporates decision consequences and preferences.[3]
Interpretation
[edit]
A decision curve analysis graph is drawn by plotting threshold probability on the horizontal axis and net benefit on the vertical axis, illustrating the trade-offs between benefit (true positives) and harm (false positives) as the threshold probability (preference) is varied across a range of reasonable threshold probabilities.[2]
The calculation of net benefit from true positives and false positives is analogous to profit. Consider a wine importer who pays €1m to buy wine in France and sells it for $1.5m in the United States. To calculate the profit, an exchange rate between euros and dollars must be used to put cost and revenue on the same scale. Similarly, the costs (false positives) and revenue (true positives) of the predictor must be compared on the same scale to calculate net benefit. The factor expresses the relative harms and benefits of the different clinical consequences of a decision and is therefore used as the exchange rate in net benefit.
The figure gives a hypothetical example of biopsy for cancer. Given the relative benefits and harms of cancer early detection and avoidable biopsy, we would consider it unreasonable to opt for a biopsy if the risk of cancer was less than 5% or, alternatively, to refuse biopsy if given a risk of more than 25%. Hence the best strategy is that with the highest net benefit across the range of threshold probabilities between 5 – 25%, in this case, model A. If no strategy has highest net benefit across the full range, that is, if the decision curves cross, then the decision curve analysis is equivocal.[4]
The default strategies of assuming all or no observations are positive are often interpreted as “Treat all” (or “Intervention for all”) and “Treat none” (or “Intervention for none”) respectively. The curve for “Treat none” is fixed at a net benefit of 0. The curve for “Treat all” crosses the both axes at the event prevalence.[2]
Net benefit on the vertical axis is expressed in units of true positives per person.[4] For instance, a difference in net benefit of 0.025 at a given threshold probability between two predictors of cancer, Model A and Model B, could be interpreted as “using Model A instead of Model B to order biopsies increases the number of cancers detected by 25 per 1000 patients, without changing the number of unnecessary biopsies.”
Further reading
[edit]Additional resources and a complete tutorial for decision curve analysis are available at decisioncurveanalysis.org.
References
[edit]- ^ Baker, Stuart G.; Cook, Nancy R.; Vickers, Andrew; Kramer, Barnett S. (2009-10-01). "Using relative utility curves to evaluate risk prediction". Journal of the Royal Statistical Society. Series A (Statistics in Society). 172 (4): 729–748. doi:10.1111/j.1467-985X.2009.00592.x. ISSN 0964-1998. PMC 2804257. PMID 20069131.
- ^ a b c Vickers, Andrew J.; Elkin, Elena B. (November 2006). "Decision curve analysis: a novel method for evaluating prediction models". Medical Decision Making. 26 (6): 565–574. doi:10.1177/0272989X06295361. ISSN 0272-989X. PMC 2577036. PMID 17099194.
- ^ Van Calster, Ben; Wynants, Laure; Verbeek, Jan F.M.; Verbakel, Jan Y.; Christodoulou, Evangelia; Vickers, Andrew J.; Roobol, Monique J.; Steyerberg, Ewout W. (December 2018). "Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators". European Urology. 74 (6): 796–804. doi:10.1016/j.eururo.2018.08.038. PMC 6261531. PMID 30241973.
- ^ a b Vickers, Andrew J.; van Calster, Ben; Steyerberg, Ewout W. (2019-10-04). "A simple, step-by-step guide to interpreting decision curve analysis". Diagnostic and Prognostic Research. 3: 18. doi:10.1186/s41512-019-0064-7. ISSN 2397-7523. PMC 6777022. PMID 31592444.