Hospital quality indicators are intended to provide public information about differences in the quality of care across hospitals. The measures — such as 30-day mortality rates and 30-day hospitalization costs for admitted patients — are adjusted for the characteristics of each hospital’s patient population. In principle, these risk-adjusted indicators are not affected if a hospital treats especially unhealthy patients. If the risk adjustment is inadequate, however, the quality measures could confound patient health status with hospital quality. 

In Are Hospital Quality Indicators Causal? (NBER Working Paper 31789), Amitabh Chandra, Maurice Dalton, and Douglas O. Staiger assess risk-adjusted hospital quality measures. Specifically, they ask how well these measures predict changes in outcomes for Medicare beneficiaries when a local hospital closure induces patients to switch to other hospitals. If hospital quality indicators capture meaningful variation in quality, the effect of a closure should depend on the difference between the quality measures of the closing hospital and of the remaining hospitals in the area.

The analysis focuses on outcomes among fee-for-service patients who were admitted to a hospital due to heart attacks, hip fractures, pneumonia, congestive heart failure, or strokes between 1992 and 2015. It evaluates four hospital quality measures that are based on different outcomes: 30-day mortality, 30-day readmissions, 30-day inpatient hospital costs, and length of stay.

The figure shows that, when a hospital closes, mortality changes in the direction that risk-adjusted 30-day mortality rates predict. When quality indicators predict that a hospital closure will worsen health outcomes, mortality rises at the time of the closure. In contrast, when quality measures predict that a hospital closure will improve health outcomes, mortality falls. Similar patterns emerge for all four measures of hospital quality. 

How well do the quality indicators predict the magnitude of the changes in outcomes? Risk-adjusted quality measures based on health outcomes perform better than those that are based on resource utilization. The risk-adjusted indicators overestimate the impacts of closures on mortality and readmissions by about 7 percent and overstate effects on costs and length of stay by around 40 percent.

Risk-adjusted quality indicators generally perform better than their non-risk-adjusted counterparts. While unadjusted quality measures accurately predict whether outcomes will improve or worsen after a closure, they overestimate changes by more than the risk-adjusted indicators. They overstate effects on mortality by 40 percent, readmissions by 30 percent, and costs and length of stay by 50 percent.

The researchers highlight that closing higher-quality hospitals may harm patients who would otherwise have used them, while closing lower-quality hospitals may help such patients. 

— Robin McKnight

The researchers acknowledge financial support from the National Institutes of Health through a Pilot grant and a P01 grant to Dartmouth and the NBER (P30AG012810 and P01AG019783).