NIH-Funded Study Raises Questions about Accuracy of Five-Star Quality Ratings

A recent study conducted with financial support from the National Institute for Health Care Management, part of the National Institutes of Health (NIH), and published in the journal Production and Operations Management found that California nursing homes may be artificially inflating their self-reported data to the Centers for Medicare and Medicaid (CMS) for the five-star rating system on Nursing Home Compare, a website used by consumers, payors, and others to evaluate nursing homes. The five-star rating, discussed at length in a previous NIC blog post, is comprised of a health inspection, staffing levels, and quality metrics. The authors of the study argue that many California nursing homes inflate their self-reported quality metrics, which measure the quality of care patients and residents receive in skilled nursing properties. Their analysis considers the financial incentives that drive such false reporting, providing one of the first analyses of how five-star scores impact profitability. The study demonstrates the need for operators and investors to consider multiple data sources and follow quality metrics closely.

According to the study, the quality metric sub score steadily improved for all properties over the period of 2009 to 2013, while the health inspection and staffing sub scores remained stable. The authors argue this difference is not the result of improved quality across the industry and is instead a reflection of nursing homes' improved ability to falsely self-report on the quality metrics sub score. They compared California properties with similar overall five-star scores to complaint data supplied by the California Department of Public Health.

The authors found that often properties with higher overall five-star scores as a result of improved quality metric sub scores within the study period did not have fewer customer complaints as compared to their peers whose scores did not increase over time because of the quality metrics component. In other words, one would expect to see complaints decrease over time for properties with overall scores that improved because of improved quality, but the researchers did not find that to be true in this case. Another finding was that improvements in quality metrics did not lead to improvements in the health inspection component of the overall score, which the authors believe to be a red flag regarding the sincerity of the quality metric reporting.

Using a California data set on profits reported by the Office of Statewide Health Planning and Development, the researchers were also able to demonstrate the importance to profitability of the CMS five-star score. In the five-year study period, they found properties with an overall score of five earned $19.80 in daily per patient profits, while those with a one-star score earned only $9.29. Three-star properties earned $10.79. They conclude that lower-rated properties have a financial incentive to improve their quality metrics, the only component of the five-star score over which they have significant control in how the data is reported. The financial incentive does not necessarily equate to falsely reported quality metrics on the part of properties just to get a bump in profitability, but it led the researchers to further analysis. When the complaint data and financial incentive data were combined, the researchers concluded that financial incentives drive some skilled nursing properties to falsely report quality metrics in an attempt to increase the overall five-star score.

This study is important to both operators and investors for a number of reasons. First, it underscores the importance of considering multiple data sources when evaluating a property's performance. CMS data can be outdated and according to these researchers, easily manipulated and may not always be an accurate tool to evaluate a property's quality of service delivery. Indeed, complaints from the industry about the accuracy and timeliness of CMS data led NIC to develop the Skilled Nursing Data Initiative, which does not address quality, but does speak to occupancy and revenue trends and serves as a valuable resource for the industry. But the Skilled Nursing Data Initiative, like CMS data, is only one resource and the need for more data resources is clear.

Furthermore, the importance of the five-star rating to property financial performance is often discussed in the abstract. This study is unique in that it delivers clear data demonstrating what we sometimes hear—the higher the five-star score, the more profitable the property. This concept influenced NIC to integrate quality metrics from multiple data sources in NIC MAP^©, to be launched in 2018. Stay tuned for details!

Lastly, the study stands as yet another example of how closely the skilled nursing industry is being watched. Indeed, investigations, new regulations, and studies like this one continue to put skilled nursing operators under a magnifying glass. CMS is constantly updating regulations to improve the five-star system, such as the move to payroll-based journaling to record staffing levels launched in 2016 or the addition of new metrics to the quality component of the score. In this era of constant revision to regulations that have real impact on the bottom line, it is more important than ever that the industry focus attention on quality.

Download 3Q17 Skilled Nursing Data Report