With a lot of fanfare but not much input from experts and stakeholders, CMS revealed its new “Medicaid scorecard” that received mixed reviews this week. CMS Administrator Seema Verma’s reluctance to talk about how the scorecard may be used in the future added suspense to speculation about the potential for CMS to use the tool in a punitive way down the road, but we’ll reserve judgment on that point for now.

One thing we agree with CMS on is that transparency and better data for Medicaid and the Children’s Health Insurance Program (CHIP) is a worthy goal. However, this inaugural version of the scorecard falls short of incorporating meaningful measures, particularly relating to state and federal administrative accountability.

Before we dive into the details, let’s start with the process used to develop the scorecard. While CMS consulted a subset of states and informed other stakeholders of their plans a few weeks in advance, it doesn’t appear there was a robust effort to engage a wide group of stakeholders or incorporate stakeholder feedback. That’s in part why reaction to the scorecard has been lukewarm.

The scorecard is designed by CMS to report data under three categories: state health system performance, state administrative accountability, and federal administrative accountability. State health system performance data is largely based on select measures from the child and adult core sets of health care quality measures. There are some concerns about how to interpret and compare these data (which are discussed below) but building on the core sets of quality indicators makes a lot more sense than the abstract indicators chosen for state and federal administrative accountability.

Another key question left unanswered by Adminstrator Verma at the rollout (and discussed below) is what has happened to the extensive multi-year effort by CMS to upgrade its state Medicaid data collection through the Transformed Medicaid Statistical Information System (aka T-MSIS)? This system is critical to better and more timely reporting of performance data.

State and Federal Administrative Accountability

The administrative accountability measures on both the state and federal sides focus on time elapsed in submitting and processing managed care rate reviews, state plan amendments and waivers. These measures seem arbitrary, leading one to wonder what the motivation could possibly be in making them a core part of the scorecard. It’s true that timeliness can be an indicator of efficiency, but shouldn’t the focus be on accountability to beneficiaries? Shouldn’t well-run organizations seek to better serve their constituents/consumers? For example, Medicaid applicants and enrollees care about how quickly their application or renewal is processed (which is not a metric in the scorecard), not about how long it took their state to respond to a request for additional information on a waiver submission.

Key questions regarding data to monitor state and federal administrative performance are:

• What other data does CMS have that it is not making public? For example, CMS is already collecting a number of performance indicators but currently reports only enrollment and applications related data.
• Are there other good performance metrics that should be measured and collected? We have been awaiting the release of the second phase of performance indicators for five years. For example, if states were required to report denial and closure reasons for non-eligibility reasons, it could enable them to pinpoint ways to improve the eligibility verification process.
• What about access to care? If CMS wants more transparency and data for stakeholders to evaluate Medicaid performance, why is it proposing changes to state required monitoring of access in Medicaid which will likely undermine access to care?
• When will T-MSIS be fully functional? CMS has been working on its upgraded data collection system for the better part of a decade. This August 2013 CMS letter to State Medicaid Directors indicates that CMS was piloting the system and expected states to begin to submit data in July 2014. Nearly four years later, we still don’t have full implementation. This is important because T-MSIS should be collecting a diverse and abundant amount of information that would allow CMS to report data on a state-level basis using consistent specifications and promoting cross-state comparison in a timely manner. For example, if CMS would enforce timely submission of Medicaid claims/encounter data from all states, T-MSIS could then be used to report the core set quality measures on a consistent and timely basis, thereby reducing administrative burden on states – an often cited goal for this administration.

State Health System Performance

As noted above, the data included in the scorecard for this category include a subset of health care quality measures from the child and adult core sets. Core set measures are adopted through an extensive and ongoing process that includes a diverse group of experts and stakeholders. Many of the measures are based on HEDIS measures used in the commercial insurance sector to assess the quality of care, and CMS has provided lots of technical assistance to the states around measurement and reporting of these data.

The “new” scorecard allows for the comparison of state-by-state data on select measures, but these data have already been publicly available for several years on the quality pages of Medicaid.gov. However, the scorecard’s presentation of the data is more user-friendly for non-researchers than data available at Medicaid.gov.

One concern expressed over use of the core set measures is the data lag. The most recent data available was reported for 2016 but may include some state data from 2015. As noted above, this issue could be addressed by a fully functioning T-MSIS.

CCF has long embraced mandatory reporting on the child core set of quality measures, which will become a reality in 2024 thanks to the ACCESS Act passed by Congress in February. But the fact that reporting on the core sets is currently voluntary does raise concerns about how the scorecard is interpreted:

• Number of states reporting a specific measure – This is more of a problem for the adult measures. At least 44 states report all of the child measures included in the scorecard (with the exception of one new measure added to the core set in 2016). Fewer states – between 25 and 37 – report on the adult measures in the scorecard. The smaller universe of reporting states on adult measures provides an incomplete comparison across states.
• Different populations or delivery systems (i.e. managed care only) – States also choose which measures to report and for which populations or delivery systems (i.e. managed care only). This can result in unequal comparisons. For example, a state reporting on all adults in Medicaid would likely experience different outcomes compared to a state reporting only on its healthier, non-disabled population. A state reporting only on children enrolled in Medicaid would likely experience lower performance on some measures than a state reporting only on children enrolled in CHIP because health status is linked to income and kids in CHIP by definition have higher incomes. Likewise, comparisons to the same measures in commercial insurance should not be made without technical adjustments that account for differences in disease prevalence and other factors that impact health and health care utilization.

While these differences are alluded to in the narrative that accompany the measures, the graphical representation of the data may lead people to skip the fine print and make assumptions that result in apples-to-oranges comparisons. One needs to have a basic understanding of the core sets and must dig into the data to assess why variability exists before drawing conclusions.

Going forward, CMS should outline a transparent process with sufficient time for earnest consideration and inclusion of stakeholder input. Transparency and accountability in Medicaid are extremely important. But the number days it takes states to provide information or CMS to act on it in processing waivers and other administrative tasks is by no means a measure of quality. Quality and efficiency in Medicaid and CHIP should be viewed through the lens of how well these coverage sources are serving beneficiaries and delivering the care they need.