Download as PDF

Evaluation of Potential ACO Performance Bias from
Retrospective and Prospective Assignment

Executive Summary:

Current claims-based attribution methodologies require ACOs to make certain tradeoffs. Prospective attribution has the advantage of timeliness while retrospective attribution has the advantage of precise overlap between assigned beneficiaries and beneficiaries receiving primary care services from the ACO during the performance year. The NAACOS Cost and Quality Data Warehouse, containing the comprehensive Medicare fee-for-service (FFS) claims dataset from 2013 through 2015, was used to quantitatively investigate differences in key metrics that impact ACO performance for 322 Medicare Shared Savings Program (MSSP) ACOs. This is the first study to simulate the prospectively and retrospectively assigned beneficiary populations for actual ACOs.  Key metrics were evaluated for each of the 322 ACOs during the 2015 performance year.  Despite significant differences between prospective and retrospective assignment, neither methodology shows performance bias.

It was discovered that 88 percent of ACOs would have experienced lower assignment under prospective assignment. Prospective assigned beneficiary count is 10 percent lower than retrospective assignment across all studied ACOs. This poses a concern for smaller ACOs that participate in prospectively assigned tracks.  On average, 72 percent of ACOs would have experienced higher risk-adjusted per beneficiary per year (PBPY) costs when transitioning to prospective assignment. Risk-adjusted PBPY costs are 2 percent higher than retrospective risk-adjusted PBPY costs overall. The higher PBPY starting point may benefit the ACO if it can identify opportunities for optimizing patient care and appropriately capturing risk adjustment.  An 0.8 percentage point higher assigned death rate from 3.5 percent to 4.3 percent was observed under prospective assignment placing higher emphasis on end-of-life expense management for ACOs that select prospective assignment. 

Of those studied, the average ACO would have experienced only 71 percent overlap between the prospectively assigned and retrospectively assigned populations. Both the prospective and retrospective assignment methodologies demonstrate significant mismatch between intervenable and attributed populations.  This suggests that more focus should be placed on non-claims-based methods of attribution, such as voluntary alignment, to achieve sustained success under the MSSP.  Table 1 below shows decision factors to weigh to help ACOs choose assignment methodology.

Table 1: High-level Factors to Consider Between Prospective and Retrospective Assignment

Decision Factor

Description

ACO Size

If the ACO has close to 5,000 assigned beneficiaries, then the ACO may favor retrospective assignment because beneficiary count is lower under prospective assignment versus retrospective assignment.

ACO Market Competitiveness

If competing ACOs in the same local market are prospectively assigned, then the ACO may favor prospective assignment because prospective assignment has precedence over retrospective assignment.

Local Medicare Enrollment Growth

If there is a high rate of new Medicare enrollment in the local market or “age-ins,” the ACO may favor retrospective assignment to the new Medicare beneficiary market share.

Duration of Medical Management Programs

If the ACO’s medical management programs require a long duration to produce a return on investment, then the ACO may favor prospective assignment because it allows early identification of assigned beneficiaries for risk identification and stratification to medical management programs.

Member Churn Rate

If there is high member churn to other ACOs, then the ACO may favor retrospective assignment because the ACO will not be financially responsible for beneficiaries who change primary care physicians, move during the performance year or don’t have an office visit.

Beneficiary Engagement

If there is low assigned beneficiary engagement, then the ACO may favor retrospective assignment because the ACO will not be financially responsible for beneficiaries who don’t have an office visit throughout the performance year.

End-of-life Management Program

If the ACO has a robust end-of-life management or outpatient advance directives program implemented, the ACO may favor prospective assignment because the ACO is responsible for a higher assigned rate of deaths having end-of-life expenses.

 

Introduction and Background:

The choice between prospective and retrospective attribution is a critical consideration during ACO track selection.  NAACOS provides a comprehensive resource containing definitions and qualitative comparisons between prospective and retrospective attribution for the MSSP and Next Generation Model.  Figure 1 below from the NAACOS resource shows the difference between the attribution timeframes for the ACO models for a 2018 performance year.

ACOs historically have had little quantitative information to support the decision between prospective and retrospective assignment.  Lewis et al. provides quantitative differences under the Pioneer ACO Program context using the 100 percent Medicare FFS claims data from 2008 and 2009 to simulate ACO networks.  The differences in attribution rates and Primary Care Provider (PCP) visits of attributed beneficiaries were studied. 

The purpose of this NAACOS study and resource is to further quantify the aforementioned qualitative differences between prospective and retrospective assignment ACOs to better inform ACO track selection. This is the first study to recalculate attribution under prospective and retrospective assignment over the same time period for actual ACOs. The quantitative metrics concerned include assigned beneficiary count, assigned beneficiary overlap, risk score, PBPY medical expenditures, risk-adjusted PBPY medical expenditures, and annual death rate.

Study Data and Methods:

The study used the NAACOS Cost and Quality Data Warehouse which contains 100 percent of ACO assigned beneficiaries in addition to 80 percent of those eligible for assignment, but not assigned from 2011 to 2015. Data elements include all institutional and provider bills for those included in the sample. The assigned beneficiary count, assigned beneficiary overlap, risk score and PBPY medical expenditures were studied for the 2015 performance year, during which 392 ACOs participated in the program. Four ACOs were eliminated from this study because they did not have complete data in the NAACOS Data Warehouse. 

Furthermore, validation checks were used to compare actual retrospective assigned beneficiary counts from the MSSP 2015 ACO public use file (PUF) with calculated retrospective assigned beneficiary counts and PBPY medical expenditures. This resulted in the exclusion of an additional 66 ACOs that exceeded a 20 percent validation threshold. Variances between the NACOS Data Warehouse and PUF could have been caused by substance abuse claims contained within the PUF and not the NACOS Data Warehouse, large claims included in the NAACOS Data Warehouse but not the PUF, or provider participant list changes from 2013 through 2015. The final sample includes 322 ACOs.

Claims-based prospectively and retrospectively assigned populations were calculated using CMS published rules for the 322 ACOs in 2015. Since the NAACOS Data Warehouse is based on assignable beneficiaries, 2013 and 2014 claims data were available to calculate the October 1, 2013 through September 30, 2014 prospective assignment window for the 2015 performance year as if the retrospective assignment ACO were instead based on prospective assignment.

ACOs frequently change provider participant composition between performance years. Changes in provider participants from the study period October 1, 2013 through December 31, 2015 were handled by only considering the provider tax identification numbers (TINs) that were continuous participants during the entire study period. This methodology allows individual providers to change participation within the study period and is consistent with CMS policy.  This caused discrepancies between the PUF and the NAACOS Data Warehouse because the PUF includes all participant TINs as they are added or removed throughout the 2013, 2014 and 2015 performance years.

Six metrics of interest were calculated under prospective and retrospective populations for each ACO to quantify the 2015 impact of prospective versus retrospective assignment. These metrics include total assigned beneficiary count, assigned beneficiary overlap or match percentage, risk score, PBPY medical expenditures, risk-adjusted PBPY medical expenditures, and assigned beneficiary death rate. The assigned beneficiary match percentage is defined as the percentage of prospectively assigned beneficiaries who were also retrospectively assigned as a proportion of the total number of retrospectively assigned beneficiaries. 

Results:

Quantitative results for beneficiary assignment counts and overlap appear in Table 2 below.  The overall difference in assigned beneficiary count is 5,538,098 under prospective assignment and 7,003,882 under retrospective assignment (79 percent).   Despite prospective assignment having precedence over retrospective assignment, 88 percent of ACOs would have experienced lower assignment under prospective assignment. Prospective assigned beneficiary count is 10 percent lower than retrospective assignment across the studied ACOs. Reasons for this include the static nature of prospective assignment, loss of opportunity to capture increasing enrollment in Medicare, and inability to add beneficiaries throughout the performance year. 

The average ACO would have experienced only 71 percent of overlap between the prospectively assigned and retrospectively assigned populations. Reasons for beneficiaries being prospectively assigned but not retrospectively assigned include change in provider and recovery from an acute episode without seeking care in the following year.  Lewis et al. looked at a similar measure of overlap and found that 31 percent of patients did not receive primary care from ACO participants during the performance year let alone become assigned to the ACO. This is at least partially offset by the phenomenon that retrospective assignment ACOs may invest in managing the care of the patient throughout the performance year only to ultimately lose their attribution to a competing ACO. 

Table 2: Assigned Beneficiary Count Differences and Degree of Overlap Between Prospectively and Retrospectively Assigned Populations

Metric
 (N=322 ACOs)

Prospective

Retrospective

Difference

Assigned Beneficiary Count

 

 

 

Total Assigned Beneficiaries

5,538,098

7,003,882

79%

ACO Average Assigned Beneficiaries

15,799

19,663

90%

Number of ACOs with Lower Beneficiary Count

283

39

88%

Assigned Beneficiary Match %

 

 

 

Total Match Percent

4,608,256

7,003,882

66%

ACO Average Match Percent

71%

100%

N/A

 

Table 3 below shows that 64 percent of ACOs have higher PBPY medical expense under prospective assignment. Prospectively assigned ACOs have 3 percent higher PBPY medical expense overall. The exact cause for this is unknown, but it is hypothesized that a slightly older population and higher assigned death rate may be contributing factors.

There was not a consistent pattern between the average HCC risk score of the prospectively and retrospectively assigned populations. Nearly half of the time the risk score would be lower, and the other half it would be higher under prospective assignment consistent with random churn of assigned beneficiary population. Note that the raw risk score was used for this study since the renormalization factor cancels for the same ACO.

Seventy-two percent of ACOs studied would have experienced higher risk-adjusted PBPY costs when transitioning to prospective assignment. Risk-adjusted PBPY costs are 2 percent higher than retrospective risk-adjusted PBPY cost overall. Depending on the ACO’s ability to identify the source of the higher cost, the higher risk-adjusted PBPY cost may present an opportunity for ACOs to improve the efficiency of care for prospectively assigned beneficiaries. Since benchmarking and performance year evaluation methodology is symmetric, it is expected that both the benchmark and performance year risk-adjusted PBPY costs would be higher under prospective assignment. 

Table 3: PBPY and Risk Score Differences Between Prospectively and Retrospectively Assigned Populations

Metric
 (N=322 ACOs)

Prospective

Retrospective

Difference

PBPY ($)

 

 

 

Average PBPY

$ 9,149

$ 8,872

103%

Number with Higher Prospective PBPY

213

119

64%

ACO Average Risk Score

 

 

 

Average Risk Score

1.23

1.24

0.99

Number with Higher Prospective Risk Score

163

159

51%

Risk-Adjusted PBPY ($)

 

 

 

Average Risk-Adjusted PBPY

 $ 8,171

 $ 8,018

102%

Number with Higher Prospective Risk-Adjusted PBPY

233

89

72%

 

The MSSP ACO program excludes beneficiaries who die prior to the performance year from final assignment. Beneficiaries who die during the performance year count for partial credit in the shared savings formula. Table 4 below shows the difference in assigned death rates between prospectively and retrospectively assigned populations. It reveals that there is an approximate 0.8 percent higher assigned death rate under prospective assignment, which helps to explain the higher PBPY medical expenses. The main effect is due to an overlap in the assignment and performance window featured in retrospective assignment. That is, some beneficiaries who would have been prospectively assigned may have died before incurring the particular primary care services that would have caused them to be retrospectively assigned, effectively lowering the observed death rate of retrospectively assigned beneficiaries. The overall Medicare FFS average death rate is closer to the prospectively assigned rate further supporting that the retrospective death rate is artificially lowered by the retrospective beneficiary sampling methodology. Additionally, there is a minor demographic effect. Medicare beneficiaries who newly enroll in Medicare throughout the performance year are unable to be attributed leading to a higher average age under prospective assignment. Further research is needed to fully theoretically explain these empirical results. 

Table 4: Difference in Death Rates Between Prospective and Retrospective Assignment ACOs

 

2015 Deaths

2015 Total Assigned

 

2015 Assigned Death  Rate

2015 Prospective

235,751

5,538,098

4.3%

2015 Retrospective

245,294

7,003,882

3.5%

 

Policymaking Implications:

There is no one-size-fits-all claims-based attribution approach that is optimal for every ACO. This suggests that ACOs should have the flexibility to choose the attribution approach that fits their local market dynamics, available resources and patient population independent of which ACO model or track is selected. NAACOS has repeatedly advocated for the choice of assignment methodology, regardless of the specific model or track in which an ACO participates.  CMS should also exclude beneficiaries who move from the ACO’s extended service area from assignment, a policy which applies to the Next Generation Model and should be added to MSSP and future Medicare ACO models. 

Additionally, the quantitative evidence detailed in this study supports that neither claims-based attribution approach is ideal for all ACOs. However, neither attribution methodology puts an ACO at a general disadvantage from being able to achieve performance goals such as earning shared savings.  As such, more emphasis should be placed on exploring improvements to the attribution methodology.  For example, a successful implementation of a voluntary alignment process, has the potential to improve assignment as well as emphasize an ACO’s relationship with its beneficiaries and further engage them in their care and involvement in the ACO.

References:

Valerie A. Lewis, Asha Belle McClurg, Jeremy Smith, Elliott S. Fisher, Julie P.W. Bynum.  Attributing Patients To Accountable Care Organizations: Performance Year Approach Aligns Stakeholders‘ Interests, Health Aff (Millwood). 2013 Mar; 32(3): 587–595. doi: 10.1377/hlthaff.2012.0489 

National Association of ACOs (NAACOS) member resource.  The Impact of Retrospective Versus Prospective Attribution on Your ACO, https://naacos.memberclicks.net/the-impact-of-retrospective-versus-prospective-attribution-on-your-aco