Recently, CMS sent out press releases touting over $1 billion in savings from Accountable Care Organizations.  Here’s the tweet from Andy Slavitt, the acting Administrator of CMS:

NEW ACO RESULTS: physicians are changing care, w better results for patients & are saving money. Over $1B. https://t.co/FEfe3QErb5

— Andy Slavitt 🇺🇦 (@ASlavitt) August 25, 2016

The link in the tweet is to a press release.  The link in the press release citing more details is to another press release.  There’s little in the way of analysis or data about how ACOs did in 2015.  So I decided to do a quick examination of how ACOs are doing and share the results below.

Basic background on ACOs

Simply put, an ACO is a group of providers that is responsible for the costs of caring for a population while hitting some basic quality metrics.  This model is meant to save money by better coordinating care. As I’ve written before, I’m a pretty big fan of the idea — I think it sets up the right incentives, and if an organization does a good job, they should be able to save money for Medicare and get some of those savings back themselves.

ACOs come in two main flavors:  Pioneers and Medicare Shared Savings Program (MSSP).  Pioneers were a small group of relatively large organizations that embarked on the ACO pathway early (as the name implies).  The Pioneer program started with 32 organizations, and only 12 remained in 2015.  It remains a relatively small part of the ACO effort and for the purposes of this discussion, I won’t focus on it further.  The other flavor is MSSP.  As of 2016, the program has more than 400 organizations participating and as opposed to Pioneers, has been growing by leaps and bounds.  It’s the dominant ACO program – and it too comes in many sub-flavors, some of which I will touch on briefly below.

A couple more quick facts:  MSSP essentially started in 2012 so for those ACOs that have been there from the beginning, we now have four years of results.  Each year, the program has added more organizations (while losing a small number).  In 2015, for instance, they added an additional 89 organizations.

So last week, when CMS announced having saved more than $1B from MSSPs, it appeared to be a big deal.  After struggling to find the underlying data, Aneesh Chopra (former Chief Technology Officer for the U.S. government) tweeted the link to me:

While there are dozens of interesting questions about the latest ACO results, here are five quick questions that I thought were worth answering:

1. How many organizations saved money and how many organizations spent more than expected?
2. How much money did the winners (those that saved money) actually save and how much money did the losers (those that lost money) actually lose?
3. How much of the difference between winners and losers was due to differences in actual spending versus differences in benchmarks (the targets that CMS has set for the organization)?
4. Given that we have to give out bonus payments to those that saved money, how did CMS (and by extension, American taxpayers) do? All in, did we come out ahead by having the ACO program in 2015 – and if yes, by how much?
5. Are ACOs that have been in the program longer doing better? This is particularly important if you believe (as Andy Slavitt has tweeted) that it takes a while to make the changes necessary to lower spending.

There are a ton of other interesting questions about ACOs that I will explore in a future blog, including looking at issues around quality of care.  Right now, as a quick look, I just focused on those five questions.

Data and approach

I downloaded the dataset and ran some pretty basic frequencies.  Here are data for the 392 ACOs for whom CMS reported results:

Question 1:  How many ACOs came in under (or over) target

Question 2:  How much did the winners save – and how much did the losers lose?

Table 1

	Number (%)	Number of Beneficiaries	Total Savings (Losses)
Winners	203 (51.8%)	3,572,193	$1,568,222,249
Losers	189 (48.2%)	3,698,040	-$1,138,967,553
Total	392 (100%)	7,270,233	$429,254,696

I define winners as those organizations that spent less than their benchmark.  Losers were organizations that spent more than their benchmarks.

Takeaway: About half the organizations lost money, and about half the organizations made money.  If you are a pessimist, you’d say, this is what we’d expect; by random chance alone, if the ACOs did nothing, you’d expect half to make money and half to lose money.  However, if you are an optimist, you might argue that 51.8 percent is more than 48.2 percent, and it looks like the tilt is towards more organizations saving money and the winners saved more money than the losers lost.

Next, we go to benchmarks (or targets) versus actual performance.  A reminder that benchmarks were set based on historical spending patterns — though CMS will now include regional spending as part of their formula in the future.

Question 3:  Did the winners spend less than the losers — or did they just have higher benchmarks to compare themselves against?

Table 2

	Per Capita Benchmark	Per Capita Actual Spending	Per Capita Savings (Losses)
Winners (n=203)	$10,580	$10,140	$439
Losers (n=189)	$9,601	$9,909	-$308
Total (n=392)	$10,082	$10,023	$59

A few thoughts on table 2.  First, the winners actually spent more money, per capita, then the losers.  They also had much higher benchmarks — maybe because they had sicker patients — or maybe because they’ve historically been high spenders.  Either way, it appears that the benchmark matters a lot when it comes to saving money or losing money.

Next, we tackle the question from the perspective of the U.S. taxpayer.  Did CMS come out ahead or behind?  Well — that should be an easy question — the program seemed to net savings.  However, remember that CMS had to share some of those savings back with the provider organizations.  And because almost every organization is in a 1-sided risk sharing program (i.e. they don’t share losses, just the gains), CMS pays out when organizations save money — but doesn’t get money back when organizations lose money.  So to be fair, from the taxpayer perspective, we have to look at the cost of the program including the checks CMS wrote to ACOs to figure out what happened.  Here’s that table:

Table 3 (these numbers are rounded)

	Total Benchmarks	Total Actual Spending	Savings to CMS	Paid out in Shared Savings to ACOs	Net impact to CMS
Total (n=392)	$73,298m	$72,868 m	$429 m	$645 m	-$216 m

According to this calculation, CMS actually lost $216 million in 2015.  This, of course, doesn’t take into account the cost of running the program.  Because most of the MSSP participants are in a one-sided track, CMS has to pay back some of the savings – but never shares in the losses it suffers when ACOs over-spend.  This is a bad deal for CMS — and as long as programs stay 1-sided, barring dramatic improvements in how much ACOs save — CMS will continue to lose money.

Finally, we look at whether savings have varied by year of enrollment.

Question #5:  Are ACOs that have been in the program longer doing better?

Table 4

Enrollment Year	Per Capita Benchmark	Per Capita Actual Spending	Per Capita Savings	Net Per Capita Savings (Including bonus payments)
2012	$10,394	$10,197	$197	$46
2013	$10,034	$10,009	$25	–$60
2014	$10,057	$10,086	-$29	-$83
2015	$9,772	$9,752	$19	-$33

These results are straightforward — almost all the savings are coming from the 2012 cohort.    A few things worth pointing out.  First, the actual spending of the 2012 cohort is also the highest — they just had the highest benchmarks.  The 2013-2015 cohorts look about the same.  So if you are pessimistic about ACOs — you’d say that the 2012 cohort was a self-selected group of high-spending providers who got in early and because of their high benchmarks, are enjoying the savings.  Their results are not generalizable.  However, if you are optimistic about ACOs, you’d see these results differently — you might argue that it takes about 3 to 4 years to really retool healthcare services — which is why only the 2012 ACOs have done well.  Give the later cohorts more time and we will see real gains.

Final thoughts

This is decidedly mixed news for the ACO program.  I’ve been hopeful that ACOs had the right set of incentives and enough flexibility to really begin to move the needle on costs.  It is now four years into the program and the results have not been a home run.  For those of us who are fans of ACOs, there are three things that should sustain our hope.  First, overall, the ACOs seem to be coming in under target, albeit just slightly (about 0.6 percent below target in 2015) and generating savings (as long as you don’t count what CMS pays back to ACOs).  Second, the longer standing ACOs are doing better and maybe that portends good things for the future — or maybe it’s just a self-selected group that with experience that isn’t generalizable.  And finally, and this is the most important issue of all — we have to continue to move towards getting all these organizations into a two-sided model where CMS can recoup some of the losses.  Right now, we have a classic “heads — ACO wins, tails — CMS loses” situation, and it simply isn’t financially sustainable.  Senior policymakers need to continue to push ACOs into a two-sided model, where they can share in savings but also have to pay back losses.  Barring that, there is little reason to think that ACOs will bend the cost curve in a meaningful way.

Ashish Jha is an associate professor of health policy and management, Harvard School of Public Health, Boston, MA.  He blogs at An Ounce of Evidence and can be found on Twitter @ashishkjha.

Image credit: Shutterstock.com