Rural Hospital Financial Performance: Tools for Data-Driven Decision Making

Featured Speaker

	Michael Fallahkhair, Deputy Associate Administrator for Rural Health, Health Resources and Services Administration
	Sara J. Couture, Social Science Analyst, Division of Healthcare Quality and Outcomes, Office of Health Policy at the Office of the Assistant Secretary for Planning and Evaluation
	Mark Holmes, Director, North Carolina Rural Health Research Center

Join us for an overview of two innovative resources designed to support a better understanding of rural hospital financial performance. This webinar will feature subject matter experts from the Office of the Assistant Secretary for Planning and Evaluation (ASPE), showcasing its Rural Hospital Financial Performance Dashboard, followed by researchers from the North Carolina Rural Health Research Center presenting the Rural Hospital Financial Distress Index. Together these objective, data-driven tools provide valuable insights into the financial health of rural hospitals and can help inform planning, policy, research, and decision-making. Attendees will learn about each resource, see live demonstrations, and explore how these complementary tools can be used to better understand financial challenges facing rural hospitals.

From This Webinar

• PowerPoint Handout
• Listen to or download an audio version [MP3]

Transcript

Kristine Sande: Hello everyone and welcome to today's webinar. I'm Kristine Sande and I'm the program director for the Rural Health Information Hub. And we are delighted to be collaborating with the Federal Office of Rural Health Policy to host today's webinar, Rural Hospital Financial Performance: Tools for Data-Driven Decision Making.

It's my pleasure to introduce Michael Fallahkhair from the Federal Office of Rural Health Policy where he is deputy associate administrator. FORHP is located in the Health Resources and Services Administration of the US Department of Health and Human Services. In his role, Mr. Fallahkhair helps to lead the work of FORHP, which is charged with advising the secretary of HHS on rural health issues and improving the delivery of rural healthcare. Mr. Fallahkhair previously served as the Chief of Staff for the Office of the Assistant Secretary for Health, Executive Officer in HHS Immediate Office of the Secretary, and as principal advisor in FORHP. He has his past work experience in the Office of Budget of the Assistant Secretary for Financial Resources at HHS as well as the Office of Management and Budget. Welcome, Michael. And with that, I'll turn it over to you to introduce our speakers.

Michael Fallahkhair: Thank you, Kristine. I really appreciate that. And good afternoon everyone. Thank you for joining today's webinar on rural hospital financial performance and data -driven decision making. So as Kristine said, as you may know, FORHP supports the rural communities, the hospitals, the clinics in the broader rural healthcare delivery system through grants, technical assistance, research, policy analysis, and the greater coordination across the Health Resources and Services Administration, but also the Department of Health and Human Services. And so in that with that in mind, we're holding this webinar because rural hospital financial performance still remains one of the most important issues facing rural healthcare. Rural hospitals provide care across the entire continuum. So if you think about it's from emergency, inpatient, outpatient, primary care, post-acute care, and even in some communities, long-term care. And when these hospitals are financially vulnerable, the effects reach patients, they affect the workers, the local economies, and access to care close to home.

These hospitals often face structural pressures. They're different from other many hospitals, especially urban hospitals. They're often lower patient volumes, workforce shortages, there are reimbursement pressures, uncompensated care, changes in service demand all affect how rural hospitals operate and whether they can even maintain the key services they need to for their communities. Now, the Assistant Secretary for Planning and Evaluation or ASPE, their recent work here helps provide national view of those pressures. Their report found that rural hospitals are more vulnerable to closure or conversion to outpatient-only facilities than urban hospitals. That's the significant differences ASPE will definitely get into more. Now, at the Federal Office of Rural Health Policy, this work, our work, directly informs how we then think about supporting rural hospitals. We have a hospital state division that administers all of our programs like the Flex, the SHIP, and other hospital-related technical assistance programs. Our policy research division reviews federal policy regulations and rulemaking, including CMS policies for potential rural impact.

Our grantees and partners work directly with rural hospitals and community. So today's webinar also reflects importance of this collaboration that we have across our office, but also at HHS. Rural hospital sustainability is not just a FORHP issue or an ASPE issue. It requires shared data, analysis and coordination across the department, including, like I said, ASPE, HRSA, CMS, but also FORHP-funded research partners like the University of North Carolina where Mark Holmes comes from.

Dashboard we'll discuss today includes data through 2023. Provides a tool for examining hospital level financials and structural information. And it's really important to remember that it does not replace local context or answer every question, but it helps to give policymakers, state researchers, rural stakeholders like you all, a common evidence base to better understand the trends and identify areas of concern. This is especially timely as HHS and CMS continue to focus on rural hospital sustainability and implementation through programs like the Rural Health Transformation Program. So better data and analysis can really help states and ourselves and communities understand where these hospitals are financially vulnerable, where services may be at risk, where TA technical assistance or policy attention may be most useful.

I also want to highlight and thank for funding the Rural Health Information Hub. Thank you guys so much for hosting this webinar, being the key statutory clearinghouse that we have in our office that can help inform decision makers and rural stakeholders at all levels, national, state, community about rural health. It's especially meaningful that the secretary's office through Assistant Secretary for Planning and Evaluation, ASPE and Sara took this important topic very seriously and their desire to work with RHIhub together on this is very important as well. We really thank you all for that. So with that context, I'm very pleased to introduce today's presenters. So first we have Sara Couture.

She's a Social Science Analyst in ASPE's Office of Health Policy, and her work focuses on using healthcare data to inform policy and developing accessible data-driven resources. It's amazing. Mark Holmes is the PhD. He's the director of the North Carolina Rural Health Research Center. His research examines why rural hospitals experience this financial distress, how payment policy shapes their viability and where and what that evidence suggests about preserving access in rural America. So thank you all for joining us. Thank you to our speakers for being here. We look forward to a really practical discussion on how these tools can help inform planning, policy, research, and decision-making. Handing over to you guys. Thank you.

Sara J. Couture: Awesome. Thank you very much, Michael. As Michael said, my name is Sara Couture and I am part of the work that ASPE's been doing on looking at the state of rural hospitals in the United States. The usual disclaimer that the findings and conclusions are those of the authors do not necessarily represent the position of the Department of Health and Human Services. And I just want to take a moment to thank the team that's really been working on a variety of different projects around rural health. The dashboard was developed with myself and my colleague, Fredo Louis, and with a lot of the data work in helping us identify which graphics was really work with Kaushik, Eden, Ge, and Scott. We want to thank everybody for their hard work on all of these important projects.

So ASPE had kind of a suite of rural hospital projects. The goal of these really was to provide internal policymakers hospital level data to help understand the financial and environmental risks that influence the financial state enclosures within rural hospitals. There was a variety of different projects. The main one was focusing on providing a model to predict risk of closures. And when we were putting those information together, we were able to get hospital level information and we want to be able to provide in a way that was accessible. And we also want to improve how to understand where residents get their care and the impact that hospital closures have on those particular patterns.

To do this for both the initial issue brief as well as a dashboard, we've combined multiple data sets, including the Medicare cost reports data, the American Hospital Association's annual survey, the area health resource files, and county level information. We were able to make this data more accessible using a dashboard, which I'll do a demo of here soon.

But we also used this data to develop a Cox model to predict risk closures that was published as a separate issue brief. The Cox model is not in the dashboard, but a lot of the same variables from the model are what showcased in the dashboard. We also did a few other projects, including a Medicare and Medicaid claims level analysis of care patterns in rural areas as well as a literature review.

So the goal of the dashboard itself was to provide the public with comprehensive information on rural hospitals that are both open and closed. And we were able to do it from 2012 to 2023, although we are looking on updating the 2024 data. The dashboard includes three main components. The first one is a national level map that shows open and closed rural hospitals by year. And it allows users to filter on multiple things, including identifying specific hospitals, filtering by closure risks, the occupancy rates, quartiles—so highest occupancy versus lowest occupancies—ownership type, and system affiliations.

We're also then able to select a specific hospital and drill in to see a longitudinal view of some of their data, including their occupancy rates, number of beds, Medicare and Medicaid share of discharges, liability asset ratio, and annual profit margins. And then we also want to be able to provide a bigger context of what was happening in the county that that hospital was located. So it includes things of the amount of access to the healthcare workforce, other healthcare facilities, as well as some demographic information on the county level.

So some key findings is the dashboard itself, like I said, aligns with the model, although the model itself is not in the dashboard, that helps identify key determinants of rural hospital closures or conversions in outpatient-only facilities. The analysis, that you can see the issue brief is linked down here below, did show that low occupancy rates, for-profit ownerships, and proximity to an urban counties are key predictors of closure or conversion.

I'm going to quickly share my screen and show a couple examples of how to use the dashboard. So when you go to the ASPE Rural Health Dashboard, you're going to first get this page, which provides an overview as well as going over some of the variables and characteristics that I did in the presentation. From here, you can proceed to the dashboard and it'll have this map that shows the number of rural hospitals and in this case in 2023, so that's a default right now, but you can use this number line to see it over time and it'll showcase the number of hospitals, how many closed that year, and how many total open hospitals there were. You can also see the number of hospitals by state down here, and you can do it from 2012 to 2023.

Over here, you're also able to use those filtering mechanisms that I mentioned earlier. I'm going to go back and show you the hospital name and ID in a few minutes, but first I'll go through some of the other filters. So you can filter on looking at just the closed hospitals. So here's an example. These are in red and you can also see the hospitals and where they were located down in the bar graph below.

You can filter by specific states. So you can see the total number of hospitals by the state either using the filter here. You can also click on the state within the bar chart here so you can see Texas is being highlighted.

You can filter by occupancy rates. So you can see in 2023 which hospitals had the highest occupancy and you can also filter by which ones had the lowest occupancy rates as well as being a critical access hospital. And this one, as I stated, was kind of important is which hospitals are in a county that are adjacent to an urban county.

So to be able to identify a specific hospital, you can use one or two ways. The way we recommend is if you have the hospital ID is to enter it here. You click there, push enter, and then you'll be able to find that particular hospital on the map. Please note that if this hospital say closed in 2016, it would not be on the 2023 map. So if you're having a hard time finding a hospital that closed, make sure you go to a year that it was opened or the year that it closed.

So to be able to drill through, there's a few ways of doing it. One is to hover over this dot and you'll see this drill through and then you click on hospital characteristics. Another way is you can just click on the dots and use this button here.

This provides an overview of the hospital itself. So we provide the information. So this one is Indiana University's Health Bedford Hospital that's in Hartford City, Indiana. You can see it's a nonprofit hospital that's system affiliated and critical access, and it closed in 2023. You can also see which county it's in up here. And we tried to provide some longitudinal data that we found in the ASPE brief were important for risk of closure. So occupancy rate's a major one. You can see in this hospital that the occupancy rate was decreasing over the last 10 years before it closed. And the actual bed numbers itself was 15 and that was pretty consistent across. You can see the Medicaid and Medicare share of discharges per year. It's pretty common, and you see here that it's a high percentage, especially of Medicare patients that was part of their discharge. You can also see their profit margin by year as well as their liability asset ratio. In order to see a little bit more information about Bedford County itself and get a bit more perspective on what's going on in this particular hospital, you can go up here to click on county profiles and here provides a bit more information about Bedford County, including its population density from 2010 to 2020. Its root code as well as the average population over the past 10 years and if it's adjacent to an urban county or not. And as I stated earlier, this does have a significant increase in closure risk.

From here, we have a couple different ways of being able to explore a little bit more about the county itself. So the first one is looking at access to healthcare providers. The default will be looking at primary care providers and we look at a ratio of population per provider over time. So over the past 10 years, about a little less than 2,000 people per primary care provider within Blackford County. If you click this dropdown menu, you can explore other types of providers. So say nurse practitioners, you'll see it over time. If there are no nurse practitioners or if the data is missing, it'll look blank like you could see it over here. So that can mean a couple things. It's not necessarily that there's no providers. It could be that we don't have the data for that particular year.

We can also look at access to other healthcare facilities. So this one's for rural health clinics, so either they didn't have one until 2021 or we just didn't have the data for it, but they have about one rural healthcare clinic per 12,000 people. You can also explore other types of clinics. So there's no federally qualified health center here or they don't provide the data for it, but there's many different types that would be available depending on the county.

We also provide some information on the length of life in this county compared to other counties within the state. So we looked at this at quartile risk. So if it's a one, it means that it's in the top 25% of counties in that particular state, in this case Indiana. If it's four, it's at the bottom 25% of counties in that state. So here the length of life was in the bottom 25% of counties in Indiana.

And then finally, we're able to explore some of the county demographic information, including the household income as well as some others such as the population total. So this one has been decreasing over time. And you can also see the age groups. So if you look here, there's a pretty high proportion of people under the age of 18 and over the age of 65, but not a whole lot of people of working age, for example.

I'm going to do one more example to just see a comparison of a hospital that is open, at least in 2023, and show how you can also find a hospital by name. So for this one, we are looking for Helena Hospital in Arkansas. If it is a common name, we'll get multiple ones in there. So when you do that, you'll have to click on each hospital to see which one you're interested in. There are many common hospital names. It's also pretty common if there's a larger hospital system, each of the little clinics will have a separate name based on the town. So the hospital ID really is the easiest way of finding it, but you have this ability too.

All right, so we're going to look at this hospital in Arkansas. So this is Helena Regional Medical Center in Helena, Arkansas. It's a for-profit system-affiliated hospital and a non-critical access hospital. And you're seeing some interesting things here as well. So one, you see the occupancy rate is also overall decreasing and the number of hospital beds had a pretty sharp decrease and then a bit of an increase in 2023. You can see its share of Medicare and Medicaid data, which still is pretty significant, although less than the previous hospital, as well as its profit margins and liability to asset ratios. So you can see financially this hospital does seem to be having some potential problems.

If we go to the county profile, we'll get a similar view as we saw before. You'll see the density from 2010 to 2020 and as well as be able to explore the different context of what's going on within the hospital system, including access to providers, which this one has been improving over time. Access to other types of healthcare facilities. Federally Qualified Health Centers here. It's standing, which was also in the bottom 25% for length of life and its demographic information, including household incomes and population, which has been showing a decline.

So that's overall the demonstration of the Rural Health Dashboard. We have seen that it has been useful for policymakers internally and we hope will be helpful for other researchers as well.

Mark Holmes: Thank you, Sara. I think I'm just going to jump right in without a Kristine and Michael buffer, so appreciate that. I want to thank RHIhub and the federal office for inviting us here today and in particular to share the webinar with ASPE. It's a really impressive tool. And if you haven't checked it out, I encourage you to spend some time poking around like it because it's really impressive.

So my role today is to talk about the Financial Distress Index and I'm going to spend a fair amount of time. That's what I'm going to really be focusing on. And I have a colleague that many of you may know, Erin Fraher at the Sheps Center who likes to do the talk in one slide. And this way you can read this and then zone out and update your MySpace account or whatever. My students like to do it this way as well, but really five bullets here.

And we're talking a lot about rural hospital closures, and I think the smarter question is upstream and how healthy is it before it closes? And I think that's consistent with the ASPE presentation you just saw and really thinking about it not as a closure as the final outcome, but what can we do upstream and prevent those from, or at least prepare for those better?

The Sheps Center built the Financial Distress Index or FDI. We'll talk about that. Again, that's the bulk of what I'll be speaking about today, but it's a simple plain language measure of a rural hospital's financial health and is really trying to give you an easy thing to conceptualize around. Built for rural hospitals, open source, forward-looking and based on public data with an annual update.

It’s going to be a double-edged sword here in some sense, but it works. We're going to present some evidence of its predictive power and talking about how well it's forecasting outcomes two years down the road. But I also want to spend a fair amount of time talking about it as a screen, not a diagnosis. It's a prediction. It's not as good as weather predictions to be honest. And so if you look at it's something that you want to keep an eye on, but it's going to involve a human in the loop and that's going to be a major portion of the secondhand. If you look at your weather app on your phone and it says it's going to rain today, you're going to grab an umbrella, but you may or may not need it, but it's something that you can look at in order to prepare.

First we're going to start with the why and give the background for what we're trying to do. You saw the ASPE website that talks about hospitals closures and what's open. We've been doing a closure website for a while as well and the closures are getting the headlines. This is our map and you could see we could talk about this particular map for a while in terms of trends and where these are located and not. But how can we assess the financial health of rural hospitals before they close? And by doing so, it allows us, interpreted broadly, to be better prepared to prevent closures or prepare for more orderly transition. So I'm going to use both languages throughout the next 30 minutes in talking about preventing them or preparing a community for the transition to a post-hospital healthcare ecosystem.

Since we're on here, the other thing I'll say that I jotted down during Sara's talk is that people sometimes get surprised, but counting closures is not as straightforward as you might think. I think it's an easy thing for us to conceptualize in that there's a hospital that's open today and it's not tomorrow, but there's a lot of gray area or grayness into what constitutes a hospital closure. The standard that we're taking is, does a community lose access to inpatient facility? Which may be different from other definitions. And so it may be the case that you pull up the ASPE website and you pull up our website and say, "There's a discrepancy here." I haven't spent time looking at all 136 or whatever we're at today to compare them, but it is a difference in terms of how to think about it. So for example, we count a closure if a hospital were to move 25 miles away from the town, that community lost its access to its inpatient even if it built a new facility 25 miles away and different places are going to count that as a closure. We also have discussions about, for example, a hospital that closes its inpatient wing, we define it as a closure even if it continues to provide outpatient services. So, the notion of what closure is is also a little nuanced.

So financial analysis is hard and here we have the wizard, George Pink trying to teach the frustrated student here how to think about financial analysis. But when I started working with George about more than two decades ago, I came in as an economist and was like, "All right, so let's do some easy things like, higher profit good." Well, not necessarily. It depends. Maybe you're over investing and maybe it's a one-time thing. And what I quickly learned is that to really have a sense of what you're doing of what the hospital looks like requires a comprehensive overview. And this is why people get degrees in finance and hire consultants because it's not as simple as a quick measure.

I would rather have a profit margin of eight than a loss of four, but there's a wide variety of circumstances that fit behind that. And so what we tried to do and what we have done is to build this comprehensive, simple measure to more quickly assess a hospital's financial position, even if it is only a rough estimate.

So why we built this? And there's a long history of financial indices to capture financial distress. And some of the early examples are, I think the father of this is the Altman Z score or Z score depending on whether you're from Canada or the United States, but it was built for publicly traded companies and really calibrated on manufacturing firms. That's not going to translate well to hospitals for a couple of reasons. One, healthcare's pretty different for manufacturing, but second, very few hospitals are publicly traded and almost no rural hospitals are publicly traded. So it's not going to work as well for that.

Bill Cleverley and other associates have designed a couple different indexes, the Financial Distress Index among others. And that was designed for hospitals, but not as calibrated as much for rural. And we felt like we could do better than that. So that's what we set to do in the mid-2010s to really find something that works better for rural hospitals. And the concept here is that there's a measure of financial distress that's unobserved. So we don't know that a hospital is in distress. We can see markers of it. If it closes, that's the ultimate sign of financial distress or a bankruptcy is also a sign of financial distress, but negative profit is also a sign of financial distress. Having negative equity or a negative fund balance, depending on how you think about accounting procedures. These are all signs of distress.

What we're trying to do is really measure that unobserved latent variable kind of concept of, what is it that's behind the scenes that is really leading to these signs of things that are developing?

So these are the principles that we sat down as we were thinking about this, and we started with a non-proprietary open source formula where the formula instructions are available to the public. And I'm going to take a moment here to commend ASPE because they've done exactly this as well. If you look, Sara presented the dashboard, but there's a findings brief that's associated with that, with these Cox regression model that they use in order to predict it. And you can see the coefficients right there. They're very clear about where the data come from. So that is open and anyone can use it to reconstruct their own or to reconstruct the data and say, "Well, here's how to do the data. Here's the formulas that go in. We add them together and we get these predictions that come out of it." And we felt that was really important to be transparent in that respect.

It's designed for rural hospitals. We calibrate it among rural hospitals and rural hospitals has quotes around it because we're defining that semi-broadly, not just hospitals that are located in rural areas, but critical access hospitals, for example. So it's designed for rural hospitals, which look pretty different than urban hospitals in many respects. They're smaller, they generally have lower profit margin, all the things that we know about for those who are following rural hospitals. And so it was important not to build this on hospitals in Boston and New York City and Chicago and LA, but to build it on hospitals in the thumb of Michigan and the panhandle of Oklahoma, because that's going to be more relevant for rural hospitals.

We wanted it to be forward-looking and predicting future outcomes. So thinking about today based on what we observed, what do we think is going to happen in the future? Because that's going to be important if we're going to think about how do we intervene? How do we prepare? How do we prevent? It's got to have something that looks forward.

Consistent with that first bullet, we wanted it to use public data. So it's largely built on HCRIS [Healthcare Provider Cost Reporting Information System] or Medicare cost reports, but it has other elements as well that go into it. But all the data that are behind it are available to the public.

We wanted it to be useful for simulation. And so for example, we wanted it to be able to say what happens if revenue falls by 20%? What happens if this policy is enacted? What happens if a hospital stops reinvesting? What happens…variety of thought experiments in order to predict what would happen? For example, we did this about 10 years ago using this model to look at what would happen if critical access hospitals were to lose their cost-based reimbursement policy, go to PPS payment using estimates that were developed by others of a 20 to 30% decline in Medicare revenue. We ran that through. We were able to see what would happen in terms of the financial distress.

We want to face validity so that when we listed the predictors, people look at it and go, "Yeah, that seems right. I think that's something that belongs in there." Again, very similar to what ASPE's model as well. I'll also take another moment here. An advantage, I think, and a strength of having a couple different models and multiple teams looking at this is that people are thinking of new elements that go in there that are a little creative. For example, we have not included a contiguous proximate, I forget the exact noun, but next to an urban county. And I think they showed that that was important. That's something that we should be thinking about as we update our model as well.

Using that as a kickoff, update annually, but revising regularly, this is version 3.0 of the FDI. The first one was built in 2017. We updated it with new data, so with new cost report data, with new other elements that go into it, but then revise it for a different model, a post-COVID world, for example, regularly, roughly every three to five years.

And then finally, the results should be easy to understand. And I have a nota bene on the results there. The results, the implication should be easy to understand even if the Greek letters that are behind it are a little harder to follow. Again, consistent with ASPE, many people on this call don't know what a Cox model is, but they can look at the output of it and go, "Okay, higher is bad. I get it."

Okay, so that's the “why.” So now we're going to get into the “what.” What is the FDI? So we're going to tell a two-part story here and on the left here we have GPT's representation of George Pink, Tyler Malone, and myself who are behind this version of the FDI. And on the right we have Tom Morris. And here we have this really fancy automobile and what we're going to hear about first is how awesome it is. And then we're going to get into the second part of the talk where we talk about some of its limitations.

So rather than start with Greek letters, I think it's easiest to think about this in terms of a boxes and arrows kind of concept. And so I'm going to walk through this, but let me orient you the big idea here. The current information are all the variables that are going into the model.

So we use all of these data, profitability, looking at this year, last year and two years ago, outpatient revenue, uncompensated care, benchmark performance, et cetera, et cetera. You can read the whole box. We put that into our model to predict distress and out of that come the following four categories of results: highest risk, mid-highest risk, mid-lowest, and lowest. We spent three months coming up with what to call those four levels and academics are not known for being creative, and so that's what we came up with. The brief where we talk about this is on the bottom right there and you can blow it up and look at it.

So let's walk through each of these components. So the first box here is the finances, and this is playing a major role, and this is the heavy lifting of the model. A lot of the predictive power comes out of these indicators, both direct financial performance profitability is probably unsurprisingly a huge predictor of future distress. Think of it's the opposite of the mutual fund. Past performance does not predict future results. Here we're saying it does, and in fact it does.

But we also include government reimbursement policy because that's going to be real important. ASPE's results had CAH as an important indicator. They included Medicare and Medicaid payer mixes as predictors. We have measures of the fee index, which is a measure at the state level, I'll use the word generosity, but the percent of Medicare fees that Medicaid program in the state offers. And those are going to be important drivers of the underlying finances. And so think of that as a structural thing. And so this is exactly the kind of thing where we can look at and say, what happens if we were to change the fee index? So we could look at it and model, "Okay, what happens if a state goes from 90% of Medicare to 100%?" And see what that does to the distress level.

Hospital characteristics, ownership size, are you a member of a system? And then market characteristics or community characteristics depending on your perspective. But the service area, the community that you're serving is going to be really important. And think about the economic condition. It's going to be a lot easier to have strong finances if you're in a place with a high per capita income than you're in a community with a low per capita income and all the things that go behind that. Again, market size. We know that smaller hospitals and smaller markets generally lead to higher distress levels.

And then so we're looking at the risk in two years hence. We'll talk about that in a little bit. But again, this is the forward-looking part of the model. And then finally, next slide. It's a straightforward categorization, but it has a gradient. And so rather than saying at distress or not, it's really recognizing that there's a continuum here. And we could break it into 10 groups. We figured four was a pretty nice round number. We might go crazy in the next iteration, have five groups. Stay tuned to see how wild and crazy we feel at that year.

And then what we've done here is you can see the new variables that are outlined in red here to show what's changed since the previous. And so again, this reinforces this notion of we're revising it regularly and recognizing that the conditions that lead to distress in the mid-2020s are probably different than they were in the early 2010s. And so continuing to refine the model based on new updates.

All right, so how well does it work? Would love to nerd out and talk about Greek letters here, but this is a representation of the prediction. And so that purple line, what you can think of, for example, is the horizontal access is the probability of distress, but think of it as a probability of a certain signal. And so anything on that purple line means that we've completely nailed it. And so anything in the zero to 10% bucket has an average mark of distress between zero and 10%.

The way we did this in the second bullet there is the training sample and test sample. So we took the list or the collection of rural hospitals in multiple years and we looked at a subset of them and built our model and then said, "All right, how does that predict in the set that weren't included in the model?" So what we're doing in this and this is good predictive practice, because we're not predicting what the model has already seen. We're using the model to predict things it hasn't seen yet and seeing how well that works.

In terms of the lower side, that green circle there, look how well those dots are tracking with that perfect prediction line. We got a fist bump there. We're pretty excited about that how well we were doing. The size of the bubble represents the number of hospitals in that. And so you can see that the biggest bubble is those at the really low risk. And you can see that as probability, the predicted probability goes up. The observed probability of bad outcomes also increasing right on that purple line. So that was great news.

You can see that at the higher levels of prediction, we're under predicting distress a little bit, particularly those what, four, well, three points at the bottom left of the circle. So for those guys, we're predicting pretty high risk. Sorry, we're under-predicting or over-predicting distress. I said under-predicting there. We're predicting a higher risk than we actually are. So that's something that we're going to look at refining as we go forward.

Okay, so that was the overall prediction. Let's look at what this means from the signals. And so distress is an assessment, but what we really see are the signals of that. And so for this model, we're using closure, which is again, the ultimate signal, the hospital negative equity. And so the hospital owes more than it's worth and a negative cashflow margin that it is not meeting its bills effectively. And you can see here that we have the bucket's lowest, mid-lowest, mid-highest, highest. And for all four signals, the higher your risk category, the more likely you are to have that signal. So among the lowest risk categories, 0% closed. Among the highest risk, 3% closed.

Looking at the negative cashflow margin among the lowest risk, 5% or almost 6% had a negative cashflow margin compared to 62% in the highest risk. And so this is I think evidence that shows that this model's working pretty well. It does a pretty nice job of predicting who's going to have signals of distress two years later.

So limitations, how to use responsibly. So I've spent the first 15 minutes or so talking about how great this is. I'm selling you on this some kind of sports car, I'm not really sure what. Now we're going to talk about how to use this responsibility responsibly or the limitations of the FDI. So here's the same deal. And George, Tyler, and Mark are saying that, well, the FDI was, or whatever it was, efficient and fast and fun, but it does a terrible job carrying 2x4s. And so the way to think about this is that the FDI is a nice to great tool. I would say it's great. Other people might say it's fine, but it's not going to work in all circumstances. And you can see here that it's a terrible tool for certain applications such as this.

I'm going to start with this one, and these are numbered in terms of the limitations to think about with FDI. It is not a perfect predictor and we like to consider it as a screen or as a diagnosis. It is a unidimensional. Again, just we have the four buckets based on cost reports and other public data. It has no private information. It works pretty well in populations. That was the bar chart, but it's going to be less accurate. And again, less accurate, not inaccurate, but less accurate for individual cases like hospitals. And we've tried a number of analogies over the years. The one I'm trying today is a body mass index. Most people here are familiar with the notion of a body mass index. And generally if you have a lower BMI, you're going to have lower risk for CVD and other outcomes. And so we know that US Preventative Services Task Force and there are other best practices that recommend screening for CVD and other risk based on your BMI.

But it's omitting important data, your age, your smoking status, your cholesterol level. I remember when I met with my PCP about 15 years ago, he said, "All right, here's your BMI. Here's your risk of cardiac outcome in the next five years." That's nothing. I can live with that. He says, "Now, if you come back 10 years from now and look like the same thing, here's your risk of a cardiac event." I said, "All right, I guess I better do something." So that's a great example right there. BMI is a unidimensional. With age, all of a sudden we had a very different assessment of what my risk is in the next five years. Saquon Barkley is a NFL running back. He and I have roughly similar BMI. One of us is in much better shape. One of us is going to be at much higher risk for CVD and other outcomes. Again, another example of that. And if I found, I'm sure there's a smoker out there with a BMI that's lower than mine, but has high cholesterol, has high blood pressure, probably not healthier than me, almost certainly going to have higher risk for CVD as outcomes. So FDI is one measure that can be useful for prediction, but only think of as a screen concept.

Second, we say at risk. We have these four categories, highest, mid-highest, mid-lowest, and lowest. But what does at risk for distress mean? And I don't know what it means to be at risk for something. I define it as non-zero. I am at risk for a meteor to come crashing through this ceiling, especially how weak this building is in the next two seconds. Didn't happen. It was a non-zero risk. It was highly unlikely, but it was non-zero.

But other people have a different concept of what at risk means. Is it 5%, more than 50%? And there are a variety of other rural hospital distress indexes that are using different thresholds. And I'll have a slide with examples of that, but none of us are right and none of us are wrong. And the picture on the right here is a pretty clever thing that was done a few years ago. And I don't remember exactly what they did, but it was something like these are words, would these be adverbs I think? I think these are adverbs that are used in Dutch news stories. Always certain, almost certain, almost always. And they took these adverbs and they asked people, lay people, I think it's 650 people or something like that, what does almost mean? What does always mean? What percent would you ascribe to that? And there's a couple takeaways here, but first of all, I would say that looking at this in always, the average probability if something is always happening as assessed by people is 96%. So as a statistician, I'm going to start there and I have some questions.

But look at the wide variety on some of these words and how wide the spread is and what people think about when it comes to that. I blow a section of this up and here are some things that probably kind of mean at risk, maybe, I don't know. Probable, possible, maybe, uncertain, chance, liable to happen. All of these are sort of measuring something that means could happen. But I think, no, I know that when we talk about at risk for distress, there's going to be a wide variety of what people think. And look at all this. Basically the most common answer for all of these between possible, maybe uncertain chance liable to happen is right there at 50. And the opportunity I didn't have here that I'm kicking myself, but I'll use in the next version is that classic Stepbrothers episode with, "You're saying there's a chance."

And in that meme, "You're saying there's a chance." That was really like a 0.1%, but people hear chance and think 50%. So at risk for distress means a lot of different things to different people. It definitely does not mean is going to close, but it means there's some chance between zero and between 100.

So here's an example. You can see the wide variety of what at risk means. Vulnerable to closure, at risk of closure, a third of rural hospitals, the highest, here's our percentage of the map where we show our assessment of the risk of closure. And I did not cite any particular indices here because that's not the point. The point isn't to say that these guys say there's more, these guys say there's less. What the point is, is that I don't know what the correct metric is here. Generally, I probably would say that every hospital's at risk for closure, but I think in some respect, if we think about those probabilities that we saw in the previous slide, anything that looks like 2,100 hospitals are at risk of closure is just ridiculous. So we need to better understand when we're talking about at risk for distress, what does that mean? And it's really hard to compare apples when you have different indices using different cut points.

Third comment, it's not a statement about management quality or the system's commitment to the hospital or, or, or. Hospitals at high risk may survive for years and there are plenty of examples that since we started this in 2017 of rural hospitals that have been highest risk for 10 straight years and they're still here. There may be an atypical financial model and there's at least one hospital that's basically financed by an extremely large endowment of a certain private equity. And so their operating margin is like negative 10, negative 15% each year, but they slide that endowment over income over and they're doing just fine.

It may be that a high distress hospital is supported by a system for strategic purposes. Maybe for example, to have a beachhead and say, "Well, we're going to lose money at this particular rural location, but with that spot there, it helps us with referrals for hips and cardiac at the flagship facility." Maybe supported by local government, cost report or other data may be inaccurate. When we say that we use cost report data, you can tell the people who are familiar with cost report data and they go, "Oh." I mean, cost report data does have challenges. It's two years lagged often, but it's not bad usually depending on which parts you're looking at. And so the degree to which those data are inaccurate are going to influence the predictive power of the model. And generally, we're more concerned with hospitals on a trend of increasing risk than those that are at perpetually high risk.

You need a human in the loop. And if we think about as we're all building our AI agents, we probably don't want to hit go and let them run amuck and start changing things on your server of your institution. It's great for a great first draft, but you need to go behind and verify it. It's the same idea with FDI. If you are someone who's trying to look at hospitals in your state and figure out, where are my best resources best spent? It's a good first start, but then you go from there and work there and think about other sources of information. Well, these guys are high risk, but I know that they're highly supported by the system. They're never going to close. These guys are reported as mid-low, but these data are two years ago and the plant just closed six months ago and I know that they've been texting us to say they're having trouble meeting payroll. That's human in the loop that needs to be used to be updated, that FDI account.

And I think it's really important to use these responsibly for all the reasons that we've said before. These are not a definitive diagnosis. And so anytime you're using the FDI or any index like this, you need to recognize that anything released to the community could have a self-fulfilling cycle. And saying, "Elm Valley Hospital is in financial distress." Is really hard because the community says, "Well, geez, I guess we shouldn't go there anymore." And it may lead to that. So these FDIs are really important for policymakers and practitioners, particularly on the government side, internal planning purposes, anything that's designed for those interventions and preparing. And they are extremely sensitive and those who are using it need to be recognizing the power and what can happen with irresponsible use.

All right, conclusion. Three bullets. So here's the comic all put together. FDI has some very fancy things and we think it is really nice, but you need to recognize the job that you're using it for and recognize its limitations.

Okay, so key takeaways. I think we've covered all these rather well. I'm not going to read the list, but just really talking here about using them as a point of information. It is an easy assessment. It's like a traffic light, but you need to understand, well, traffic light's probably not a good one, but it's a good screen to let you look at it and get a quick sense of at a high level, how do we think this hospital's doing? But really you need to go behind it and think about, get that human in the loop. What other information do we have? What else can we use behind it? And thinking about what this “at-risk” notion means.

Here's my contact information, and all the things that you have, if you want to pull out your QR code. The standard disclaimer. And I also wanted to recognize how many people have contributed to this over the years. There's a long list there ranged alphabetical by last, but a number of people have updated it or revised it or published on it or refined it. And it really is a brainchild of multiple people and some of whom have left our center and some of whom are still around.

For rural health research like this, the Rural Health Research Gateway is a great resource for you to go and check out. If you're not familiar with that, give it a look up. And there's tons of information about rural health research and policy. And I think that concludes today's presentation. Thank you very much.

Kristine Sande: We do have a few questions that have been entered while you all were speaking. So we'll start with those. There's a few here for Sara to get started. The first question is, "Can you look at hospital closure status by percent of hospitals within a state?"

Sara J. Couture: Right now, the dashboard is not designed to do that, but I think that's a very interesting enhancement to put in our backlog as we're updating the dashboard.

Kristine Sande: All right. And the next question is, "This is great facility data. How do you see policymakers using the information?"

Sara J. Couture: Yeah, so we've already been in conversations with CMS. The Rural Hospital Fund as part of the One Big Beautiful Bill has already been discussed as this is a potential tool as they're considering ways of targeting funding as part of that initiative, as well as a few other initiatives by the White House.

Kristine Sande: All right. Next question is, "This is a great tool. Why are there a few states with no data on the dashboard such as New Jersey and Connecticut?"

Sara J. Couture: Yeah, so the definition of rural hospital was using USDA's urban rural county continuum codes, and they defined it as four to six on those code scales. So for New Jersey, there's two counties in New Jersey that are in there, but they either didn't have a hospital that we would consider a rural hospital in there. And then Connecticut's an interesting case because the way that their counties are defined also is not super aligned with those codes. So I was going to flag that for our team as well as like, we should make sure we're not missing any counties in Connecticut, but the counties that are in there are urban counties.

Mark Holmes: And Kristine, I'll just add that we use a slightly different definition. And so this I think is important to keep in mind as you're looking at these different tools. We're using the HRSA or federal office definition plus CAHs, as I mentioned. And so there's just different ways to approach it and it's just useful to look at it. And so that would mean that we have more hospitals if I'm doing that math quickly in my head than ASPE.

Kristine Sande: Yeah. And that's a great reminder. Anytime you're looking at a rural tool or a rural stat, it's important to ask what definition is being used here. So now there's a couple of questions for Mark. "So is the FDI a tool that can give specific hospitals at risk in each state?"

Mark Holmes: So great question. And this comes back to point number five, I think, in that we generally like to hold that data as close as possible. Hospitals can see their own and they can look at trends. Critical access hospitals can look at it using a tool called CAHMPAS, C-A-H-M-P-A-S, that hopefully critical access hospitals are aware of. That's a portal that's funded by the federal office through the Flex monitoring team. We provide the hospital level specific data to the State Offices of Rural Health in each state. So Connecticut SORH can see the distress for the rural hospitals in Connecticut, but not anywhere else. So that's the degree to which those data are shared.

Kristine Sande: So Mark, "Does the FDI find the same predictors such as system affiliation, continuous urban hospitals, et cetera, as the dashboard research?"

Mark Holmes: No, it does not. So again, that's part of the fun of doing things like this. Is there an example of, again, Sara, help me out. It wasn't contiguous. It was next to, proximate.

Sara J. Couture: Adjacent to an urban county.

Mark Holmes: Adjacent. Yes, thank you. Adjacent to an urban county is the one that I remembered that they have that we don't. And what? We have Medicare Advantage, I think is an example of something we have that they don't. So again, just different perspectives. I wonder, certainly we know that profitability is carrying and other financials are carrying a lot of the heavy lift. Sara, do you know, certainly there's lots of P values there, most of those were significant. Are you able to speculate or remember in terms of the predictive power? Are your measures of financial performance really carrying the bulk of that prediction as well?

Sara J. Couture: Occupancy rate I think was our top one, profit margin, and liability assets. We're also pretty significant, but occupancy rate was a pretty significant one for us.

Kristine Sande: This person says, "I was trying out the ASPE tool when Sara was doing the demonstration. How are the county data located?"

Sara J. Couture: So for the county information, we use it based off the hospital cost reports and then we link it with a variety of different types of county data to include stuff from the American Hospital Association data as well as some of the county rank files. I can provide a link of the full data sources in the description, if I'm understanding that correctly. We also did geocode all of our hospitals.

Kristine Sande: All right. And another question is, "What role does the number of outpatient service lines play in impacting financial status or risk?"

Sara J. Couture: I'd have to turn back on my team on that particular one. I was less involved in the development of the Cox model myself. So if anyone on my team wanted to answer that, feel free to.

Mark Holmes: But I think the other thing I'll say is Brian Whitacre out of Oklahoma State has done modeling as well as some teams from ETSU, East Tennessee State, looking at specifically this history. There's another example, this was presented at National Rural Health Association in May, I think, that really underscores another example of variables that other people are considering that we look at and go, "Wow, that's pretty clever. We should think about that as well." So we don't have outpatient service lines in our model today, but do think that that's something we want to think about for the future.

Kristine Sande: So on behalf of RHIhub, I'd like to thank our speakers for the valuable information and insights you've shared with us today. And thanks also to our participants for joining us.

The slides that are used in today's webinar are currently available at www.ruralhealthinfo.org/webinars. So thanks again for joining us and have a great day, everyone.