Home » Posts tagged 'Metrics'
Tag Archives: Metrics
1. The Streetlight Effect and Measuring What Matters
It was dark and a man lost his keys. He searches for them under a streetlight, and a friend comes over to help. Eventually, the friend asks, “Are you sure you lost your keys here?” The man says, “No. I lost them in the park.” So the friend asks, “Why are we looking here?” The man answers, “Because this is where the light is.”
This story describes a form of observational bias called the streetlight effect, in which we look for things where it’s easy, not where it’s important.
“I would argue that we do this all the time in health care measurement,” says Ari Robicsek, Chief Medical Analytics Officer for Providence St. Joseph Health.
Every hospital measures length of stay, for example, and many use this measure as a surrogate catch-all for quality and efficiency. “But let me ask, who really cares about length of stay?” says Robicsek. “Is it patients? Is that the first metric that comes to mind when a patient is thinking about hospital quality? Is it doctors? Probably not. Is it administrators? Even from an administrative point of view, you’re not going to realize the financial benefit of reduced length of stay, unless at the same time you reduce labor, or you find ways to fill those empty beds with paying customers, which is a much more complex measure than simply looking at length of stay.”
Length of stay may not be a great measure, but if we have to start somewhere with health care measurement, what’s the harm in tracking it? “If we assign resources to working on the wrong problem, those resources aren’t working on the right problem,” warns Robicsek.
Additionally, with length of stay specifically, a big push to get patients out the door risks sending them home before they’re ready — and when that happens, those patients may end up with complications and get readmitted. “We see one streetlight metric, length of stay, giving birth to another streetlight metric, 30-day readmissions, and so on,” he says.
“My modest proposal: We should measure the things that matter,” says Robicsek. “Yes, sometimes that’s going to mean that we need to collect data differently than the way we do today, or, said another way, sometimes we’re going to have to put up some lights in the park.”
2. Balancing Risk Adjustment
Robicsek shares a map showing distribution of glycemic control in diabetic patients on Chicago’s North Shore, where green is good and red is bad. The map overlays closely with an income map.
If we set up a bonus program for primary care doctors where they receive more money if their patients have better glycemic control, it’s easy to guess where most physicians will want to practice. This is why we need risk adjustment.
“Absent good risk adjustment, physicians working in disadvantage geographies are going to have the worst-looking outcomes,” Robicsek explains. “They’re going to get paid less. The poor get poorer, etc. Absent good risk adjustment, physicians are going to have an incentive to cherry-pick, that is, focus on the patients who are going to make them look good.”
“But with good risk adjustment, we have the opportunity to identify those providers who are outperforming expectations, who are doing a great job with the difficult-to-manage patients, and we can learn from them.”
There are disadvantages to risk adjustment, however, when done poorly. The most common problem is doing little more than creating the illusion that risk adjustment has occurred. “A lot of the risk adjustment models in use are lousy, including some of the ones used by CMS (Centers for Medicare and Medicaid Services),” Robicsek says. “I would argue that those do very little other than creating the patina of fairness, and I would argue when that happens, we’ve probably done more harm than good with risk adjustment.”
Another concern is that sometimes risk adjustment can justify outcome disparities that are amenable to management. A blood-pressure management metric risk-adjusted on race, for example, could remove the incentive for physicians to determine how to manage blood pressure in African-American patients, perversely promoting or entrenching existing inequalities.
“My proposal here: For every new measure that we build, we need to have a conversation about what amount of risk adjustment is enough,” says Robicsek.
3. Measuring to Learn
How much is enough? When we can learn from the measure, he explains. “So much of the health care measurement that we do is for the purpose of rank-ordering or some form of reward or punishment. I would argue that most of the measurement that we do should be taking into account the fact that, as humans, we’re curious and we’re altruistic — most of it should be to learn.”
In a graph of total knee replacement at Providence St. Joseph Health, each circle represents one high-volume orthopedic surgeon. Each of these surgeons performs high volumes of elective primary unilateral total knee replacement, and they all have great outcomes. But the difference between them is cost.
Every circle above the line represents a surgeon whose cost per case is high. Every circle below represents a surgeon who cost per case is low relative to their colleagues.
Are the doctors low on implant costs consistently low across other elements of care? Not necessarily. “In medicine, variation is the state of nature,” says Robicsek. “There are almost no clinicians who are consistently high cost, or consistently low cost across these elements of care. My takeaway from this is that we all have an opportunity to learn from each other.”
“My modest proposal: Most of the health care measurement that we do should not be for reward or punishment. It should be to learn.”
4. Whose Patient Is It?
Providence recently evaluated OPPE (Ongoing Professional Practice Evaluation), which the health system uses, and found that it was assigning 40% of hospital patients to the wrong doctor. “Who can blame them?” Robicsek asks. “It’s easy in a hospitalization for a patient to have three different, or five different, attendings of record. How do you know who to assign that patient to?”
“In a world where we’re measuring for reward and punishment, we feel obligated to assign one outcome or one hospitalization to a single clinician, but imagine if we were able to move away from that and we were measuring to learn,” he says. “Then we would have the ability to do things like ignore who the provider was and ask ourselves what specific elements of care, what specific combinations of behaviors, lead to the best outcomes.”
“Or we could recognize that medicine is a team sport,” he adds. “Let’s ask the question, can we tie outcomes to teams rather than to individuals? My modest proposal here: We practice in teams. Let’s recognize that in the way we measure.”
5. Metrics Aren’t Free
“To anyone who has ever said, ‘Let’s just add one more thing to this dashboard’: Metrics are not free.”
“Every time we build the metric, if it is done correctly, somebody needs to build business specs, technical specs. Someone needs to do data governance, coding. Somebody needs to do validation, automation, documentation, visualization, and then somebody needs to maintain the thing moving forward. Easily that’s a cost of $10,000,” says Robicsek.
6. The “Give a Darn” Test for Health Care Measurement
When measuring what matters, how do we know what that is? Robicsek describes a thought experiment where he sits with a small group of physicians considering a metric. “Imagine I told you that you’re doing better than your colleague on this measure,” he says to them. “Would you feel good about yourself? Imagine I told you you’re doing worse than your colleagues on this measure. Would you feel motivated to change your practice?”
“If the answer to both of those questions is not yes, let’s not build this measure. It’s not worth our time. We’ll go focus on something else.”
“Sitting at the front of this room is my partner in crime, Dr. Caleb Stowell, looking like the cat who ate the canary. He’s showing [the surgeons] the results of the process that I’ve described. They’re measuring to learn. He’s identified a measure that passes the ‘give a darn’ test for them, and some of those surgeons are literally leaning in. I work for 51-hospital system, but where this change happens, where you win hearts and minds, is in rooms like this.”
Robicsek’s final proposal: Try the “give a darn” test for health care measurement. And note that in many “give-a-darn” conversations, one metric that comes up as incredibly important to physicians is patient-reported outcomes.
From the NEJM Catalyst event Provider-Driven Data Analytics to Improve Outcomes, held at Cedars-Sinai Medical Center, January 31, 2019.
Carly Barry 27 February, 2013
While value stream mapping, or VSM, is a key tool used in many Lean Six Sigma projects for manufacturing, it’s also widely used in healthcare.
Value stream mapping can help you map, visualize, and understand the flow of patients, materials (e.g., bags of screened blood or plasma), and information. The “value stream” is all of the actions required to complete a particular process, and the goal of VSM is to identify improvements that can be made to reduce waste (e.g., patient wait times).
How is VSM applied to healthcare?
When used within healthcare, one obvious application for VSM is mapping a patient’s path to treatment to improve service and minimize delays.
To accurately map a system, obtaining high-quality, reliable data about the flow of information and the time a patient spends at or between steps is key. Accurately timing process steps and using multi-departmental teams is essential to obtain a true picture of what’s going on.
To map a patient’s path to treatment, a current state map can be created in a VSM tool (we offer a powerful one in Companion by Minitab) to act as a baseline and to identify areas for improvement:
In this example, the first step a patient takes is to visit his general physician (abbrev. “GP” above), and this is represented as a rectangular process shape in the VSM. The time the patient spends at this step can be broken down into value-added (“VA”) and non value-added (“NVA”) cycle times. VA is time the customer is willing to pay for: that is, the 20 minutes spent consulting with the GP. NVA is the time the customer is not willing to pay for, i.e., the 20 minutes spent in the waiting room before the appointment.
The dotted line arrow between process steps is called a push arrow. This shows that once a patient completes a step, they are “pushed” to the next step. This is inefficient, and a more efficient process can be designed by changing push steps to continuous flow or “pull” steps. The yellow triangles indicate the time a patient spends waiting for the next process. These steps are a non-value added action for the patient.
While VSM can certainly be done by-hand on paper, using computer-based tools like those in Quality Companion makes the process a lot easier. For example, Quality Companion automatically calculates and displays a timeline underneath the VSM, which adds up the total time to go through the entire system (aka “lead time”) and displays summary information.
By identifying all of the steps, you can start to map the whole process out, moving from left to right. Once you have mapped out the entire system, an ideal future state map can be created, and possibly a series of future states in between. These can identify areas for improvement, and once implemented, they can become the “new” current state map as part of an iterative quality improvement process.
How do you improve the current state map?
When looking for areas of improvement, try to focus on changes to improve the flow of patients through the process. Continuous flow is the ideal and moves patients through the system without them having to wait. However, continuous flow is not always possible, so instead other changes might be introduced—such as first-in first-out (FIFO).
Also be sure to take a look at the takt time, which can help you decipher the pace of customer demand. In this case, takt time can be interpreted as the number of patients that can be treated per unit of time. Quality Companion will calculate takt time automatically.
Once you have completed the current and future state maps, you can compare the two, quantify improvement opportunities, and look at how to implement the changes. In this example, the triage and sort/appointment steps might be combined so that fewer visits to the hospital were required by the patient and they receive treatment faster.
To see another example value stream mapping, check out this video that features a scenario from Companion’s extensive help system:
Learn more about lean six sigma in healthcare : Six Sigma Master Class – Improving Healthcare Processes
If winning isn’t everything, why do they keep score?
– Vince Lombardi
The quote above embodies the purest notion about the nature of work. There is a scale by which we measure the success of our endeavors and someone is always keeping score. For managers, this means there is a scale by which the consequences of our decisions and actions are measured. Our hope is that the results of our efforts are favorable for those who entrust us with decision-making authority. In a profit seeking organization, the shareholders, employees and customers measure the outcomes of the decisions we make and the actions we take. In government, this task is given to the voting public. For non-profits, managers are brought to account by fundraisers, contributors and the beneficiaries of the organization’s services.
In our world of healthcare, the ultimate trust is that which exists between our patients and us. When they require care, they rely on us to provide the right diagnosis, early and timely treatment, and the best conditions for a complete recovery. And yes, they too are keeping score.
This writer can think of at least 100 reasons why a healthcare leadership team should design and deploy a performance measurement system. For the goals of this article, we will concentrate on the most important reasons. First, a healthcare leadership team should deploy a measurement system to reinforce the organization’s mission. A common joke about measurement systems is, “If we didn’t measure things, we wouldn’t know how good we are at measuring the things we are measuring!” Yes, a measurement system should have a purpose. Secondly, a healthcare leadership team should deploy a measurement system to promote process perfection and a reduction of medical errors. A study recently conducted to identify medical errors asserts that as many as 90% of hospital mistakes go overlooked. Finally, a good healthcare measurement system will associate patient outcomes with the performance and protocols of the system’s experts. Accuracy, accountability and purpose are three essential characteristics of a healthy performance measurement system, but the number one reason a healthcare organization should deploy quality metrics can be summed in one word = PRODUCTIVITY!
Now, more than ever, the healthcare industry needs to embrace the economic value proposition of improving productivity. For the past 20 years, the industry has experienced negative productivity growth. The economic consequences of this type of industry performance are stunning. U.S. health care costs currently exceed 17% of GDP and continue to rise. A PricewaterhouseCoopers report projects that health care costs will increase 7.5 percent in 2013. That is more than three times the rate of inflation and the projected rate of US economic growth. That same report also notes that health insurance premiums are expected to rise 5.5 percent, in large part because employers are shifting costs to their employees. Medicare’s Office of the Actuary forecasts that health care spending will jump to more than 7 percent in 2014. At the same time, healthcare providers will face unprecedented cuts in reimbursement rates from Medicare and other third party payors. The bottom line is that until true health care cost reform becomes a reality, these pressures will continue to cause problems for providers, for people’s health care and for the nation’s economy. Healthcare organizations should use these pressures as motivation to embark upon a relentless pursuit of ever-increasing productivity.
Why is pursuing productivity so important to a healthcare delivery organization?
Improving productivity helps providers stay in business and grow. Improved productivity is an organizational competence that insulates healthcare organizations from destructive actions like avoid across-the-board cuts in expensive services, staff compensation, and head count. In fact, a nationwide improvement in productivity among healthcare systems can reverse the negative economic consequences forecasted on the horizon.
Economists Mark Whitehouse and Tim Aeppel aptly describe the economic value of productivity growth in a November 3, 2006 Wall Street Journal Article:
“Productivity matters for everyone, because it provides the essential ingredient that makes nations rich! When companies produce more for each hour their employees work, they can pay higher wages or reap bigger profits without having to raise prices. Annual productivity growth of 2% would more than double inflation-adjusted wages over 40 years, all else being equal. Add another percentage point in productivity growth, and wages would more than triple!”
What is Productivity?
Essentially, productivity is defined as output per unit of input. As illustrated in the figure below , productivity is the difference between the value produced by an organization and the cost of the basic resources brought to bear in a productive process.
From the perspective of the patient, the ultimate measure of productivity (patient value) is the full set of favorable health outcomes over the cycle of care divided by the total cost of care of the patient’s condition.
A healthcare organization can increase productivity by doing the following:
- Increase the number of patients served (the numerator) while keeping the cost of the inputs (denominator) fixed.
- Keep the number of patients served constant while decreasing the cost of clinical and administrative processes.
- Accomplish a combination of the two examples above.
Productivity and Quality in Healthcare Delivery Organizations
Harvard Professor, Michael Porter argues that quality and performance improvement are key drivers of cost containment and higher value, where quality is health outcomes. Stated another way, the drive for better quality of health outcomes and the drive for increased productivity are not mutually exclusive. In fact, poor quality is not only poor for healthcare outcomes but also creates a drag on productivity.
The figure below illustrates how poor quality can cause problems for productivity. It describes the generic equation of productivity and adds in certain non-productive outcomes organizations typically produce. Along with salable goods and service transactions, organizations produce defects, errors, rework, customer credits, fines, accidents, lawsuits etc… These outcomes are work products of what six sigma black belts call the hidden factory; and the costs associated with them are referred to as the Cost of Poor Quality (COPQ). They reflect the failures and dysfunctions of organizations and rob industries of the productive use of human and financial capital.
In healthcare systems, the hidden factory can be found in both administrative and clinical areas. A recent study found that medical errors cost Medicare more than $324 million per month. A USA Today article reported that 80% of medical bills are inaccurate and a 2009 study conducted by University of Minnesota health finance professor Stephen Parente found up to 40% of hospital insurance claim statements contain errors. To make matters worse, many patients are paying a heavy price for the hidden factory. One out of every three people encounter an adverse event when admitted to a hospital, according to a recent study (4/6/2011) published in the health policy journal Health Affairs. The study also found that the hidden factory is, indeed, hidden – about 90 percent of all hospital mistakes go unreported.
The cost of poor quality not only adversely affects patient outcomes (numerator) but also provider costs (denominator). The High Value Healthcare Collaborative, an organization consisting of 20 major Hospital Groups who serve 70M people, estimate that the cost of the hidden factory is more than 30% of all healthcare delivery cost. These costs create a significant drag on productivity.
Why do we measure? We all know that what gets measured gets improved. Healthcare leaders should deploy a system of quality and performance metrics to control and improve overall healthcare quality.
The figure below is an adaptation of the Deming Chain Reaction and illustrates the affect that improving quality has on productivity. It states that when provider organizations improve quality, their costs go down. Their costs go down to the tune of 20 – 40% of total operating expenses. The decrease occurs because the costs of wasted effort reworking problems, correcting medical errors, reassuring dissatisfied patients, and reconciling invoices are eliminated. As these costs go down, productivity naturally improves. Productivity improves because of the increased use of human capital, technology and working capital in producing favorable patient outcomes.
Better patient outcomes and the termination of the hidden factory lead to greater profit margins and enhanced economic value. The additional economic value funds growth and innovation, which leads to improved healthcare quality and high value jobs.
Quality and Performance Metrics – Best Practices
When designing a system of quality and performance metrics, an organization should follow seven proven practices:
- The metrics should be defined and understood by all within the organization. Operational definitions help clearly define what is being measured. There are three basic elements of an operational definition: the measure, the instrument being used, and the procedure for measuring.
- Performance metrics should be strategically integrated for tracking daily operations. Good performance metrics are engineered based on cause and effect relationships. In the world of statistics this is called engineering an explanatory response distinction; where the extent of an outcome is leveraged by the degree of influence a decision maker’s actions has on organizational processes.
- Data derived from measurement systems contribute to operational and strategic decision-making. This practice supports the principle of management-by-fact where decision makers not only understand the activities and work products of their organizational processes, but they also track performance records over time and keep numerical facts for analysis and decision-making.
- Metrics should be compared with industry ratios and benchmarks. Comparisons should represent best practices for similar activities, inside or outside the health care industry. Such data might be derived from surveys, published and public studies, participation in indicator programs or other sources.
- Performance results should be trended over time and communicated cross functionally and vertically within the organization.
- Measurements should align with organizational critical success factors and strategic plans. Doing so empowers leadership to review progress and assess organizational performance relative to strategic objectives and action plans.
- Metrics should ultimately impact a financial component. Healthcare delivery organizations are not immune from the virtues or vices of capitalism. Whether a provider is a not-for-profit organization or is held by a private sector company, it must create economic value to sustain itself.
Quality and Performance Metrics an Evaluation
Quality and performance metrics should collect an analyze data on three essential perspectives: operational excellence, value proposition, and economic value.
Operational excellence is defined as the extent to which the core administrative and clinical functions are managed efficiently. Operational excellence is a key driver of patient outcomes and economic value. Appropriate metrics should evaluate the extent to which the professional expertise, technology and protocols deliver care productively. Metrics should track throughput, cycle times, cost and defect levels.
Healthcare delivery organizations primarily extend value to patients, their families and third party payors. Metrics should evaluate the quality of healthcare outcomes and the degree to which the outcomes are favorable or unfavorable over the care cycle. They should also measure the extent to which the needs, attitudes and perceptions of the patient and their families have been served (patient satisfaction). Performance metrics should also measure the extent to which administrative functions satisfy payor requirements.
Quality and performance metrics should evaluate the extent to which healthcare delivery operations adequately provide financial surpluses or profits to sustain its operations and fund long-term growth. Performance metrics should include measures of financial return, financial viability and budget performance.
Quality and Performance Metrics – Examples
Poudre Valley Health System (PVHS) is a 2008 recipient of the Malcolm Baldrige National Quality Award. The health system is a locally owned, private, not-for- profit provider to residents of northern Colorado, Nebraska, and Wyoming. PVHS has an abundance of performance measures; this article provides three examples for your consideration.
Example – Operational Excellence Metric
For efficiency, PVHS’s measures OR turnaround times and measures on time first case starts as a metric for effective resource utilization.
Example – Value Proposition Metric
As the best key performance indicator of cardiac outcomes, PVHS focuses on acute myocardial infarction (AMI, heart attack) patient mortality rates for six months after hospitalization. This metric is also viewed as a key measurement of outpatient care provided after discharge.
Example – Economic Value Metric
As a key performance indicator of economic value, PVHS monitors profit per discharge. Profit per discharge is a critical success factor, which enables PVHS to maintain financial sustainability. The organization monitors profit per discharge to ensure the viability and sufficiency of investments for the future.
PVHS uses a comprehensive system of strategic and operational metrics by which to analyze its operations and progress towards its strategic objectives. Does measurement work for healthcare? The market and community it serves has many requirements, among them being serviceability, high quality care and low cost. PVHS maintains its industry position as a low-cost provider in its market. Since 2001, the systems charges have been consistently lower than the competition while its profit per discharge to surpassed the U.S. top 10 percent. If you are keeping score, then you understand that PVHS is winning.
Gerald Taylor LSSMBB PMBscM is a Managing Director for TPMG Consulting, a recognized leader in performance improvement consulting He can be reached at: www.helpingmakeithappen.com
At TPMG Consulting, we have developed an effective approach to Performance Analytics. It’s called FOCUS®. The approach is built on one basic principle: concentrate on improving those activities that enhance patient satisfaction, improve healthcare outcomes and reduce the cost of care. To learn more about establishing a system of Key Performance Indicators in Healthcare, Click: TPMG Healthcare Key Performance Indicators and Dashboards