Innumeracy at the Centers for Medicare & Medicaid Services (CMS)

A physician participant in an American Medical Informatics Association listserv I subscribe to had some time ago commented that the CMS, in their requirement that physicians meet certain percentages on measures regarding adherence to the "Meaningful Use" incentive program for EHRs, were calculating those percentages incorrectly.

He followed up with an actual response from CMS, and has given me permission to repost.  I am redacting his name:

... I have posted previously that the CMS calculator found at http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Stage_2.html appears to calculate improperly.  Last night I received a response from CMS regarding my concern:

“Dear Mr. (sic) [name redacted],

Thank you for your recent inquiry. Below is the information you requested about the attestation calculator.

As adopted by CMS, measures generally require that providers achieve "more than" a specific percentage. The language in these measures is clear; "more than 50 percent" would equate to 51 percent or more. This has consistently been the policy from the beginning of the program.

The attestation calculator now rounds fractions of a percent to the closest whole number. This correctly identifies when the required performance level is reached.

The tracking number for this inquiry is 2014-XXXXX. . . . “

My examples included a numerator of 549 with a denominator of 10,000 for a metric that required “greater than 5%”.  These are screen shots of the submission:

Note that the calculator appears to use 3 significant figures, but in fact rejects the calculation by rounding to only 1 significant figure.  

Years ago at a CMS conference I remember the speaker remarking that physicians and hospitals only had to enter the numbers “and we could agree on the arithmetic.”  I learned in fourth grade how to round numbers, and how to count significant digits.  I do not agree with their arithmetic, nor do I accept their determination that “more than 50 percent" would equate to 51 percent or more”.  Granted, one hopes that those attempting to attest will not have numbers that skim so closely to the metric limit, but it could happen.

I note in the reply from CMS this statement:

"The language in these measures is clear; 'more than 50 percent' would equate to 51 percent or more."

I emphasize:   

THE - LANGUAGE - IN - THESE - MEASURES - IS - CLEAR.

Here are  actual instructions for, say, CPOE measurement on medication ordering (http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/downloads/Stage2_EPCore_1_CPOE_MedicationOrders.pdf):

Attestation Requirements
DENOMINATOR/NUMERATOR/THRESHOLD/EXCLUSION

Measure 1: Medication

DENOMINATOR: Number of medication orders created by the EP during the EHR reporting period.

NUMERATOR: The number of orders in the denominator recorded using CPOE.

THRESHOLD: The resulting percentage must be more than 60 percent in order for an EP to meet this measure.

EXCLUSION: Any EP who writes fewer than 100 medication orders during the HER (sic) reporting period.

Now, I had a perfect "800" in math on my SAT test in 1974, so (while probably excluding me from some employment in today's culture of stupid due to having "too much experience"), I do believe I am perfectly well qualified to determine that "more than 60 percent" would mean, anything more than 60.0, such as, say, 60.1%.

Let's consider possible explanations.

Possibility (1)
Perhaps CMS is performing their calculations with an old Pentium-1 computer whose CPU has the FDIV bug causing math errors (http://en.wikipedia.org/wiki/Pentium_FDIV_bug)?

On second thought:

I don't think that is a good explanation.

Possibility (2)
Perhaps I have become demented from too much exposure to the numeracy-challenged since 1974?

An experiment is in order:

In 1969, we landed men on the Moon.  It took a lot of math to perform that feat, and miscalculation would have led to disaster, for example on the 25,000-MPH re-entry into Earth's atmosphere at precisely the correct angle required to avoid burn up, or skipping off the atmosphere like a stone off a pond, back out into space to be lost forever.

I learned computing about that time on a Digital Equipment Corporation PDP-8/S, a computer from that era (http://cci.drexel.edu/faculty/ssilverstein/medinformatic1/gw-pdp8s.htm).

I wrote and ran a small program on a PDP-8 emulator WinEight (available free at http://www.sparetimegizmos.com/Software/WinEight.htm) that perfectly emulates the old PDP-8.  Here is the program in the programming language FOCAL, 1969 version:

CONGRATULATIONS!!
YOU HAVE SUCCESSFULLY LOADED 'FOCAL,1969' ON A PDP-8 COMPUTER.

SHALL I RETAIN LOG, EXP, ATN ?:YES

PROCEED.

*1.1 SET P=(601/1000)*100
*1.2 IF (P-60) 1.3, 1.4, 1.5
*1.3 TYPE "P IS LESS THAN 60%";G 1.6
*1.4 TYPE "P EQUALS 60%";G 1.6
*1.5 TYPE "P IS MORE THAN 60%"
*1.6 TYPE !;QUIT

The program sets variable "P" to 601/1000, then multiplies by 100 to derive a percent.  It then compares the result stored in "P" to 60, and branches to the correct statement regarding the relation of P to 60%.  Let's run the program:

*
*GO
P IS MORE THAN 60%
*

Good news.

I guess I am not demented after all.

Possibility (3)
Someone (or a group of people) at CMS doesn't or don't realize that in large datasets, percentages are often not integer whole numbers.

Possibility #3 can be illustrated best by a cartoon:

Click to enlarge.  1+3 = 13, right?

Or perhaps this:

Math for Underachievers.  Click to enlarge.

I leave it to the reader to determine which of the three possibilities is the explanation for the assertion by CMS that "The language in these measures is clear; 'more than 50 percent' would equate to 51 percent or more."

Note: this post was meant as satire and humor and irony, but innumeracy in our healthcare regulators is anything but funny.

-- SS

EHRs and "Meaningful Use": Begging the Question in the New England Journal of Medicine

I used to have great respect for the New England Journal of Medicine.

No more.

That respect was reduced significantly in 2010 when I saw the following passage (as I wrote in my July 2010 post "Science or Politics? The New England Journal and The 'Meaningful Use Regulation for Electronic Health Records"):

... In the NEJM article "The 'Meaningful Use' Regulation for Electronic Health Records", David Blumenthal, M.D., M.P.P. (ONC Chair) and Marilyn Tavenner, R.N., M.H.A. (10.1056/NEJMp1006114, July 13, 2010) available at this link, the opening statement is (emphases mine):

The widespread use of electronic health records (EHRs) in the United States is inevitable. EHRs will improve caregivers’ decisions and patients’ outcomes. Once patients experience the benefits of this technology, they will demand nothing less from their providers. Hundreds of thousands of physicians have already seen these benefits in their clinical practice.

I think it fair to say those are grandiose statements and predictions presented with a tone of utmost certainty in one of the world's most respected scientific medical journals.

Even though it is a "perspectives" article, I once long ago learned that in writing in esteemed scientific journals of worldwide impact, statements of certainty were at best avoided, or if made should be exceptionally well referenced. I note the lack of footnotes showing the source(s) of these statements.

I also note the lack of mention of literature refuting or potentially refuting these statements of certainty. I can think of more than a few examples of the latter just off the top of my head [ref. 1-15 below, certainly not a comprehensive list but merely skimming the surface].

... So, did the NEJM publish fact, or political platitude?

Can someone provide a list of peer reviewed, rigorous studies that back the assertions of certainty in 10.1056/NEJMp1006114, and override the body of literature that could cast doubt on these assertions of certainty?

The negative if not bitter reader comments to yesterday's New York Times article on the health IT lobby (link) certainly cast some doubt.

My respect has dropped several more notches in 2013.

The NEJM has served as a PR journal for health IT once again.

In a correspondence piece "Early Results of the Meaningful Use Program for Electronic Health Records", N Engl J Med 2013; 368:779-780,  February 21, 2013 by Adam Wright, Ph.D., Stanislav Henkin, B.A., Joshua Feblowitz, M.S., Allison B. McCoy, Ph.D., David W. Bates, M.D. and Dean F. Sittig, Ph.D., the following statement is published (emphases mine):

... The downstream effects of meaningful use on quality, safety, and efficiency are not yet known, and further increases in EHR adoption, functionality for clinical decision support systems, and research are needed to ensure the effectiveness of the meaningful use program

I note that Begging the Question is a fallacy in which the premises include the claim that the conclusion is true or (directly or indirectly) assume that the conclusion is true.

Further, I made the following points in my Jan. 2010 post "Meaningfully Experimental Protocols and Interfaces to Nowhere? Nagging Questions On Healthcare IT Remain":

... there is a major problem with the term "meaningful use" itself:

This [term "meaningful use"] is an example of putting the cart before the horse, and is a semantically-based, self contained logical fallacy of sorts. If a health IT system is harmful, the term "meaningful use" is itself Orwellian. If we don't know if HIT is beneficial, or have doubts, then such as term presupposes that health IT is inherently beneficial. A better term would have been "good faith use" - use based on the faith or hope that health IT will have an overall positive effect. The term "meaningful use" jumps the gun and is more a political slogan than a "meaningful term."

I go further. Use of a term that a priori assumes some outcome reflects the antithesis of science. The term "meaningful use" in the domain of technology implies that those following the recipe for use of some technology, as well as their subjects, will experience meaningful outcomes. A parallel is in the logical fallacy of begging the question or circular argument, where the conclusion of an argument is among its premises.

In this current NEJM passage, the "question" of MU effectiveness is shamelessly begged like a hungry puppy begging for a Snausage doggie snack.

"Ensure the effectiveness?"  The inherent assumption is that MU (developed by consensus committee without supporting rigorous evidence of effectiveness), whose 'downstream effects' are admittedly unknown, WILL be effective - if only we spend MORE billions of dollars on the technology.

At best, the appropriate statement to have been made is this:

... The downstream effects of meaningful use on quality, safety, and efficiency are not yet known, and further increases in EHR adoption, functionality for clinical decision support systems, and research are needed to determine if meaningful use will have any positive impact on healthcare quality, safety and efficiency."

From a physician commenter overseas who excels in demolishing health IT propaganda: 

Seriously, did they proof read the last paragraph? How did the NEJM editors let that get printed? 

Given the recent food scandal in the UK, it is like the PM saying we are not quite sure that there's horsemeat in the Burgers and Lasagna so we are going to need to import a lot more of this muck from France, Poland & Romania before drawing any conclusions...

Shame on the New England Journal.  My respect for them is even lower than after the 2010 health IT advert as above.  This current letter's conclusion is a high school-level faux pas and, in my opinion, should not have been published in its present form.

-- SS

Addendum:  Let's see if they publish my Letter to the Editor on this matter, submitted today.

Second addendum:  a reader opined that MU could have a negative impact on quality, safety and efficiency (cf. yesterday's NYT article and reader comments, link).  While less likely, we just don't know.

Addendum Mar. 18, 2013. 

I received this today from the NEJM:

Your letter referring to the Wright article of 21-Feb-2013  has been received.  Because of the limited availability of space, we can publish only a fraction of the letters we receive.  Although we will not be able to print yours, we have forwarded a copy to the authors in case they wish to reply directly to you.

 
This is why I blog.

-- SS

Good HIT, Bad HIT, HIMSS And Reckless Technology Advocacy: Will This Hurt President Obama?

HIMSS, the Health Information Systems Management Society, is the large vendor trade group representing healthcare IT sellers.

At the HIMSS blog entitled "Health IT is an essential element to transform the Nation’s healthcare system" (link), Carla M Smith, MA, CNM, FHIMSS writes this with regard to the House  letter to HHS Secretary Sebelius asking her to suspend payments for the EHR Incentive Payments authorized in the American Recovery & Reinvestment Act of 2009:

HIMSS opposes halting the Meaningful Use EHR Incentive Program. Health IT is an essential, foundational element of any meaningful transformation of the Nation’s healthcare delivery system. 

(Of course, not mentioned is "transformed" into what, exactly; this utopian ideation is a topic for another time.)

A chart is then presented as to "how US civilian hospitals have, since the first incentive payments were made in second quarter, 2011, matured in their use of health IT."  Then this statement is made:

Healthcare providers are adopting certified EHRs and using them for meaningful purposes; thus, achieving Congressional intent to improve the quality, safety, and cost-effectiveness of care in U.S.

Really?  (See my Feb. 2012 post "Hospitals and Doctors Use Health IT at Their Own Risk - Even if Certified.")

This non-evidence based, amoral advocacy by HIMSS for health IT may cost the President the November election.  HIMSS has beguiled the president into similar unquestioning advocacy for the technology in its present form, which his opponents are now (rightfully) seizing upon as in the House letter.

The reckless mistakes made by HIMSS and their advocates include these two:

1.   The unquestioned belief that this expensive technology would save billions of dollars in healthcare costs, instead of depleting precious healthcare resources better spent on, say, improvement of healthcare services for the poor.

2.  More importantly, the appallingly naïve belief that any health IT is good health IT, and that any health IT is only capable of good, not bad.

HIMSS has thus failed to recognize - or perhaps worse, recognized but recklessly ignored - the profound difference between good health IT (GHIT) and bad health IT (BHIT).

As I defined at my teaching site on Medical Informatics:

Good Health IT ("GHIT") is defined as IT that provides a good user experience, enhances cognitive function, puts essential information as effortlessly as possible into the physician’s hands, keeps eHealth information secure, protects patient privacy and facilitates better practice of medicine and better outcomes.

Bad Health IT ("BHIT") is defined as IT that is ill-suited to purpose, hard to use, unreliable, loses data or provides incorrect data, causes cognitive overload, slows rather than facilitates users, lacks appropriate alerts, creates the need for hypervigilance (i.e., towards avoiding IT-related mishaps) that increases stress, is lacking in security, compromises patient privacy or otherwise demonstrates suboptimal design and/or implementation.  

I am an advocate of the former, and an opponent of the latter.

Bad health IT is prevalent in 2012 due to lack of meaningful quality control, software validation, usability standards and testing, and regulation of any type - a situation HIMSS long favored.   (The failed National Programme for HIT [NPfIT] in the NHS learned this the hard way, as will Australians needing emergency care, I predict.)

Put bluntly, in the end BHIT maims and kills patients, squanders precious healthcare resources, and drains the treasury into IT industry pockets (see my Oct. 3, 2012 post "Honesty and Good Sense on Electronic Medical Records From Down Under" and this query link on health IT risks). 

Further, the failure to recognize that the technology's downsides need to be understood and remediated before national deployment occurs and under controlled conditions, not after (which uses patients as non-consenting experimental subjects for software debugging), speaks to gross corporate negligence on the part of HIMSS.  It's not as if they did not have advance warning of all of healthcare IT's deficiencies. 

BHIT also permits record alterations after the fact that may be to conceal medical error.  I am aware of numerous instances of such alterations, fortunately caught by critical-thinking, detail-minded attorneys.  However, like health IT harms, the incidences of known alteration attempts likely reflect the "tip of the iceberg."

HIMSS and its fellow travelers have thus led this administration down the Garden Path of health IT perdition.

I warned of this in a Feb. 18, 2009 Wall Street Journal letter to the editor:

Dear Wall Street Journal:

You observe that the true political goal is socialized medicine facilitated by health care information technology. You note that the public is being deceived, as the rules behind this takeover were stealthily inserted in the stimulus bill.

I have a different view on who is deceiving whom. In fact, it is the government that has been deceived by the HIT industry and its pundits. Stated directly, the administration is deluded about the true difficulty of making large-scale health IT work. The beneficiaries will largely be the IT industry and IT management consultants.

For £12.7 billion the U.K., which already has socialized medicine, still does not have a working national HIT system, but instead has a major IT quagmire, some of it caused by U.S. HIT vendors. [That project, the NPfIT in the NHS, has now been abandoned - ed.]

HIT (with a few exceptions) is largely a disaster. I'm far more concerned about a mega-expensive IT misadventure than an IT-empowered takeover of medicine.

The stimulus bill, to its credit, recognizes the need for research on improving HIT. However this is a tool to facilitate clinical care, not a cybernetic miracle to revolutionize medicine. The government has bought the IT magic bullet exuberance hook, line and sinker.

I can only hope patients get something worthwhile for the $20 billion.

Scot Silverstein, MD

Mr. President, again, quite bluntly, the health IT industry took you for a ride, and the damage is done and is continuing due to lack of any meaningful health IT post market surveillance.   (As I wrote here, the untoward results have already been used against the current administration, with more perhaps to follow.)

In the U.S. we have the 1938 Federal Food, Drug, and Cosmetic Act (FD&C Act) in place:

The introduction of this act was influenced by the death of more than 100 patients due to a sulfanilamide medication where diethylene glycol was used to dissolve the drug and make a liquid form.  See Elixir Sulfanilamide disaster. It replaced the earlier Pure Food and Drug Act of 1906.

It should be honored, not ignored via special accommodation to the health IT industry and its trade group.

HITECH also needs to be put in dormancy until the problems with these unregulated medical devices get worked out in relatively small, controlled settings to minimize risk, with patient informed consent.  This is in accord with human rights documents dating at least to the Nuremberg Code, as in any new, experimental or partly-experimental medical device, pharmaceutical, or therapy.

-- SS
 

The Scientific Justification for Meaningul Use, Stage 2: The NWB Methodology

The Final Rule for Meaningful Use Stage 2 has been released.  It is at this link (PDF).  It is a mere 672 pages in length, a quick read during one's evenings of leisure.

Here is, from page 18 verbatim, the scientific justification for the program.  The finest scientific methods were used to achieve these criteria in justification of spending of $15 billion of taxpayer money in "incentives" (probably a low estimate), not counting the additional hundreds of billions the buyers themselves will spend that is diverted from your healthcare to the IT sector:

3. Summary of Costs and Benefits

This final rule is anticipated to have an annual effect on the economy of $100 million or more, making it an economically significant rule under the Executive Order and a major rule under the Congressional Review Act.

Accordingly, we have prepared a Regulatory Impact Analysis that to the best of our ability presents the costs and benefits of the final rule. The total Federal cost of the Medicare and Medicaid EHR Incentive Programs between 2014 and 2019 is estimated to be $15.4 billion (these estimates include net payment adjustments for Medicare providers who do not achieve meaningful use in 2015 and subsequent years in the amount of $2.1 billion).

In this final rule we have not quantified the overall benefits to the industry, nor to EPs, eligible hospitals, or CAHs participating in the Medicare and Medicaid EHR Incentive Programs. Information on the costs and benefits of adopting systems specifically meeting the requirements for the EHR Incentive Programs has not yet been collected and information on costs and benefits overall is limited. 

Nonetheless, we believe there are substantial benefits that can be obtained by eligible hospitals and EPs, including reductions in medical recordkeeping costs, reductions in repeat tests, decreases in length of stay, increased patient safety, and reduced medical errors. There is evidence to support the cost-saving benefits anticipated from wider adoption of EHRs.

There's no truly robust evidence of generalizable benefit, no randomized trials, there's significant evidence to the contrary (that, incidentally, is deliberately being ignored), there's risk to safety that this disruptive technology causes in its present state (but the magnitude is unknown, see quotes from 2012 IOM study here) that MU and "certification" do not address, there's a plethora of hair-raising defect reports from the only seller that reports such things, but CMS justifies the program with the line:

"Evidence [on benefits] is limited ... Nonetheless, we believe there are substantial benefits that can be obtained by eligible hospitals and EPs ... There is evidence to support the cost-saving benefits anticipated from wider adoption of EHRs."

I am deeply impressed by the level of rigorous science here.  We are truly in a golden age of science.

I recommend NIH, NSF, FDA and all other research and regulatory agencies immediately adopt this rigorous HHS methodology (called the "NWB" methodology for "nonetheless we believe") in their professional pursuits and research grant approval processes.

-- SS