Course on Trustworthiness of Clinical Practice Guidelines
Dr Roy Poses and Dr Wally Smith will be teaching a course on Trustworthiness of Clinical Practice Guidelines at the Annual North American meeting of the Society for Medical Decision Making, Doral, Florida, 18-22 October, 2014.
Particularly relevant to Health Care Renewal, the course will focus on reasons physicians may fail to trust clinical practice guidelines, including concerns about the integrity of the evidence base supporting the guidelines, particularly due to manipulation and suppression of clinical research, and concerns about the guideline development process, including conflicts of interest. Furthermore, we will discuss measures that might improve the trustworthiness of the CPG development process, with emphasis on the recent report by the US Institute of Medicine, of which Dr Smith was an author.
Special Symposium on Understanding and Challenging Health Care Corruption
Dr Roy Poses and Dr Wally Smith will be presenting a special symposium, Defense against the Dark Arts: Understanding and Challenging Health Care Corruption at the Annual Meeting of the Society for General Internal Medicine, Toronto, Ontario, Canada, 22- 25 April, 2015.
We plan to summarize the scope of health care corruption, and discuss the impunity of those involved, and how discussion of it became taboo, before brainstorming ways to begin to challenge corruption.
Showing posts with label guidelines. Show all posts
Showing posts with label guidelines. Show all posts
Monday, September 22, 2014
Thursday, January 23, 2014
Putting in the Ski Lifts - National Quality Forum Leaders Improve Conflict of Interest Policies in Response to CareFusion Case
The complex case that came to light with reports of a legal settlement by CareFusion of allegations that it paid kickbacks to promote its ChloraPrep product just got even more complex, but now also a bit more hopeful, and hence much more hopeful than most of the cases we discuss..
Background
As we noted yesterday, the CareFusion settlement seemed routine when it was first briefly reported in the media. Then Modern Healthcare published a series of articles about its ramifications, the latest appearing today.
In summary prior to the latest article,
- the CareFusion settlement for $40.1 million was made in response to allegations that kickbacks were made to promote ChloraPrep, a solution meant for preoperative skin cleaning
- the Department of Justice news release also alleged that payments were made to a corporation called Health Care Concepts to conceal kickbacks made to its owner, Dr Charles Denham
- the implication was that Dr Denham was supposed to influence a standard writing committee run by the National Quality Forum, a well known organization that promotes quality improvement, issues authoritative practice standards, a form of clinical practice guidelines, and has contracts with the US government for quality of care activities
- the draft of the standard written by the committee allegedly included the use of ChloraPrep, although mention of that specific medication was removed in a revision
We noted that the NQF policy on conflicts of interest for its standard writing committees seemed somewhat weak. The current (October, 2013) policy asks members to internally disclose conflicts and recuse themselves from " discussion of the applicable measure or measures, and in some instances competing and related measures." However it does not ban members with conflicts, nor require public disclosure of conflicts. We also noted that NQF has several board members who have full-time leadership positions with health care corporations, financial firms that invest in health care, or health care trade associations, and NQF received funding from pharmaceutical companies and their in-house foundations. Thus NQF appears to have its own institutional conflicts of interest.
So we thought that the case raised questions about how well NQF standards are protected from the influence of conflicts of interest, and whether they should remain so influential in the absence of stronger protections?
NQF Clarifies its Conflicts of Interest Policies
As reported by Modern Healthcare, it took only a day for the NQF to issue yet another press release to clarify its conflict of interest polcies. In particular,
So NQF appeared to react ad hoc to concerns that Dr Denham was too avid about the use of a particular product. Its response included strengthening its COI policy and banning one particular type of conflict, receiving funding from an organization controlled by a member of a standard writing committee. However, as we noted above, the 2013 policy still has its weaknesses.
Furthermore, an NQF executive volunteered to inform me that while the current written COI policy does not require public disclosure, in practice the organization requires public verbal disclosure of conflicts at the beginning of each committee meeting, and that the transcripts of these disclosures are available on the organization's website. However, how to reach these disclosures is not obvious, and I so far have not been able to find the transcript that supposedly contains the disclosures for the meeting which Dr Denham co-chaired
Summary and a Few Optimistic Thoughts (Rare as They are for Us)
What I must say is now most unusual in the CareFusion/ Dr Denham/ NQF case is that NQF leadership, rather than hunkering down in a defensive posture we have seen many other leaders adapt, seems to be trying to take steps toward addressing problems posed by conflicts of interest affecting its standard setting, guideline development, and quality assurance. First, it is clear that NQF leaders did react to concerns about Dr Denham, and did somewhat tighten up their conflict of interest policies in response to them. Second, it is clear that they are willing to listen to public questions and criticism, and at least consider further change in response to them. They thus appear far more open than do leaders of many other organizations.
IMHO, NQF leadership is to be congratulated for this more transparent approach. If only the leaders of other health care organizations would do as much.
Since they do seem open to criticism and change, in the spirit of constructive criticism, I do hope that NQF leaders might also consider:
- further strengthening their conflict of interest policy for standard setting committees, particularly by minimizing the number of committee members with conflicts, and banning individuals with conflicts from chairing committees, as per the IOM report on standards for trustworthy clinical practice guideline development
- increasing transparency about conflicts of interest, particularly by making disclosures for all projects available without complex web searching
- decreasing institutional conflicts of interest by refusing funding from health care corporations whose revenues might be affected by the content of standards and guidelines, and reducing conflicts affecting NQF board members and executives
Health care leaders who see transparency and reduction of conflicts of interest as important goals show us that true health care reform might actually be possible.
ADDENDUM (24 January, 2014 - I was told that many people may need an explanation of the title. "Putting in the ski lifts" is a shortened version of an old college saying. The location for the ski lifts was a place that is traditionally very hot, think fire and brimstone. The phrase refers to an extremely improbably event. Sorry about any confusion.
Background
As we noted yesterday, the CareFusion settlement seemed routine when it was first briefly reported in the media. Then Modern Healthcare published a series of articles about its ramifications, the latest appearing today.
In summary prior to the latest article,
- the CareFusion settlement for $40.1 million was made in response to allegations that kickbacks were made to promote ChloraPrep, a solution meant for preoperative skin cleaning
- the Department of Justice news release also alleged that payments were made to a corporation called Health Care Concepts to conceal kickbacks made to its owner, Dr Charles Denham
- the implication was that Dr Denham was supposed to influence a standard writing committee run by the National Quality Forum, a well known organization that promotes quality improvement, issues authoritative practice standards, a form of clinical practice guidelines, and has contracts with the US government for quality of care activities
- the draft of the standard written by the committee allegedly included the use of ChloraPrep, although mention of that specific medication was removed in a revision
We noted that the NQF policy on conflicts of interest for its standard writing committees seemed somewhat weak. The current (October, 2013) policy asks members to internally disclose conflicts and recuse themselves from " discussion of the applicable measure or measures, and in some instances competing and related measures." However it does not ban members with conflicts, nor require public disclosure of conflicts. We also noted that NQF has several board members who have full-time leadership positions with health care corporations, financial firms that invest in health care, or health care trade associations, and NQF received funding from pharmaceutical companies and their in-house foundations. Thus NQF appears to have its own institutional conflicts of interest.
So we thought that the case raised questions about how well NQF standards are protected from the influence of conflicts of interest, and whether they should remain so influential in the absence of stronger protections?
NQF Clarifies its Conflicts of Interest Policies
As reported by Modern Healthcare, it took only a day for the NQF to issue yet another press release to clarify its conflict of interest polcies. In particular,
I n response to concerns that the community raised about the evidence base underlying Safe Practice #22 and Dr. Denham’s inordinate interest in this particular Safe Practice, NQF took the following steps in early 2010:
- Severed its relationship with Dr. Denham who has not been involved in any other NQF work since March 2010;
- Made a decision to refuse offers for financial support from Dr. Denham’s foundation, discontinuing a 2008 - 2013 grant agreement that had more than three years remaining ;
- Determined that it would not enter into grant agreements where the funder is on the endorsement committee, even as a non-voting member;
- Reviewed committee reports that Dr. Denham was involved in to be certain that he did not influence their outcome; NQF staff believe that he did not; and
- Updated and enhanced its conflict of interest policy – in 2010 and again in 2013;
So NQF appeared to react ad hoc to concerns that Dr Denham was too avid about the use of a particular product. Its response included strengthening its COI policy and banning one particular type of conflict, receiving funding from an organization controlled by a member of a standard writing committee. However, as we noted above, the 2013 policy still has its weaknesses.
Furthermore, an NQF executive volunteered to inform me that while the current written COI policy does not require public disclosure, in practice the organization requires public verbal disclosure of conflicts at the beginning of each committee meeting, and that the transcripts of these disclosures are available on the organization's website. However, how to reach these disclosures is not obvious, and I so far have not been able to find the transcript that supposedly contains the disclosures for the meeting which Dr Denham co-chaired
Summary and a Few Optimistic Thoughts (Rare as They are for Us)
What I must say is now most unusual in the CareFusion/ Dr Denham/ NQF case is that NQF leadership, rather than hunkering down in a defensive posture we have seen many other leaders adapt, seems to be trying to take steps toward addressing problems posed by conflicts of interest affecting its standard setting, guideline development, and quality assurance. First, it is clear that NQF leaders did react to concerns about Dr Denham, and did somewhat tighten up their conflict of interest policies in response to them. Second, it is clear that they are willing to listen to public questions and criticism, and at least consider further change in response to them. They thus appear far more open than do leaders of many other organizations.
IMHO, NQF leadership is to be congratulated for this more transparent approach. If only the leaders of other health care organizations would do as much.
Since they do seem open to criticism and change, in the spirit of constructive criticism, I do hope that NQF leaders might also consider:
- further strengthening their conflict of interest policy for standard setting committees, particularly by minimizing the number of committee members with conflicts, and banning individuals with conflicts from chairing committees, as per the IOM report on standards for trustworthy clinical practice guideline development
- increasing transparency about conflicts of interest, particularly by making disclosures for all projects available without complex web searching
- decreasing institutional conflicts of interest by refusing funding from health care corporations whose revenues might be affected by the content of standards and guidelines, and reducing conflicts affecting NQF board members and executives
Health care leaders who see transparency and reduction of conflicts of interest as important goals show us that true health care reform might actually be possible.
ADDENDUM (24 January, 2014 - I was told that many people may need an explanation of the title. "Putting in the ski lifts" is a shortened version of an old college saying. The location for the ski lifts was a place that is traditionally very hot, think fire and brimstone. The phrase refers to an extremely improbably event. Sorry about any confusion.
Wednesday, January 22, 2014
Our Man in the NQF? - CareFusion Settles Kickback Allegations for $40.1 Million and Government Alleges They Meant to Manipulate National Quality Forum Standards
What appeared to be yet another entry in the march of legal settlements turns out to be more interesting than it first appeared.
The Basic Story: Off-Label Marketing
The story was initially briefly reported in the media. A report in the Seattle Post-Intelligencer on 9 January, 2014, was perhaps the most complete,
In particular,
So this follows a template we have seen before, most recently here. A company promotes a drug or device for "off-label" uses. While physicians are legally allowed to prescribe a drug or use a device for reasons other than its official indications, companies are not legally allowed to promote such uses. Sometimes it turns out that the drug has benefits that outweigh its harms when used off-label, in which case the violation could be considered technical. In other cases, the drug might turn out to be useless or even net harmful when used off-label, so the violation indicates putting revenue ahead of patients' welfare.
The reporting on the CareFusion Chloraprep settlement did not clearly suggest that the off-label use was harmful to patients.
Allegations of Kickbacks to Promote Off-Label Use, but How?
It gets more interesting. The Post-Intelligencer also noted,
Now the allegations also include kickbacks to the "physician-owner" of a company to facilitate off-label promotion. So the issue goes beyond a technical violation of the law. The Post-Intelligence report, however, did not make clear what Health Care Concepts Inc actually did to contribute to off-label promotion.
A quick attempt to find out more about this company via the web was not fruitful. There are lots of companies/ organizations called Health Care Concepts or some variant. The most relevant may be this one, which lists CareFusion as a "partner" on its main web-page. What this company actually does is a bit unclear. Its business model is described as "a private incubator of early products, services, and technologies, as well as an accelerator of mature solutions that will be taken to scale." What that has to do with off-label promotion of a product of a big established company like CareFusion is not clear. How this company could have influenced providers is not clear, given that it is not an advertising firm, public relations firm, or medical education and communication company (MECC).
So what is going on here?
Allegations of a $11.6 Million Payment to Manipulate the National Quality Forum Standards
Yesterday, 21 January, 2014, Modern Healthcare began to explain the mystery.
The allegations about the role of Dr Denham in the NQF also did appear in the the official US Department of Justice press release.
Allegations of Bogus Contracts, and Non-Denial Denials
The allegations produced some non-denial denials (in the phrase found in All the President's Men).
Dr Denham responded, but through a lawyer:
Note that while the attorney denied that Dr Denham was named in the underlying lawsuit, he did not apparently specifically deny the allegations.
The NQF also reponded,
Note that the NQF press release did not deny that the CareFusion money was given to Dr Denham's organization, nor that the NQF report might have promoted the use of a class of products in which CareFusion's Chloraprep resides.
A follow-up story in Modern Healthcare noted that Dr Denham confirmed the payment of the money to his company, but not its purpose,
Dr Denham's lawyer confirmed the payment, but did not further explain it:
And,
Again, note that Mr Gondelman did not specifically deny that the money served as a kickback.
Meanwhile, the Department of Justice alleged something more elaborate, and nefarious,
Interval Summary
So what is clear so far is that:
- CareFusion agreed to a $40.1 million settlement of allegations that it provided kickbacks to promote off-label use of its product.
- CareFusion paid $11.6 million to Health Care Concepts
- Health Care Concepts is run (and possibly owned) by Dr Charles Denham, who chaired a National Quality Forum committee that recommended use of CareFusion's product ChloraPrep in a performance standard and the final version of the NQF standard recommended use of a group of products including ChloraPrep
We have frequently discussed how health care corporations may promote goods, particularly drugs and devices and services through deceptive and unethical practices, including inducements ranging form creation of conflicts of interest to outright bribes and kickbacks. We have also discussed the influence of individual and institutional conflicts of interest on clinical practice guidelines. This has become an important enough problem for the US Institute of Medicine to weigh in with standards for trustworthy guidelines that markedly limit such conflicts of interest.
This case seems notable because it may link the problems of deceptive and unethical promotion with the influencing of clinical practice guidelines. It is particularly notable because the organization generating the guidelines in question, the National Quality Forum, has become large, influential, and authoritative. Re the latter point, the latest Modern Healthcare article affirmed,
In fact, while the NQF performance standards can function as guidelines, they also may be bases for pay for performance schemes that attach real incentives to guidelines, which could become perverse if the guidelines have been manipulated.
Can the NQF Defend Against Conflicts, and If Not, Should it Remain So Authoritative?
So the big question is whether the NQF, given its unique influence, has any meaningful approach to prevent its standards from being influenced by conflicts of interest?
The NQF does have a recent conflicts of interest policy. However, according to my reading, it neither bans committee members with conflicts of interest, nor even requires public disclosure of conflicts of interest. All it seems to require is that committee members disclose conflicts internally andvoluntarily recuse themselves from deliberations which may be affected by them. Although the policy states that conflicts of interest may be made public, I could find nothing in the report which Dr Denham co-authored about conflicts of interest or competing interests affecting any of the authors, including Dr Denham
Furthermore, a very quick look at information publicly available on the NQF web-site suggests the organization itself may have major institutional conflicts of interest. The NQF is one of those "public-private partnerships" that have become all the rage. Several members of its board of directors have leadership positions in large health corporations, financial firms with large health care interests, or trade associations for large health care corporations:
- Elizabeth J. Fowler, PhD, JD, is Vice President, Global Health Policy, in the Government Affairs & Policy group at Johnson & Johnson.
- Robert Galvin, MD, is Chief Executive Officer, Equity Healthcare, at The Blackstone Group.
- Karen Ignagni, President and Chief Executive Officer of America’s Health Insurance Plans (AHIP), is the voice of health insurance plans....
- J Mark Overhage MD, PhD, Chief Medical Informatics Officer, Health Services, Siemens Healthcare
Thus it is conceivable that they could personally profit, and that their organizations could institutionally profit from standards that favor their organizations' interests.
Furthermore, NQF receives funding from the US government and private foundations, but also from pharmaceutical companies and their in-house foundations:
Unless the current case leads to big changes in the prevalence of conflicts of interest among the NQF board, the continuation of its institutional conflicts of interest generated by its sources of funding, and its weak policy on conflicts affecting committee members, the question becomes whether it should continue to have such an authoritative role?
As we have said over and over, pervasive conflicts of interest affect health care decision-makers and policy makers and the organizations within which they function. Such conflicts threaten to put various private interests way ahead of patients' and the public's health. If we really want health care that will put patients' and the public's health first, we need to end such arrangements that favor private gain.
Roy M. Poses MD on Health Care Renewal
ADDENDUM (23 January, 2014) - A closer re-reading of the NQF conflict of interest policy made it clear that recusal is not voluntary, and that conflicts of interest may be revealed publicly, but that such disclosure is not mandatory. The paragraph above dealing with these issues was revised accordingly.
The Basic Story: Off-Label Marketing
The story was initially briefly reported in the media. A report in the Seattle Post-Intelligencer on 9 January, 2014, was perhaps the most complete,
CareFusion, a manufacturer of medical and surgical supplies and medical devices, has agreed to settle charges of illegal marketing practices and kickback payments for promoting sales of the company’s surgical preparation solution, Chloraprep.
Under the terms of the settlement with Washington, other states, and the federal government, CareFusion will pay a total of $40.1 million.
In particular,
When the FDA approves a surgical solution such as Chloraprep as safe and effective, its manufacturer can’t market or promote it for an 'off-label' use – any use not approved by the FDA.
Chloraprep was approved for specific inpatient hospital procedures, including the preparation of a patient’s skin prior to surgery or an injection. The FDA rejected its use for prepping the skin prior to inserting catheters into veins for administering medications, or cleaning the skin as part of closing wounds.
The lawsuit alleges that from Sept. 1, 2009 to Aug. 31, 2011, CareFusion promoted Chloraprep for improper uses. CareFusion also allegedly publicized unverified information about Chloraprep during the same time period, Ferguson said.
So this follows a template we have seen before, most recently here. A company promotes a drug or device for "off-label" uses. While physicians are legally allowed to prescribe a drug or use a device for reasons other than its official indications, companies are not legally allowed to promote such uses. Sometimes it turns out that the drug has benefits that outweigh its harms when used off-label, in which case the violation could be considered technical. In other cases, the drug might turn out to be useless or even net harmful when used off-label, so the violation indicates putting revenue ahead of patients' welfare.
The reporting on the CareFusion Chloraprep settlement did not clearly suggest that the off-label use was harmful to patients.
Allegations of Kickbacks to Promote Off-Label Use, but How?
It gets more interesting. The Post-Intelligencer also noted,
The states also contend that in 2008 CareFusion’s predecessor corporation entered into agreements that CareFusion assumed legal and financial responsibility for. This included payments to Health Care Concepts Inc., payments that were allegedly made to hide kickbacks to the physician-owner of HCC for promoting and influencing providers to use Chloraprep, he said. This violates federal and state anti-kickback laws.
Now the allegations also include kickbacks to the "physician-owner" of a company to facilitate off-label promotion. So the issue goes beyond a technical violation of the law. The Post-Intelligence report, however, did not make clear what Health Care Concepts Inc actually did to contribute to off-label promotion.
A quick attempt to find out more about this company via the web was not fruitful. There are lots of companies/ organizations called Health Care Concepts or some variant. The most relevant may be this one, which lists CareFusion as a "partner" on its main web-page. What this company actually does is a bit unclear. Its business model is described as "a private incubator of early products, services, and technologies, as well as an accelerator of mature solutions that will be taken to scale." What that has to do with off-label promotion of a product of a big established company like CareFusion is not clear. How this company could have influenced providers is not clear, given that it is not an advertising firm, public relations firm, or medical education and communication company (MECC).
So what is going on here?
Allegations of a $11.6 Million Payment to Manipulate the National Quality Forum Standards
Yesterday, 21 January, 2014, Modern Healthcare began to explain the mystery.
Many healthcare companies have been accused of paying kickbacks to influence physicians' medical decisions, but one major supplier allegedly tried to steer an entire industry by influencing the recommendations of the National Quality Forum
The NQF, which reviews medical evidence and makes recommendations for improving healthcare, acknowledges that a well-known patient safety advocate is alleged to have received millions of dollars to promote a surgical sterilization product while co-chairing an NQF committee that suggested hospitals use that product.
Federal officials say Dr. Charles Denham, co-chair of NQF's Safe Practices Committee in 2010, received $11.6 million from San Diego-based CareFusion to promote the company's ChloraPrep line of skin-preparation products. Denham's committee at the NQF also recommended surgeons use ChloraPrep products to prevent surgical infections, the NQF said.
The allegations about the role of Dr Denham in the NQF also did appear in the the official US Department of Justice press release.
Allegations of Bogus Contracts, and Non-Denial Denials
The allegations produced some non-denial denials (in the phrase found in All the President's Men).
Dr Denham responded, but through a lawyer:
A statement from Denham's attorney, Larry Gondelman of Powers Pyles Sutter & Verville in Washington, said reports that the underlying lawsuit involved Dr. Denham were 'blatantly false.'
Denham was not accused of wrong-doing in the whistleblower lawsuit, and Gondelman said Denham has and continues to cooperate with investigators in the case.
Note that while the attorney denied that Dr Denham was named in the underlying lawsuit, he did not apparently specifically deny the allegations.
The NQF also reponded,
An NQF review committee removed the reference to the specific product before final publication of 'Safe Practices for Better Healthcare' in 2010.
'An NQF ad hoc review did not find sufficient evidence to support one skin preparation over another,' a statement from the organization said. 'The ad hoc review effectively served its role of rapidly responding when a potential issue is identified.'
Note that the NQF press release did not deny that the CareFusion money was given to Dr Denham's organization, nor that the NQF report might have promoted the use of a class of products in which CareFusion's Chloraprep resides.
A follow-up story in Modern Healthcare noted that Dr Denham confirmed the payment of the money to his company, but not its purpose,
Renowned quality expert Dr. Charles Denham confirms that CareFusion paid his company more than $11 million, but Denham says he was surprised to see the U.S. Justice Department describe the money as kickbacks intended to influence national quality-of-care guidelines published by the National Quality Forum.
Dr Denham's lawyer confirmed the payment, but did not further explain it:
Denham's lawyer, Larry Gondelman of Powers Pyles Sutter & Verville in Washington, acknowledged that CareFusion did pay Denham's company under two contracts.
And,
Gondelman said the $11.6 million paid to Denham's company covered 'an array of services' and that he could not elaborate.
Again, note that Mr Gondelman did not specifically deny that the money served as a kickback.
Meanwhile, the Department of Justice alleged something more elaborate, and nefarious,
The Justice Department, though, says the contracts were written to look as though CareFusion was buying consulting, software development and strategic marketing services, as well as the completion of three unnamed projects. The value of the deals, the government alleges, far exceeded the fair-market costs of those services, and that one of the actual purposes of the deals was to conceal kickbacks to Denham while he served on the National Quality Forum.
Interval Summary
So what is clear so far is that:
- CareFusion agreed to a $40.1 million settlement of allegations that it provided kickbacks to promote off-label use of its product.
- CareFusion paid $11.6 million to Health Care Concepts
- Health Care Concepts is run (and possibly owned) by Dr Charles Denham, who chaired a National Quality Forum committee that recommended use of CareFusion's product ChloraPrep in a performance standard and the final version of the NQF standard recommended use of a group of products including ChloraPrep
We have frequently discussed how health care corporations may promote goods, particularly drugs and devices and services through deceptive and unethical practices, including inducements ranging form creation of conflicts of interest to outright bribes and kickbacks. We have also discussed the influence of individual and institutional conflicts of interest on clinical practice guidelines. This has become an important enough problem for the US Institute of Medicine to weigh in with standards for trustworthy guidelines that markedly limit such conflicts of interest.
This case seems notable because it may link the problems of deceptive and unethical promotion with the influencing of clinical practice guidelines. It is particularly notable because the organization generating the guidelines in question, the National Quality Forum, has become large, influential, and authoritative. Re the latter point, the latest Modern Healthcare article affirmed,
The NQF is a standards-setting organization for the healthcare provider industry. It holds the exclusive contract to provide Medicare and Medicaid with recommendations on a national quality-of-care strategy and the measures used to gauge improvements.
In fact, while the NQF performance standards can function as guidelines, they also may be bases for pay for performance schemes that attach real incentives to guidelines, which could become perverse if the guidelines have been manipulated.
Can the NQF Defend Against Conflicts, and If Not, Should it Remain So Authoritative?
So the big question is whether the NQF, given its unique influence, has any meaningful approach to prevent its standards from being influenced by conflicts of interest?
The NQF does have a recent conflicts of interest policy. However, according to my reading, it neither bans committee members with conflicts of interest, nor even requires public disclosure of conflicts of interest. All it seems to require is that committee members disclose conflicts internally and
Furthermore, a very quick look at information publicly available on the NQF web-site suggests the organization itself may have major institutional conflicts of interest. The NQF is one of those "public-private partnerships" that have become all the rage. Several members of its board of directors have leadership positions in large health corporations, financial firms with large health care interests, or trade associations for large health care corporations:
- Elizabeth J. Fowler, PhD, JD, is Vice President, Global Health Policy, in the Government Affairs & Policy group at Johnson & Johnson.
- Robert Galvin, MD, is Chief Executive Officer, Equity Healthcare, at The Blackstone Group.
- Karen Ignagni, President and Chief Executive Officer of America’s Health Insurance Plans (AHIP), is the voice of health insurance plans....
- J Mark Overhage MD, PhD, Chief Medical Informatics Officer, Health Services, Siemens Healthcare
Thus it is conceivable that they could personally profit, and that their organizations could institutionally profit from standards that favor their organizations' interests.
Furthermore, NQF receives funding from the US government and private foundations, but also from pharmaceutical companies and their in-house foundations:
the Bristol-Myers Squibb Foundation, the Cardinal Health Foundation, Pfizer Inc., Sanofi-aventis,...
Unless the current case leads to big changes in the prevalence of conflicts of interest among the NQF board, the continuation of its institutional conflicts of interest generated by its sources of funding, and its weak policy on conflicts affecting committee members, the question becomes whether it should continue to have such an authoritative role?
As we have said over and over, pervasive conflicts of interest affect health care decision-makers and policy makers and the organizations within which they function. Such conflicts threaten to put various private interests way ahead of patients' and the public's health. If we really want health care that will put patients' and the public's health first, we need to end such arrangements that favor private gain.
Roy M. Poses MD on Health Care Renewal
ADDENDUM (23 January, 2014) - A closer re-reading of the NQF conflict of interest policy made it clear that recusal is not voluntary, and that conflicts of interest may be revealed publicly, but that such disclosure is not mandatory. The paragraph above dealing with these issues was revised accordingly.
Tuesday, December 3, 2013
How Manipulated Clinical Evidence Could Distort Guidelines - the Case of Statins for Primary Prevention
The new American College of Cardiology (ACC)/ American Heart Association (AHA) guidelines on the primary prevention of cardiovascular disease, which we discussed here, continue to generate controversy.
Articles in the media and online-first in medical journals underscored some of the issues we discussed before. Jeanne Lenzer, in a news article in the British Medical Journal, found that the chair of the guideline panel had important past conflicts of interest that were not previously disclosed.(1) The chair denied their significance (look here). Guideline panel members continued to justify their efforts, but in my humble opinion, raised no new evidence or logic to support it (look here and here)
Bigger Questions about the Validity of the Clinical Evidence
However, several new articles suggested the need for a deeper look at one particular aspect of this case, the validity of the evidence from clinical research about the benefits and harms of statins for primary prevention.
JAMA published a summary of the Cochrane review that provided a basis for the guideline developers' confidence in the worth of statin therapy in primary prevention.(2)
However, almost as an aside, it noted,
Primary prevention in this case is defined as prevention for patients without existing cardiovascular disease. There is evidence that statins may well have benefits that outweigh harms when given to patients with known cardiovascular disease, particularly documented coronary artery disease (CAD). Mixing such patients in any numbers into "primary prevention trials" would likely exaggerate the benefits of statins. Yet such not quite primary prevention studies were included in a systematic review of primary prevention.
In addition, a commentary by one of the guideline developers defending the group's work also underscored the fact that many of the supposedly primary prevention trials they used as evidence were not pure primary prevention trials.(3)
The wording again suggests that all the other trials used as evidence about primary prevention were NOT "exclusively primary prevention studies," and hence, as I noted above, may have been biased so that they would likely exaggerate the apparent benefits of statins in primary prevention.
This suggested to me that the systematic review that provided a basis for the guidelines' aggressive recommendations about statins for primary prevention, and the trials on which it was based needed further skeptical, critical reviews.
The Cochrane Review: Was Evidence about Statin Benefits vs Harms Manipulated?
We have frequently discussed the manipulation of clinical research. By that we have meant design, implementation, analysis or dissemination of research in ways likely to further vested interests. In particular, when drug, biotechnology and device companies sponsor and control clinical research on their own products, they may set up the research in ways likely to make their products look better than they actually are.
We looked through the most recent Cochrane review of statins for primary prevention(7) with a skeptical eye out for suggestions that the trials reviewed could have been so manipulated. We found quite a bit.
First, we found that not all trials reported on the "hard" outcomes that one needs to consider when evaluating benefit vs harm of statins in primary prevention.
And,
Also, data on specific adverse events was often not reported: myalgia and rhabdomyolysis were reported in only 9/19 trials; new onset diabetes mellitus in only 2/18; hemmorhagic stroke in only 2/19; abnormal liver tests in only 10/19; kidney dysfunction in only 4/19; arthritis in only 2/19, and by implication, cognitive dysfunction in 0/19.
Failure to look for all the possible bad outcomes of treatment could obviously bias the study in the direction of minimizing the harms of the treatment.
A substantial number of trials either failed to report on crucial aspects of their methods, or admitted to flaws that could have induced important biases.: 3/19 did not described randomization methods; 4/19 did not use double-blinding; 6/19 did not use intention to treat analysis; 7/19 did not report their drop-out rates.
So it is very surprising to me that the authors concluded,
Review of the "3 Exclusively Primary Prevention Statin Trials"
Given the Cochrane review's apparent lack of skepticism about methodologic problems in the industry funded statin prevention trials, I endeavored to take a closer look at the three trials that Dr Robinson held out as the real primary prevention trials. Instances of manipulation, as we defined it above, for each trial are described below
AFCAPS/ TexCAPS(4)
Narrow Patient Population - This study excluded many patient for whom the statins were not contraindicated or warned against: uncontrolled hypertension; type 1 or type 2 diabetes mellitus on insulin or with a HgBA1C at least 10%; and body weight more than 50% "desirable limit for height." (Based on the official contraindications and warnings for commonly used statins, e.g., see contraindications for Lipitor here, active liver disease, pregnancy for likely to become pregnant, nursing mothers, hypersensitivity to the medicine; and warnings: use of cyclosprine or strong CYP3A4 inhibitors, uncontrolled hypothyroidism, renal impairment.) Thus the results may not generalize to many patients who would otherwise be considered statin candidates. By excluding such patients, the results may bias the study towards minimizing the probabilities of harms that might occur were statins used on a wide population for primary prevention.
Unknown Randomization and Allocation Concealment Procedures - According to the Cochrane Review, the study report did not explain how randomization or allocation concealment were accomplished.
Early Termination/ Multiple Comparisons - The study was terminated early based on an early look at the number of outcome events. Two such early or interim analyses were planned. Taking multiple looks at the data over time raises a multiple comparisons problem, and may lead to exaggerating the benefits of the treatment.(8). Furthermore, stopping early decreases the sample size and hence the power to find adverse effects of treatment.
Implausible Dropout Rate, Missing Data - According to the Cochrane Review, the study reported no dropouts. This seems somewhat improbable, suggesting skepticism about the accuracy and completeness of the data collection. On the other hand, a study chronology suggests that of 6605 patients who started the study, 6540 had data on complete endpoint status, suggesting missing data. Since dropouts and missing data may be due to different reasons in different arms of the study, they threaten the validity of data about benefits and harms.
Adverse Effects not Reported - The study provided no data about development of diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, suggesting incomplete data about harms, and hence bias towards minimizing harms.
MEGA (5)
Narrow Patient Population - [The patient population was not described in the main report, but only in an earlier methods article.](9) The study excluded patients with congenital or rheumatic heart disease; chronic atrial fibrillation, current diagnosis of malignancy; poorly controlled hypertension or diabetes mellitus; current use of oral or parenteral corticosteroids; and other conditions at the discretion of the physician. These exclusions seem unrelated to the contraindications or warnings on the stain label. Again, such a narrow patient population reduces the generalizability of the study results, and may bias the study to minimizing the harms of statins.
Only Single Blind - This was an open-label study, so patients and physicians knew who got statins and who got placebo. Such knowledge could have biased patient management, including how diligently particular diagnoses and outcomes were pursued, and biased data collected from patients or physicians.
Adverse Effects Not Reported - The study provided no data about diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms.
JUPITER(6)
Narrow, Unusual Patient Population - The study was limited to patients without hyperlipidemia but with an increased C-reactive protein. Thus it is not clear that its results would generalize to a more typically defined primary prevention population. The study excluded patients receiving post-menopausal hormone-replacement; with diabetes; uncontrolled hyertension; cancer other than non-melanoma skin cancer within 5 years; recent history of drug or alcohol abuse; "another medical condition that might compromise safety or the successful completion of the study;" also patients with "inflammatory conditions such as severe arthritis, lupus, or inflammatory bowel disease...;" and also "patients taking immunosuppressant agents such as cyclosporine, tacrolimus, azathioprine, or long-term oral glucocorticoids." Again, this narrow patient population would reduce generalizability and bias towards minimizing harms.
Early Termination/ Multiple Comparisons - This study was terminated early after an early look at the data. Allowing for multiple looks at the data may exaggerate efficacy.
Implausible Dropout Rate - According to the Cochrane Review, the study had no dropouts. This value seems implausible, again suggesting data collection problems.
Adverse Effects Not Reported - The study provided no data abouthemmorhagic stroke, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms. [Revised December 9, 2013 - see comment below by Marilyn Mann.]
Summary
Aspects of the continuing controversy over the new ACC/ AHA guidelines for statins in the primary prevention of cardiovascular disease hinted that the clinical trials which provided the evidentiary basis for the guidelines, and for the use of statins in primary prevention in general, was more flawed than is widely appreciated. The latest Cochrane Collaboration review of this data acknowledged multiple, important flaws affecting most of the studies. Our more detailed review of the three studies held out as the purest found additional flaws. Many of these flaws seemed likely to bias the studies towards exaggerating the efficacy and/or minimizing the harms of statins in primary prevention. Since all these trials were funded, and presumably influenced by pharmaceutical companies that make statins, these flaws seem to be examples of manipulation of the clinical evidence. Rather than being the result of simple mistakes, or inevitable trade-offs, they seem to be study features intended the support the vested interests of the study sponsors.
It is not clear why the Cochrane review did not temper its conclusions based on the flaws in the studies, and particularly by the possibility that these flaws represented study manipulation.
The multiple flaws, possibly due to study manipulation, in the clinical evidence about statins in primary prevention suggest that we should be extremely skeptical about whether the benefits of such treatment outweighs its harms, and hence about whether the recommendations in the latest guidelines to give statins to all patients predicted (but perhaps not accurately) to be at even slightly elevated risk are warranted.
The flaws in multiple large studies of a very common clinical problem, and their effects on systematic reviews and clinical practice guidelines suggest that suppression and manipulation of research are rife in medicine and health care, presumably fueled by the pervasive web of conflicts of interest that spans health care. We need extreme skepticism about the integrity of clinical research, especially research sponsored by those whose products and services are being studied, and who thus have vested interests in the research turning out to make their products and services look good.
The good news is that we may not have to look too far to find ways to improve the trustworthiness of guidelines and the soundness of medical decision making. Implementation of the Institute of Medicine's recommendations on reducing conflicts of interest (look here), and developing trustworthy guidelines (look here) might lead to the development of sound guidelines in the future.
Furthermore, while endless discoveries of manipulated and suppressed research may have lead some evidence-based medicine advocates to despair, our latest exercise suggests that the principles of evidence-based medicine, unflinchingly applied, could really do good. Review of the three statin studies above based on standard principles of critical review readily spotted the multiple signs of manipulation. The problem with the Cochrane review was not that it missed these signs. Rather, the reviewers for some reason noted most of them, but then did not react. If systematic reviews were done with sufficient skepticism about the possibility of manipulation of clinical research, and were willing to call out when the emperor seemed short on fabric, then a lot of mischief could be avoided.
References
1. Lenzer J. Majority of panelists on controversial new cholesterol guideline have current or recent ties to drug manufacturers. Brit Med J 2013. Link here.
2. Taylor FC, Huffman M, Shah E. Statin therapy for primary prevention of cardiovascular disease. JAMA 2013. Link here.
3. Robinson JG. Accumulating evidence for statins in primary prevention. JAMA 2013; Link here.
4. Downs JR, Clearfield M, Weis S, et al; for the AFCAPS/TexCAPS Research Group. Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. JAMA. 1998;279(20):1615-1622. Link here.
5. Nakamura H, Arakawa K, Itakura H, et al; MEGA Study Group. Primary prevention of cardiovascular disease with pravastatin in Japan (MEGA study): a prospective randomised controlled trial. Lancet. 2006;368(9542):1155-1163. Link here.
6. Ridker PM, Danielson E, Fonseca FA, et al; JUPITER Study Group. Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein. N Engl J Med. 2008;359(21):2195-2207. Link here.
7. Taylor F, Huffman MD, Macedo AF et al. Statins for the prevention of cardiovascular disease. Cochrane Library 2013. Link here.
8. Mueller PS, Montori VM, Bassler D et al. Ethical issues in stopping randomized trials early because of apparent benefit. Ann Intern Med 2007; 146: 878-881. Link here.
9. Management of Elevated Cholesterol in the Primary Prevention Group of the Adult Japanese (MEGA) Study Group. Design and baseline characteristics of a study of primary prevention of coronary events with pravastatin among Japanese with mildly elevated cholesterol levels. Circ J 2004; 68: 860-867. Link here.
Articles in the media and online-first in medical journals underscored some of the issues we discussed before. Jeanne Lenzer, in a news article in the British Medical Journal, found that the chair of the guideline panel had important past conflicts of interest that were not previously disclosed.(1) The chair denied their significance (look here). Guideline panel members continued to justify their efforts, but in my humble opinion, raised no new evidence or logic to support it (look here and here)
Bigger Questions about the Validity of the Clinical Evidence
However, several new articles suggested the need for a deeper look at one particular aspect of this case, the validity of the evidence from clinical research about the benefits and harms of statins for primary prevention.
JAMA published a summary of the Cochrane review that provided a basis for the guideline developers' confidence in the worth of statin therapy in primary prevention.(2)
results suggest that the benefits of statin therapy outweigh serious life-threatening hazards.
However, almost as an aside, it noted,
Some trials included participants with CVD, but rather than exclude these trials, we included trials that contained 10% or fewer participants with documented CVD.
Primary prevention in this case is defined as prevention for patients without existing cardiovascular disease. There is evidence that statins may well have benefits that outweigh harms when given to patients with known cardiovascular disease, particularly documented coronary artery disease (CAD). Mixing such patients in any numbers into "primary prevention trials" would likely exaggerate the benefits of statins. Yet such not quite primary prevention studies were included in a systematic review of primary prevention.
In addition, a commentary by one of the guideline developers defending the group's work also underscored the fact that many of the supposedly primary prevention trials they used as evidence were not pure primary prevention trials.(3)
Notably, the 2013 cholesterol guideline cut points were derived from the placebo rates for myocardial infarction, stroke, and cardiovascular disease death observed in the 3 exclusively primary prevention statin trials, Air Force/Texas Coronary Atherosclerosis Prevention Study (AFCAPS/TexCAPS), Management of Elevated Cholesterol in the Primary Prevention Group of Adult Japanese (MEGA) study, and the Justification for the Use of Statins in Primary Prevention: an Intervention Trial Evaluating Rosuvastatin (JUPITER).[4-6]
The wording again suggests that all the other trials used as evidence about primary prevention were NOT "exclusively primary prevention studies," and hence, as I noted above, may have been biased so that they would likely exaggerate the apparent benefits of statins in primary prevention.
This suggested to me that the systematic review that provided a basis for the guidelines' aggressive recommendations about statins for primary prevention, and the trials on which it was based needed further skeptical, critical reviews.
The Cochrane Review: Was Evidence about Statin Benefits vs Harms Manipulated?
We have frequently discussed the manipulation of clinical research. By that we have meant design, implementation, analysis or dissemination of research in ways likely to further vested interests. In particular, when drug, biotechnology and device companies sponsor and control clinical research on their own products, they may set up the research in ways likely to make their products look better than they actually are.
We looked through the most recent Cochrane review of statins for primary prevention(7) with a skeptical eye out for suggestions that the trials reviewed could have been so manipulated. We found quite a bit.
First, we found that not all trials reported on the "hard" outcomes that one needs to consider when evaluating benefit vs harm of statins in primary prevention.
Data on all-cause mortality were provided in 11 trials.[of 19, and hence missing in 8].
And,
Twelve trials provided data on adverse events.[and hence 7 did not.]
Also, data on specific adverse events was often not reported: myalgia and rhabdomyolysis were reported in only 9/19 trials; new onset diabetes mellitus in only 2/18; hemmorhagic stroke in only 2/19; abnormal liver tests in only 10/19; kidney dysfunction in only 4/19; arthritis in only 2/19, and by implication, cognitive dysfunction in 0/19.
Failure to look for all the possible bad outcomes of treatment could obviously bias the study in the direction of minimizing the harms of the treatment.
A substantial number of trials either failed to report on crucial aspects of their methods, or admitted to flaws that could have induced important biases.: 3/19 did not described randomization methods; 4/19 did not use double-blinding; 6/19 did not use intention to treat analysis; 7/19 did not report their drop-out rates.
So it is very surprising to me that the authors concluded,
In general there was low risk of bias ... though all trials were either fully or partially funded by pharmaceutical companies.In my humble opinion, the Cochrane review showed many trials that had flaws could have biased their outcomes, and hence the outcomes of the overall review. Some of the flaws clearly could have lead to biases that would have made statins look more efficacious, or less dangerous than they might actually be. I do not understand the conclusion that the risk of bias was slow, and the lack of discussion about the direction the bias could have taken.
Review of the "3 Exclusively Primary Prevention Statin Trials"
Given the Cochrane review's apparent lack of skepticism about methodologic problems in the industry funded statin prevention trials, I endeavored to take a closer look at the three trials that Dr Robinson held out as the real primary prevention trials. Instances of manipulation, as we defined it above, for each trial are described below
AFCAPS/ TexCAPS(4)
Narrow Patient Population - This study excluded many patient for whom the statins were not contraindicated or warned against: uncontrolled hypertension; type 1 or type 2 diabetes mellitus on insulin or with a HgBA1C at least 10%; and body weight more than 50% "desirable limit for height." (Based on the official contraindications and warnings for commonly used statins, e.g., see contraindications for Lipitor here, active liver disease, pregnancy for likely to become pregnant, nursing mothers, hypersensitivity to the medicine; and warnings: use of cyclosprine or strong CYP3A4 inhibitors, uncontrolled hypothyroidism, renal impairment.) Thus the results may not generalize to many patients who would otherwise be considered statin candidates. By excluding such patients, the results may bias the study towards minimizing the probabilities of harms that might occur were statins used on a wide population for primary prevention.
Unknown Randomization and Allocation Concealment Procedures - According to the Cochrane Review, the study report did not explain how randomization or allocation concealment were accomplished.
Early Termination/ Multiple Comparisons - The study was terminated early based on an early look at the number of outcome events. Two such early or interim analyses were planned. Taking multiple looks at the data over time raises a multiple comparisons problem, and may lead to exaggerating the benefits of the treatment.(8). Furthermore, stopping early decreases the sample size and hence the power to find adverse effects of treatment.
Implausible Dropout Rate, Missing Data - According to the Cochrane Review, the study reported no dropouts. This seems somewhat improbable, suggesting skepticism about the accuracy and completeness of the data collection. On the other hand, a study chronology suggests that of 6605 patients who started the study, 6540 had data on complete endpoint status, suggesting missing data. Since dropouts and missing data may be due to different reasons in different arms of the study, they threaten the validity of data about benefits and harms.
Adverse Effects not Reported - The study provided no data about development of diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, suggesting incomplete data about harms, and hence bias towards minimizing harms.
MEGA (5)
Narrow Patient Population - [The patient population was not described in the main report, but only in an earlier methods article.](9) The study excluded patients with congenital or rheumatic heart disease; chronic atrial fibrillation, current diagnosis of malignancy; poorly controlled hypertension or diabetes mellitus; current use of oral or parenteral corticosteroids; and other conditions at the discretion of the physician. These exclusions seem unrelated to the contraindications or warnings on the stain label. Again, such a narrow patient population reduces the generalizability of the study results, and may bias the study to minimizing the harms of statins.
Only Single Blind - This was an open-label study, so patients and physicians knew who got statins and who got placebo. Such knowledge could have biased patient management, including how diligently particular diagnoses and outcomes were pursued, and biased data collected from patients or physicians.
Adverse Effects Not Reported - The study provided no data about diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms.
JUPITER(6)
Narrow, Unusual Patient Population - The study was limited to patients without hyperlipidemia but with an increased C-reactive protein. Thus it is not clear that its results would generalize to a more typically defined primary prevention population. The study excluded patients receiving post-menopausal hormone-replacement; with diabetes; uncontrolled hyertension; cancer other than non-melanoma skin cancer within 5 years; recent history of drug or alcohol abuse; "another medical condition that might compromise safety or the successful completion of the study;" also patients with "inflammatory conditions such as severe arthritis, lupus, or inflammatory bowel disease...;" and also "patients taking immunosuppressant agents such as cyclosporine, tacrolimus, azathioprine, or long-term oral glucocorticoids." Again, this narrow patient population would reduce generalizability and bias towards minimizing harms.
Early Termination/ Multiple Comparisons - This study was terminated early after an early look at the data. Allowing for multiple looks at the data may exaggerate efficacy.
Implausible Dropout Rate - According to the Cochrane Review, the study had no dropouts. This value seems implausible, again suggesting data collection problems.
Adverse Effects Not Reported - The study provided no data about
Summary
Aspects of the continuing controversy over the new ACC/ AHA guidelines for statins in the primary prevention of cardiovascular disease hinted that the clinical trials which provided the evidentiary basis for the guidelines, and for the use of statins in primary prevention in general, was more flawed than is widely appreciated. The latest Cochrane Collaboration review of this data acknowledged multiple, important flaws affecting most of the studies. Our more detailed review of the three studies held out as the purest found additional flaws. Many of these flaws seemed likely to bias the studies towards exaggerating the efficacy and/or minimizing the harms of statins in primary prevention. Since all these trials were funded, and presumably influenced by pharmaceutical companies that make statins, these flaws seem to be examples of manipulation of the clinical evidence. Rather than being the result of simple mistakes, or inevitable trade-offs, they seem to be study features intended the support the vested interests of the study sponsors.
It is not clear why the Cochrane review did not temper its conclusions based on the flaws in the studies, and particularly by the possibility that these flaws represented study manipulation.
The multiple flaws, possibly due to study manipulation, in the clinical evidence about statins in primary prevention suggest that we should be extremely skeptical about whether the benefits of such treatment outweighs its harms, and hence about whether the recommendations in the latest guidelines to give statins to all patients predicted (but perhaps not accurately) to be at even slightly elevated risk are warranted.
The flaws in multiple large studies of a very common clinical problem, and their effects on systematic reviews and clinical practice guidelines suggest that suppression and manipulation of research are rife in medicine and health care, presumably fueled by the pervasive web of conflicts of interest that spans health care. We need extreme skepticism about the integrity of clinical research, especially research sponsored by those whose products and services are being studied, and who thus have vested interests in the research turning out to make their products and services look good.
The good news is that we may not have to look too far to find ways to improve the trustworthiness of guidelines and the soundness of medical decision making. Implementation of the Institute of Medicine's recommendations on reducing conflicts of interest (look here), and developing trustworthy guidelines (look here) might lead to the development of sound guidelines in the future.
Furthermore, while endless discoveries of manipulated and suppressed research may have lead some evidence-based medicine advocates to despair, our latest exercise suggests that the principles of evidence-based medicine, unflinchingly applied, could really do good. Review of the three statin studies above based on standard principles of critical review readily spotted the multiple signs of manipulation. The problem with the Cochrane review was not that it missed these signs. Rather, the reviewers for some reason noted most of them, but then did not react. If systematic reviews were done with sufficient skepticism about the possibility of manipulation of clinical research, and were willing to call out when the emperor seemed short on fabric, then a lot of mischief could be avoided.
References
1. Lenzer J. Majority of panelists on controversial new cholesterol guideline have current or recent ties to drug manufacturers. Brit Med J 2013. Link here.
2. Taylor FC, Huffman M, Shah E. Statin therapy for primary prevention of cardiovascular disease. JAMA 2013. Link here.
3. Robinson JG. Accumulating evidence for statins in primary prevention. JAMA 2013; Link here.
4. Downs JR, Clearfield M, Weis S, et al; for the AFCAPS/TexCAPS Research Group. Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. JAMA. 1998;279(20):1615-1622. Link here.
5. Nakamura H, Arakawa K, Itakura H, et al; MEGA Study Group. Primary prevention of cardiovascular disease with pravastatin in Japan (MEGA study): a prospective randomised controlled trial. Lancet. 2006;368(9542):1155-1163. Link here.
6. Ridker PM, Danielson E, Fonseca FA, et al; JUPITER Study Group. Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein. N Engl J Med. 2008;359(21):2195-2207. Link here.
7. Taylor F, Huffman MD, Macedo AF et al. Statins for the prevention of cardiovascular disease. Cochrane Library 2013. Link here.
8. Mueller PS, Montori VM, Bassler D et al. Ethical issues in stopping randomized trials early because of apparent benefit. Ann Intern Med 2007; 146: 878-881. Link here.
9. Management of Elevated Cholesterol in the Primary Prevention Group of the Adult Japanese (MEGA) Study Group. Design and baseline characteristics of a study of primary prevention of coronary events with pravastatin among Japanese with mildly elevated cholesterol levels. Circ J 2004; 68: 860-867. Link here.
Friday, November 22, 2013
Confused Thinking about New Cholesterol Guidelines - Were Conflicts of Interest to Blame?
For years, clinical practice guidelines promulgated by prominent health care organizations have been hailed with accolades as received wisdom. However, there is increasing reason to be skeptical of such guidelines. Many guidelines are not based on rigorous application of the principles of evidence-based medicine, and often seem to arise from the personal opinions of their authors. This is particularly troublesome when those authors have conflicts of interest, and when the organizations that sponsor guideline development have institutional conflicts of interest. Back in 2011, an Institute of Medicine panel advocated standards for guideline development, including strict limits on conflicted panel members, to make their results more trustworthy. However, as we noted here, those standards have been largely ignored.
Therefore, it is good news that the just released, long awaited guidelines on the treatment of blood cholesterol from the American College of Cardiology (ACC) and American Heart Association (AHA)(1) they provoked controversy rather than adulation. However, connecting some dots reveals that the guideline development process and the defenses of the guidelines by its developers were even more confused than they first seemed. That confusion may be explained by conflicts of interest affecting guideline development of the sort that the IOM report wanted eliminated. .
The New Cholesterol Guidelines - Background
A striking feature of the guidelines was a new approach to drug treatment for primary prevention, that is, for patients who do not already have heart disease or other atherosclerosis. Such drug based primary prevention has been controversial, although drug treatment for people with high cholesterol who also have documented coronary heart disease, secondary prevention, is well-established.
The new cholesterol treatment guideline suggested drug treatment, essentially limited to statin medications, for patients believed to be at elevated risk of developing coronary artery disease or other forms of atherosclerotic disease, even in the absence of elevated cholesterol levels.
Also,
Previously, guidelines and other recommendations for the treatment of cholesterol for primary prevention, that is, for patients without known atherosclerotic cardiovascular disease, suggested treatments according to the level of cholesterol or its components.
Less controversially, the guidelines recommended treatment for patients with existing ASCVD, very high LDL-C, and diabetes.
The new guidelines raise some questions:
- Given the past controversy, how good is the evidence supporting cholesterol lowering drug treatment in primary prevention?
- What is the evidence supporting deciding on drug treatment for primary prevention on predicted risk of atherosclerotic cardiovascular disease?
- Can physicians make good enough risk predictions to use this approach?
Evidence Supporting Drug Treatment of Cholesterol in Primary Prevention - Do Benefits Outweigh Harms?
The reason that cholesterol lowering drug use for primary prevention has been controversial is the lack of clear evidence that such drug use leads to benefits to patients that outweigh its harms. A central principle of evidence based medicine is that only treatments whose benefits clearly outweigh their harms should be prescribed. .
A recent commentary by Abramson et al in the British Medical Journal in October, 2013 outlines the issues.(2) There is no good evidence that statins used in primary prevention increase overall survival, or decrease overall incidence of adverse events, defined as death, hospital admission, prolongation of admission, cancer or permanent disability.
Individual trials and meta-analyses do show that statins lead to a small reduction in the rate of cardiovascular events. For example, the authors' re-analysis of data from a patient level meta-analysis showed that of 140 low-risk primary prevention patients treated for five years, one patient would avoid a major coronary event or stroke.
However, it is not clear that this small likelihood of benefit offsets the likelihood of adverse events due to treatment. The data about the harms of statins in primary prevention is not very clear, partially because the relevant randomized controlled trials featured "reporting of adverse events ... [that was] generally poor, 'with failure to provide details of severity and type of adverse events or to report on health-related quality of life."
In summary, Abramson et al wrote,
However, the new guideline focused on the ability of statins to prevent ASCVD,
Furthermore, the guideline seemed unreasonably optimistic about the harms of statins. In particular, it ignored evidence about harms other than diabetes, myopathy (serious muscle disease), and stroke. Yet there is evidence suggesting that statins at least might cause "liver dysfunction, acute renal failure, and cataracts; cognitive symptoms, neuropathy, and sexual dsyfunction; decreased energy and exertional fatigue; and psychiatric symptoms, including depression, memory loss, confusion, and aggressive reactions" as summarized by Abramson and colleagues.
Thus the new guidelines did not make a new and improved case that statin treatment for primary prevention has benefits that outweigh its harms. One could argue that this is their fatal flaw and thus all the rest of the guidelines' discussion about which patients should get primary prevention was pointless.
Dr Abramson and his co-author, Dr Rita Redberg, did get to repeat their arguments about why statin use for primary prevention is not justified in a NY Times op-ed, but otherwise the fundamental problem with the guideline's argument for aggressive use of statins went unnoticed.
Evidence Supporting Making Decisions about Statin Treatment as Primary Prevention According to Patients' Risks of Developing Atherosclerotic Cardiovascular Disease
The guidelines suggested statin therapy for patients judged to have at least a 7.5% risk of ASCVD. As noted above, there is no evidence that statin therapy in primary prevention in general increases survival, reduces serious events overall, improves health status, quality of life, or function, or has benefits that clearly outweigh its harms. I could not find any reference in the guidelines to clear evidence about the effects of statin treatment in primary prevention for this sub-group of patients.
In the supplemental guideline on risk assessment,(3) there was this
The wording is confusing, but might be stating that a strategy of using statins only for patients predicted to have a risk greater than 7.5% has not been assessed in a clinical trial. (It might also, however, be stating that the specific method recommended by the guideline to assess this risk has never been tested in a clinical trial, see below.)
The apparent lack of evidence in support of the specific strategy advocated by the guideline to use statins for primary prevention in patients whose risk of ASCVD exceeded the designated threshold did not otherwise attract any notice.
Evidence Supporting the Ability of Physicians to Assess Risk of ASCVD Accurately Enough to Implement the Recommended Strategy
The supplemental guideline stated that previously published methods of risk prediction would not be suitable for the new proposed approach,
Since the recommended strategy requires an estimation of risk, the lack of availability of acceptable risk assessment methods would appear to be another fatal flaw. However,
So the question now becomes: did the guideline provide sufficient evidence that the new risk prediction tool developed as part of guideline development predicts sufficiently well to be used to make decisions as recommended by the guidelines?
Note that developing, validating, and publishing a new risk score normally would be considered to be tasks that are part of research, not guideline development. Nonetheless, the guideline developers decided to take on these tasks as part of guideline development, which precluded independent publication of the the results of this research after peer review. This makes it more difficult for others to critically evaluate the results of the research concerning the new risk score. But let me attempt to do so.
The work group went ahead to develop a new multivariate prediction model for ASCVD. This means they used statistical methods to find variables, that is, patients' clinical or demographic characteristics, that independently could predict the outcome of interest, and then combined these variables in an equation (or algorithm) to make risk predictions for individual patients. Such multivariate models to make diagnoses or, as in this case, prognostic predictions have been the subject of considerable research since at least the 1970s. However, they have not been as useful as their initial advocates hoped.
The issues turn out to be a bit complex. I must digress into an area in which I previously did research.
It is quite easy to develop multivariate diagnostic or prediction models. The statistical analytic tools can almost always find multiple patient characteristics that independently correlate with the outcome of interest. The problem is that such correlations can be based on random associations, or biases produced by idiosyncrasies in the particular data set used for model development. That a new model can diagnose or predict accurately for patients who were not in the original data set thus is not assured, and should be considered to be merely a hypothesis.
Models developed on one group of patients often do not work when tried prospectively on new patients, probably because the initial development group was somehow not completely representative of all the patients of interest. For example, a model may have been developed using patients from a particular hospital which attracts different sorts of patients than found in other hospitals in which the model might be used. .Therefore, before one has confidence in such a multivariate model, one must verify that the model predicted or diagnosed well not only in the group of patients from which it was derived, but prospectively on other patients like those on whom it would be used in clinical practice.
The supplemental guidelines did assert that the new model was prospectively tested,
However, it did not specify what these cohorts were.
There is some data buried in Appendix 4 of the supplemental guideline on the performance of the model. They did not distinguish results from derived from prospective validation from those from model derivation.
These two sentences do not provide strong evidence that the model would predict well when used prospectively.
The C-statistic is an overall measure of the ability of the model to discriminate between patients who will go on to develop the diseases of interest and those who will not. The statistic is formally equal to the probability that were two patients, one who developed disease, and another who did not, selected randomly, the model would predict disease more strongly for the patient who actually got it. Thus, at best, for 20% of random pairs of African-American women, one who got disease, one who did not, the model would give a more pessimistic prediction for the women who would not get disease. This is good, but not great discrimination ability. (If this result referred to data from model derivation, not prospective validation, it is likely to be over-optimistic.)
Furthermore, since the guideline would be used to make decisions based on the absolute risk, its calibration is also important. Calibration is the measure of whether the model predictions of risk are close to reality. To assess calibration, one ought to assess the whole range of predictions made by the model. Given that the guidelines suggest a 7.5% risk threshold, it would be particularly important to determine whether patients given predictions above and below that value really have risks above and below that value.
Unfortunately, the two sentences above are not helpful in this regard. The chi-square statistic presented is a measure of overall calibration, but does not show calibration for groups of patients given predictions with particularly interesting values, like above or below 7.5%
Note that the supplemental guideline on risk assessment includes a statement that details about the model validation done "internally and externally" are in a Full Panel Report Data Supplement. The download of that supplement so far does not seem to work
Setting that aside for a moment, the published guidelines do not provide good evidence that the risk prediction tool the guidelines recommend should be used to assess ASCVD risk in order to make decisions on statin use for individual patients in primary prevention. The lack of a sufficiently accurate method to predict risk seems to a fatal flaw for a strategy that requires risk prediction.
Controversy in the Media
It turns out that I was not the first person to identify this problem. In fact, it appears that during the internal review process for the guidelines, major questions had already been raised about the risk prediction model's calibration. However, these questions did not seem to have been conveyed to the guideline authors, and hence were never addressed.
As reported by the New York Times on November 17, 2013,
The article by Ridker and Cook was indeed published. on November 19, 2013(4). It suggested major problems with the calibration of the risk assessment model,
Note that Figure 1 of the Ridker and Cook article showed the calibration of the model in the patient cohorts newly tested by these authors, focusing in particular on whether patients predicted to have risks of cardiovascular disease just over the 7.5% threshold actually had rates of such disease greater than 7.5%. They clearly did not Note also that Figure 2 showed calibration of the model when the guideline authors attempted to test it on new patient cohorts, apparently the data found in the so far inaccessible Full Panel Report Data Supplement. Again, the model overestimated risk for patients just over the crucial threshold.
So the guideline developers knew that their model overestimated risk, buried this information in supplemental data, did not admit or perhaps appreciate how it threatened the credibility of their guidelines, and somehow were not given the internal review that suggested this was a fatal flaw of the proposed guidelines .
The Guideline Developers' Responses
The media controversy over the accuracy of the prediction model incorporated into the new guidelines provoked responses from the guideline developers, but these responses were at best confused. .
First, as reported by the NY times,
Thus, after promulgating guidelines that seemed hardly based on evidence, when challenged, the guideline developers' response was confused and illogical.
What About the Issue of Benefits versus Harms?
The other problems that I identified above, lack of evidence that primary prevention provides benefits that clearly outweigh harms, and lack of evidence that a strategy based on risk assessment would provide benefits that outweighs harms, did not get media coverage, and so did not provoke a response from the developers. However, I did find that one of the members of the guideline panel implied her approach to the benefits versus harms issue in a Medscape news article. Dr Noel Bairey Merz gave a talk at the 2013 American Heart Association meeting about primary prevention for women, first saying
Also,
So Dr Merz seemed to assume that statins work for particular patients in the absence of evidence that they do not work, maybe implying a general assumption that all treatments are beneficial until proven otherwise. This stands a central precept of evidence-based medicine, and perhaps the ancient dictum to physicians to do no harm, on their heads.
References
1. Stone NJ, Robinson J, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2013. Link here.
2. Abramson JD, Rosenberg HG, Jewell N et al. Should statins be prescribed to people at low risk of cardiovascular disease. Brit Med J 2013; 347: 15-17. Link here.
3. Goff Jr DC, Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA Guideline on the Assessment of Cardiovascular
Risk. Journal of the American College of Cardiology (2013), doi: 10.1016/j.jacc.2013.11.005. Link here.
4. Rikder PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013; Link here.
5. Collier J. The price of independence. Br Med J 2006; 332: 1447-9. Link here
Therefore, it is good news that the just released, long awaited guidelines on the treatment of blood cholesterol from the American College of Cardiology (ACC) and American Heart Association (AHA)(1) they provoked controversy rather than adulation. However, connecting some dots reveals that the guideline development process and the defenses of the guidelines by its developers were even more confused than they first seemed. That confusion may be explained by conflicts of interest affecting guideline development of the sort that the IOM report wanted eliminated. .
The New Cholesterol Guidelines - Background
A striking feature of the guidelines was a new approach to drug treatment for primary prevention, that is, for patients who do not already have heart disease or other atherosclerosis. Such drug based primary prevention has been controversial, although drug treatment for people with high cholesterol who also have documented coronary heart disease, secondary prevention, is well-established.
The new cholesterol treatment guideline suggested drug treatment, essentially limited to statin medications, for patients believed to be at elevated risk of developing coronary artery disease or other forms of atherosclerotic disease, even in the absence of elevated cholesterol levels.
Adults 40 to 75 years of age with LDL–C [so called "bad cholesterol"] ]70 to 189 mg/dL, without clinical ASCVD* [atherosclerotic cardiovascular disease] or diabetes and an estimated 10-year ASCVD risk ≥7.5% should be treated with moderate- to high-intensity statin therapy.
Also,
It is reasonable to offer treatment with a moderate intensity statin to adults 40 to 75 years of age, with LDL–C 70 to 189 mg/dL, without clinical ASCVD* or diabetes and an estimated 10-year ASCVD risk of 5% to [less than] ... 7.5%
Previously, guidelines and other recommendations for the treatment of cholesterol for primary prevention, that is, for patients without known atherosclerotic cardiovascular disease, suggested treatments according to the level of cholesterol or its components.
Less controversially, the guidelines recommended treatment for patients with existing ASCVD, very high LDL-C, and diabetes.
The new guidelines raise some questions:
- Given the past controversy, how good is the evidence supporting cholesterol lowering drug treatment in primary prevention?
- What is the evidence supporting deciding on drug treatment for primary prevention on predicted risk of atherosclerotic cardiovascular disease?
- Can physicians make good enough risk predictions to use this approach?
Evidence Supporting Drug Treatment of Cholesterol in Primary Prevention - Do Benefits Outweigh Harms?
The reason that cholesterol lowering drug use for primary prevention has been controversial is the lack of clear evidence that such drug use leads to benefits to patients that outweigh its harms. A central principle of evidence based medicine is that only treatments whose benefits clearly outweigh their harms should be prescribed. .
A recent commentary by Abramson et al in the British Medical Journal in October, 2013 outlines the issues.(2) There is no good evidence that statins used in primary prevention increase overall survival, or decrease overall incidence of adverse events, defined as death, hospital admission, prolongation of admission, cancer or permanent disability.
Individual trials and meta-analyses do show that statins lead to a small reduction in the rate of cardiovascular events. For example, the authors' re-analysis of data from a patient level meta-analysis showed that of 140 low-risk primary prevention patients treated for five years, one patient would avoid a major coronary event or stroke.
However, it is not clear that this small likelihood of benefit offsets the likelihood of adverse events due to treatment. The data about the harms of statins in primary prevention is not very clear, partially because the relevant randomized controlled trials featured "reporting of adverse events ... [that was] generally poor, 'with failure to provide details of severity and type of adverse events or to report on health-related quality of life."
In summary, Abramson et al wrote,
statin therapy prevents one serious cardiovascular event per 140 low risk people (five year risk ... [less than] 10%) treated for five years. Statin therapy in low risk people does not reduce all cause mortality or serious illness and has about an 18% risk of causing side effects that range from minor and reversible to serious and irreversible. Broadening the recommendations in cholesterol lowering guidelines to include statin therapy for low risk individuals will unnecessarily increase the incidence of adverse effects without providing overall health benefit.
However, the new guideline focused on the ability of statins to prevent ASCVD,
The RCTs identified in the systematic evidence review indicated a consistent reduction in ASCVD eventsHowever, the new guideline ignored the absence of evidence that statin treatment prolonged life, prevented serious illness overall, or improved health status, function or quality of life.
from 3-hydroxy-3-methylglutaryl-coenzyme A reductase inhibitors (statins) therapy in secondary and primary prevention populations....
Furthermore, the guideline seemed unreasonably optimistic about the harms of statins. In particular, it ignored evidence about harms other than diabetes, myopathy (serious muscle disease), and stroke. Yet there is evidence suggesting that statins at least might cause "liver dysfunction, acute renal failure, and cataracts; cognitive symptoms, neuropathy, and sexual dsyfunction; decreased energy and exertional fatigue; and psychiatric symptoms, including depression, memory loss, confusion, and aggressive reactions" as summarized by Abramson and colleagues.
Thus the new guidelines did not make a new and improved case that statin treatment for primary prevention has benefits that outweigh its harms. One could argue that this is their fatal flaw and thus all the rest of the guidelines' discussion about which patients should get primary prevention was pointless.
Dr Abramson and his co-author, Dr Rita Redberg, did get to repeat their arguments about why statin use for primary prevention is not justified in a NY Times op-ed, but otherwise the fundamental problem with the guideline's argument for aggressive use of statins went unnoticed.
Evidence Supporting Making Decisions about Statin Treatment as Primary Prevention According to Patients' Risks of Developing Atherosclerotic Cardiovascular Disease
The guidelines suggested statin therapy for patients judged to have at least a 7.5% risk of ASCVD. As noted above, there is no evidence that statin therapy in primary prevention in general increases survival, reduces serious events overall, improves health status, quality of life, or function, or has benefits that clearly outweigh its harms. I could not find any reference in the guidelines to clear evidence about the effects of statin treatment in primary prevention for this sub-group of patients.
In the supplemental guideline on risk assessment,(3) there was this
After deliberation, the Work Group endorsed the existing and widely employed paradigm of matching the intensity of preventive efforts with the individual’s absolute risk.. The Work Group acknowledges that none of the risk assessment tools or novel risk markers examined in the present document have been formally evaluated in randomized controlled trials of screening strategies with clinical events as outcomes.
The wording is confusing, but might be stating that a strategy of using statins only for patients predicted to have a risk greater than 7.5% has not been assessed in a clinical trial. (It might also, however, be stating that the specific method recommended by the guideline to assess this risk has never been tested in a clinical trial, see below.)
The apparent lack of evidence in support of the specific strategy advocated by the guideline to use statins for primary prevention in patients whose risk of ASCVD exceeded the designated threshold did not otherwise attract any notice.
Evidence Supporting the Ability of Physicians to Assess Risk of ASCVD Accurately Enough to Implement the Recommended Strategy
The supplemental guideline stated that previously published methods of risk prediction would not be suitable for the new proposed approach,
As part of its deliberations, the Work Group considered previously published risk scores with validation in NHLBI cohort data as 1 possible approach. However, a number of persistent concerns with existing risk equations were identified including nonrepresentative or historically dated populations, limited ethnic diversity, narrowly defined endpoints, endpoints influenced by provider preferences (e.g., revascularizations), and endpoints with poor reliability (e.g., angina and heart failure [HF]).
Since the recommended strategy requires an estimation of risk, the lack of availability of acceptable risk assessment methods would appear to be another fatal flaw. However,
Given the inherent limitations of existing scores, the Work Group judged that a new risk score was needed to address some of the deficiencies of existing scores, such as utilizing a population sample that approaches, to the degree possible, the ideal sample for algorithm development and closely represents the U.S. population.
So the question now becomes: did the guideline provide sufficient evidence that the new risk prediction tool developed as part of guideline development predicts sufficiently well to be used to make decisions as recommended by the guidelines?
Note that developing, validating, and publishing a new risk score normally would be considered to be tasks that are part of research, not guideline development. Nonetheless, the guideline developers decided to take on these tasks as part of guideline development, which precluded independent publication of the the results of this research after peer review. This makes it more difficult for others to critically evaluate the results of the research concerning the new risk score. But let me attempt to do so.
The work group went ahead to develop a new multivariate prediction model for ASCVD. This means they used statistical methods to find variables, that is, patients' clinical or demographic characteristics, that independently could predict the outcome of interest, and then combined these variables in an equation (or algorithm) to make risk predictions for individual patients. Such multivariate models to make diagnoses or, as in this case, prognostic predictions have been the subject of considerable research since at least the 1970s. However, they have not been as useful as their initial advocates hoped.
The issues turn out to be a bit complex. I must digress into an area in which I previously did research.
It is quite easy to develop multivariate diagnostic or prediction models. The statistical analytic tools can almost always find multiple patient characteristics that independently correlate with the outcome of interest. The problem is that such correlations can be based on random associations, or biases produced by idiosyncrasies in the particular data set used for model development. That a new model can diagnose or predict accurately for patients who were not in the original data set thus is not assured, and should be considered to be merely a hypothesis.
Models developed on one group of patients often do not work when tried prospectively on new patients, probably because the initial development group was somehow not completely representative of all the patients of interest. For example, a model may have been developed using patients from a particular hospital which attracts different sorts of patients than found in other hospitals in which the model might be used. .Therefore, before one has confidence in such a multivariate model, one must verify that the model predicted or diagnosed well not only in the group of patients from which it was derived, but prospectively on other patients like those on whom it would be used in clinical practice.
The supplemental guidelines did assert that the new model was prospectively tested,
The equations were also assessed in external validation studies using data from other available cohorts
However, it did not specify what these cohorts were.
There is some data buried in Appendix 4 of the supplemental guideline on the performance of the model. They did not distinguish results from derived from prospective validation from those from model derivation.
In summary, discrimination and calibration of the models were very good. C statistics ranged from a low of 0.713 (African-American men) to a high of 0.818 (African-American women). Calibration chi-square statistics ranged from a low of 4.86 (nonHispanic White men) to a high of 7.25 (African-American women).
These two sentences do not provide strong evidence that the model would predict well when used prospectively.
The C-statistic is an overall measure of the ability of the model to discriminate between patients who will go on to develop the diseases of interest and those who will not. The statistic is formally equal to the probability that were two patients, one who developed disease, and another who did not, selected randomly, the model would predict disease more strongly for the patient who actually got it. Thus, at best, for 20% of random pairs of African-American women, one who got disease, one who did not, the model would give a more pessimistic prediction for the women who would not get disease. This is good, but not great discrimination ability. (If this result referred to data from model derivation, not prospective validation, it is likely to be over-optimistic.)
Furthermore, since the guideline would be used to make decisions based on the absolute risk, its calibration is also important. Calibration is the measure of whether the model predictions of risk are close to reality. To assess calibration, one ought to assess the whole range of predictions made by the model. Given that the guidelines suggest a 7.5% risk threshold, it would be particularly important to determine whether patients given predictions above and below that value really have risks above and below that value.
Unfortunately, the two sentences above are not helpful in this regard. The chi-square statistic presented is a measure of overall calibration, but does not show calibration for groups of patients given predictions with particularly interesting values, like above or below 7.5%
Note that the supplemental guideline on risk assessment includes a statement that details about the model validation done "internally and externally" are in a Full Panel Report Data Supplement. The download of that supplement so far does not seem to work
Setting that aside for a moment, the published guidelines do not provide good evidence that the risk prediction tool the guidelines recommend should be used to assess ASCVD risk in order to make decisions on statin use for individual patients in primary prevention. The lack of a sufficiently accurate method to predict risk seems to a fatal flaw for a strategy that requires risk prediction.
Controversy in the Media
It turns out that I was not the first person to identify this problem. In fact, it appears that during the internal review process for the guidelines, major questions had already been raised about the risk prediction model's calibration. However, these questions did not seem to have been conveyed to the guideline authors, and hence were never addressed.
As reported by the New York Times on November 17, 2013,
The problems were identified by two Harvard Medical School professors whose findings will be published Tuesday in a commentary in The Lancet, a major medical journal. The professors, Dr. Paul M. Ridker and Dr. Nancy Cook, had pointed out the problems a year earlier when the National Institutes of Health’s National Heart, Lung, and Blood Institute, which originally was developing the guidelines, sent a draft to each professor independently to review. Both reported back that the calculator was not working among the populations it was tested on by the guideline makers.
That was unfortunate because the committee thought the researchers had been given the professors’ responses, said Dr. Donald Lloyd-Jones, co-chairman of the guidelines task force and chairman of the department of preventive medicine at Northwestern University.
The article by Ridker and Cook was indeed published. on November 19, 2013(4). It suggested major problems with the calibration of the risk assessment model,
Another concern for clinicians is whether the new prediction algorithm created by the ACC/AHA correctly assesses the level of vascular risk. To be useful, prediction models must not only discriminate between individuals with and without disease, but must also calibrate well so that predicted risk estimates match as closely as possible the observed risk in external populations. We calculated predicted 10-year risks of the same atherosclerotic events using the new ACC/AHA risk prediction algorithm and compared these estimates with observed event rates in three large-scale primary prevention cohorts, the Women's Health Study, the Physicians' Health Study, and the Women's Health Initiative Observational Study.
As shown in figure 1, in all three of these primary prevention cohorts, the new ACC/AHA risk prediction algorithm systematically overestimated observed risks by 75–150%, roughly doubling the actual observed risk. As shown in figure 2, similar overestimation of risk was observed in two external validation cohorts used by the guideline developers themselves, an issue readily acknowledged in the report. Thus, on the basis of data from these five external validation cohorts, it is possible that as many as 40–50% of the 33 million middle-aged Americans targeted by the new ACC/AHA guidelines for statin therapy do not actually have risk thresholds that exceed the 7·5% threshold suggested for treatment. Miscalibration to this extent should be reconciled and addressed in additional external validation cohorts before these new prediction models are widely implemented. It is possible, for example, that the five external validation cohorts are more contemporary than the cohorts used in the risk prediction algorithm and thus reflect secular improvements in overall health and lifestyle patterns in the USA over the past 25 years.
Note that Figure 1 of the Ridker and Cook article showed the calibration of the model in the patient cohorts newly tested by these authors, focusing in particular on whether patients predicted to have risks of cardiovascular disease just over the 7.5% threshold actually had rates of such disease greater than 7.5%. They clearly did not Note also that Figure 2 showed calibration of the model when the guideline authors attempted to test it on new patient cohorts, apparently the data found in the so far inaccessible Full Panel Report Data Supplement. Again, the model overestimated risk for patients just over the crucial threshold.
So the guideline developers knew that their model overestimated risk, buried this information in supplemental data, did not admit or perhaps appreciate how it threatened the credibility of their guidelines, and somehow were not given the internal review that suggested this was a fatal flaw of the proposed guidelines .
The Guideline Developers' Responses
The media controversy over the accuracy of the prediction model incorporated into the new guidelines provoked responses from the guideline developers, but these responses were at best confused. .
First, as reported by the NY times,
In a response on Sunday, Dr. [Sidney C] Smith of the guidelines committee said the concerns raised by Dr. Cook and Dr. Ridker 'merit attention.'
But, he continued, 'a lot of people put a lot of thought into how can we identify people who can benefit from therapy.' Further, said Dr. Smith, who is also a professor of medicine at the University of North Carolina and a past president of the American Heart Association, 'What we have come forward with represents the best efforts of people who have been working for five years.'
Note that this response includes two logical fallacies. First, it contained a straw man argument. It appeared to respond to accusations no one made. Nobody accused the guideline developers of being lazy, not putting in much effort, or not devoting much thought to the effort. The response also included an implied appeal to authority: big experts came up with these guidelines so their opinions should be credited, even in the presence of data to the contrary.
Also, as reported by CNN,
However, 'I can't speak to whether the calculator is valid or not,' Dr. Robert Eckel, co-chair of the American Heart Association committee that wrote the new guidelines and the association's past president, told CNN. 'That needs to be determined.''We trusted that the calculator worked,' he said. 'We trusted that the calculator is valid.'
This was confusing. Maybe Dr Eckel meant it to be another appeal to authority, the authority in question being that of the work group that developed the risk prediction model.
Furthermore, again according to CNN,
Researchers apparently did not receive the professors' responses, Dr. Donald Lloyd-Jones, chairman of the committee that developed the equation, told the Times.But Lloyd-Jones told reporters Monday, 'There's nothing wrong with these equations.'Committee members were aware there could be 'overestimation of risk in some populations,' he said.
In addition,
'Our risk assessment guideline doesn't tell you what to do. ... It just evaluates risk,' he said.
I am not sure this even rises to the level of a logical fallacy. It appears to be pure denial. The whole point of the risk assessment tool was to determine whether a patient's risk is above or below the thresholds suggested by the treatment guideline, and thus to tell you what to do. There clearly is something wrong with "these equations." Using them appears to vastly overestimate risk, and thus implementation of the guideline would probably lead to vast over treatment of real people .
What About the Issue of Benefits versus Harms?
The other problems that I identified above, lack of evidence that primary prevention provides benefits that clearly outweigh harms, and lack of evidence that a strategy based on risk assessment would provide benefits that outweighs harms, did not get media coverage, and so did not provoke a response from the developers. However, I did find that one of the members of the guideline panel implied her approach to the benefits versus harms issue in a Medscape news article. Dr Noel Bairey Merz gave a talk at the 2013 American Heart Association meeting about primary prevention for women, first saying
Although the randomized clinical trial evidence supporting primary prevention with statin therapy in women is not perfect, 'the absence of data means negative data.'
That is the argument formulated by Dr Noel Bairey Merz (Cedars Sinai Medical Center, Los Angeles, CA), who spoke today here at the American Heart Association 2013 Scientific Sessions.
'How confident are we that statins do not save lives in the week before a heart attack, but they do save lives the week after a heart attack, for women and men?'
Also,
in the overall JUPITER study of 18 000 patients, there was no treatment benefit when women were studied as a subgroup. Merz argues that JUPITER is powered for the total sample size only, not for women alone. In addition, the statistical test for heterogeneity revealed the interaction by sex was not statistically significant.
'Pretty much all the subgroups fall beyond the statistically significant range,' said Merz. 'So should we withhold treatment for women, who now are the majority of victims of cardiovascular disease, because of low precision and a trial that was not designed to address or answer this question?'
So Dr Merz seemed to assume that statins work for particular patients in the absence of evidence that they do not work, maybe implying a general assumption that all treatments are beneficial until proven otherwise. This stands a central precept of evidence-based medicine, and perhaps the ancient dictum to physicians to do no harm, on their heads.
Unwarranted Enthusiasm for (Over) Treatment and Conflicts of Interest
The new cholesterol guidelines, and those who developed them, seem enthused about the treatment of cholesterol with statin drugs for primary prevention in the absence of evidence that such treatment produces benefits that outweigh its harms. They also seem enthused about basing treatment decisions on a statistical prognostic model that has not been shown to be accurate, and in fact which appears to be biased towards promoting excess treatment in the context in which it would be used. The excess enthusiasm occurred in spite of evidence, and at times in spite of logic.
One possible reason that the guideline developers got so enthused that they seemed unable to think straight appears to be their own conflicts of interest, as first publicly noted in a post on Pharmalot. Reviewing the disclosure forms provided with the guidelines revealed more detail.
Of the 13 people on the main treatment guideline panel who were not NHLBI staffers serving ex-officio, 7 had financial relationships with pharmaceutical companies that manufacture statins:
- Jennifer Robinson, co-chair, research funded by AstraZeneca and Merck;
- C Noel Bairey Merz, consulting for Abbott, Bristol-Myers Squib Novartis, and Pfizer;
- Robert H Eckel, consulting for Merck, Pfizer and Abbott;
- Anne Carol Goldberg, consulting for Abbott and Merck, research funded by Abbott, Merck, and Novartis;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer;
- Karol Watson, consulting for Abbott, AstraZeneca, Merck and Pfizer, research funded by Merck;
- Peter W F Wilson, consulting for and research funded by Merck.
Of the 10 expert reviewers for this panel, 3 had financial relationships with pharmaceutical companies that manufacture statins
- William Virgin Brown, consultant for Abbott, Bristol-Myers Squibb, and Pfizer;
- Matthew Ito, consultant for Kowa;
- Robert S Rosenson, consultant for Novartis and Pfizer.
Of the 11 people on the risk prediction panel who were not NHLBI staffers serving ex-officio, 5 had financial relationships with pharmaceutical companies that manufacture statins
-David C Goff Jr, co-chair, research funded by Merck;
- Raymond Gibbons, consultant for AstraZeneca;
- Jennifer Robinson, research funded by AstraZeneca, and Merck;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer [although these relationships were not listed as relevant to this panel, but found in the listing for the panel above];
- Peter W.F Wilson, consultant for and research funded by Merck.
Also, in the Abramson and Redberg NY Times op-ed, the authors noted
As noted on Pharmalot, the prevalence of conflicted panel members did not appear to conform to the standards for the development of trustworthy guidelines recently published by the Institute of Medicine:
Also, noted by the Los Angeles Times was this comment from Dr John Abramson, lead author of the commentary on statins in primary prevention(4),
As Joe Collier observed, "people who have conflicts of interest often find giving clear advice (or opinions) particularly difficult."(5)
This difficulty giving clear advice, when amplified by a guideline for a common problem supported by prestigious non-profit organizations, and promoted by vigorous public relations, could lead to "more than 45 million middle-aged Americans who do not have cardiovascular disease being recommended for consideration of statin therapy" (per Ridker and Cook[4]) unnecessarily, likely resulting in millions suffering unneeded side effects, and billions in costs.
Summary
Guidelines for management of a very common problem promulgated by a major medical society and a major disease oriented non-profit organization suggested a strategy that would vastly increase drug treatment of currently healthy patients. The strategy appears not to have been based on good evidence. When some of the problems with this evidence were pointed out, the guideline developers responded with illogic. Apparently many of the guideline developers have financial relationships with the drug companies that would most profit from increases in drug treatment as recommended by the guidelines Implementation of the new guidelines might results in millions of people in the US receiving unneeded drugs, with resultant side effects and costs. .
Do we need more examples of how conflicts of interest are causing the poor outcomes and excess costs that are wrecking our health care system? Do we need more excuses not to eliminate conflicts of interest from guideline development? Do we need more delay implementing the standards provided by the Institute of Medicine report on trustworthy guidelines? Do we need more excuses not to drastically reduce conflicts of interest affecting academic medicine, medical societies, and disease specific non-profits, specifically starting with the earlier (and so far generally disregarded) Institute of Medicine report on conflicts of interest in medicine?
While we in the US argue incessantly in the details of minor reforms of our supposed free health care market, we ignore the rot at its foundations. True health care reform would attack the conflicts of interest that have put money, not patients at the center of health care.
The new cholesterol guidelines, and those who developed them, seem enthused about the treatment of cholesterol with statin drugs for primary prevention in the absence of evidence that such treatment produces benefits that outweigh its harms. They also seem enthused about basing treatment decisions on a statistical prognostic model that has not been shown to be accurate, and in fact which appears to be biased towards promoting excess treatment in the context in which it would be used. The excess enthusiasm occurred in spite of evidence, and at times in spite of logic.
One possible reason that the guideline developers got so enthused that they seemed unable to think straight appears to be their own conflicts of interest, as first publicly noted in a post on Pharmalot. Reviewing the disclosure forms provided with the guidelines revealed more detail.
Of the 13 people on the main treatment guideline panel who were not NHLBI staffers serving ex-officio, 7 had financial relationships with pharmaceutical companies that manufacture statins:
- Jennifer Robinson, co-chair, research funded by AstraZeneca and Merck;
- C Noel Bairey Merz, consulting for Abbott, Bristol-Myers Squib Novartis, and Pfizer;
- Robert H Eckel, consulting for Merck, Pfizer and Abbott;
- Anne Carol Goldberg, consulting for Abbott and Merck, research funded by Abbott, Merck, and Novartis;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer;
- Karol Watson, consulting for Abbott, AstraZeneca, Merck and Pfizer, research funded by Merck;
- Peter W F Wilson, consulting for and research funded by Merck.
Of the 10 expert reviewers for this panel, 3 had financial relationships with pharmaceutical companies that manufacture statins
- William Virgin Brown, consultant for Abbott, Bristol-Myers Squibb, and Pfizer;
- Matthew Ito, consultant for Kowa;
- Robert S Rosenson, consultant for Novartis and Pfizer.
Of the 11 people on the risk prediction panel who were not NHLBI staffers serving ex-officio, 5 had financial relationships with pharmaceutical companies that manufacture statins
-David C Goff Jr, co-chair, research funded by Merck;
- Raymond Gibbons, consultant for AstraZeneca;
- Jennifer Robinson, research funded by AstraZeneca, and Merck;
- J Sanford Schwartz, consulting for Abbott, Merck and Pfizer, research funded by Pfizer [although these relationships were not listed as relevant to this panel, but found in the listing for the panel above];
- Peter W.F Wilson, consultant for and research funded by Merck.
Also, in the Abramson and Redberg NY Times op-ed, the authors noted
both the American Heart Association and the American College of Cardiology, while nonprofit entities, are heavily supported by drug companies
As noted on Pharmalot, the prevalence of conflicted panel members did not appear to conform to the standards for the development of trustworthy guidelines recently published by the Institute of Medicine:
whenever possible, guideline development group members should not have conflicts of interest… and the chair or co-chairs should not be a person(s) with conflicts of interest.
Also, noted by the Los Angeles Times was this comment from Dr John Abramson, lead author of the commentary on statins in primary prevention(4),
'There is overtreatment that’s been built into the risk calculator, and this is a warning sign about the overtreatment that’s built into the guidelines themselves and the conflicts of interest in the organizations that are overseeing the production of these guidelines,' said Dr. John Abramson, a Harvard University cardiologist who has argued that statins offer little value for people with a 10-year risk level of heart attack or stroke of less than 20%. 'There aren’t brakes being put on the enthusiasm and overreaching of the experts.'
'There are statin believers, and when you hear these experts talk, they’re talking emotionally, not scientifically,' Abramson added. 'The experts are using emotion, not science.'
As Joe Collier observed, "people who have conflicts of interest often find giving clear advice (or opinions) particularly difficult."(5)
This difficulty giving clear advice, when amplified by a guideline for a common problem supported by prestigious non-profit organizations, and promoted by vigorous public relations, could lead to "more than 45 million middle-aged Americans who do not have cardiovascular disease being recommended for consideration of statin therapy" (per Ridker and Cook[4]) unnecessarily, likely resulting in millions suffering unneeded side effects, and billions in costs.
Summary
Guidelines for management of a very common problem promulgated by a major medical society and a major disease oriented non-profit organization suggested a strategy that would vastly increase drug treatment of currently healthy patients. The strategy appears not to have been based on good evidence. When some of the problems with this evidence were pointed out, the guideline developers responded with illogic. Apparently many of the guideline developers have financial relationships with the drug companies that would most profit from increases in drug treatment as recommended by the guidelines Implementation of the new guidelines might results in millions of people in the US receiving unneeded drugs, with resultant side effects and costs. .
Do we need more examples of how conflicts of interest are causing the poor outcomes and excess costs that are wrecking our health care system? Do we need more excuses not to eliminate conflicts of interest from guideline development? Do we need more delay implementing the standards provided by the Institute of Medicine report on trustworthy guidelines? Do we need more excuses not to drastically reduce conflicts of interest affecting academic medicine, medical societies, and disease specific non-profits, specifically starting with the earlier (and so far generally disregarded) Institute of Medicine report on conflicts of interest in medicine?
While we in the US argue incessantly in the details of minor reforms of our supposed free health care market, we ignore the rot at its foundations. True health care reform would attack the conflicts of interest that have put money, not patients at the center of health care.
References
1. Stone NJ, Robinson J, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2013. Link here.
2. Abramson JD, Rosenberg HG, Jewell N et al. Should statins be prescribed to people at low risk of cardiovascular disease. Brit Med J 2013; 347: 15-17. Link here.
3. Goff Jr DC, Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA Guideline on the Assessment of Cardiovascular
Risk. Journal of the American College of Cardiology (2013), doi: 10.1016/j.jacc.2013.11.005. Link here.
4. Rikder PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013; Link here.
5. Collier J. The price of independence. Br Med J 2006; 332: 1447-9. Link here
Monday, June 10, 2013
Is the Cystic Fibrosis Foundation a Charity or a Venture Capital Firm?
We have often discussed how health care organizations now seem prone to diversion from their stated missions, often when money is the object. While the organizations in question are frequently academic, teaching hospitals, academic medical centers, or medical schools in particular, in May, the Milwaukee Journal Sentinel presented an example of a disease specific charity. This article deserves considerably more attention than it apparently initially received.
Background
The background was,
Remember that while disease specific charities often sponsor basic and clinical research, in this case, the CFF sponsored drug development. In fact, much of the research on which this development was based was sponsored by charities and the US National Institutes of Health:
The Price to Patients
Despite, or perhaps because of the funding provided by CFF, Vertex chose a stratospheric price for its new drug.
To put that in perspective, the yearly cost of Kalydeco is approximately six times the median family income in the US.
Minimizing the Harms
To put it further into perspective, keep in mind that for the moment, the data from the single best published clinical trial on Kalydeco suggests that while the drug seems to help the average patient, it is not without risks, and it is certainly not a cure.
The largest published trial that followed patients for a reasonable amount of time appeared in 2011 in the New England Journal of Medicine. [Ramsey BW, Davies J, McElvaney G et al. A CFTR potentiator in patients with cystic fibrosis and the G551D mutation. N Engl J Med 2011; 365: 1663-1672. Link here.] The study followed 161 patients for 48 weeks. The patients treated with Kalydeco on average showed improved lung function (increase of FEV1 [forced expiratory volume in one second] of 10% compared to essentially no change (-0.2 percent) in the placebo group. Treated patients were less likely to have an exacerbation of their pulmonary disease requiring hospitalization (31% vs 49%). So the drug certainly seems to have benefits at least in the short term. The number needed to treat to prevent one exacerbation requiring hospitalization in one year is five, which seems quite respectable.
On the other hand, the drug may have significant harms, even thought the report of the study seems to have attempted to minimize them. The article stated
However, this statement depended on a rather peculiar definition of severe adverse events. In particular, pulmonary exacerbations of cystic fibrosis were included among severe adverse events. Yet these are, as the term suggests, manifestations of the disease that is being treated. Reduction of pulmonary adverse events should be and was considered a measure of efficacy. So placing exacerbations within the definition of adverse events essentially double counts these incidents.
Furthermore, the presence of these within the category of adverse events swamps out other events which may in fact be adverse results of the study drug. If one subtracts pulmonary exacerbations and hemoptysis from the counts of serious adverse events, what remains is that patients on Kalydeco were more likely to have a serious adverse event (10%) than those on placebo (4%). Thus the apparent number needed to harm was 17. Thus, using this peculiar definition of adverse events appears to be a way to manipulate the analysis to minimize the apparent harmfulness of the drug.
While the study did appear to show that more patient received the benefit of avoiding a hospitalization due to a pulmonary exacerbation of cystic fibrosis than had a serious adverse event, the study did not show that the drug had overwhelming efficacy, or tremendous safety. The study did not last long enough to show long term advantages, or to rule out rare but severe side effects.
This is the only drug available of this type, and it may well provide benefits that outweigh harms, at least over the short-term, but it is not a wonder drug, and the rationale to charge so much money for it, other than that is what the market will bear, is not obvious.
A Windfall for Corporate Executives, and a Question of Insider Trading
The drug's approval has lead to a lot of financial success for stock holders, particularly Vertex executives:
The executives' lavish windfall occurred in somewhat questionable circumstances:
So at least Senator Grassley raised the question of whether Vertex executives may have taken advantage of their insider knowledge to personally profit even more from this useful but not miraculous drug meant to be used on vulnerable patients.
Keep in mind that those huge trading gains were layered on top of already lavish compensation. The 2013 Vertex proxy statement, the total compensation and stock holdings of its top executives in 2012 was:
Jeffrey M Leiden, CEO $5,656,684 441,160 shares
Ian F Smith, CFO $3,109,193 795,434
Stuart A Arbuckle, Chief Commercial Officer $4,808,697 66,477
Kenneth L Horton, Chief Legal Officer $2,802,735 41,161
Peter Mueller, Chief Scientific Officer $3,614,890 997,651
Matthew W Emmens, Former CEO $6,896,029 1,486,748
David T Howton Jr, Fomer Chief Legal Officer $3,447,898 3,105
So the top executives of Vertex, while their company got $75 million from an ostensible charity to develop what became an extremely expensive drug, got very rich in the process, although how they got rich may yet attract attention from the SEC.
A Windfall for the Cystic Fibrosis Foundation and its Executives
Furthermore, it appears that the supposedly charitable Cystic Fibrosis Foundation also made quite a bit of money, and its executives, while not getting quite as rich as their associates in Vertex, did not do at all badly.
As the Journal Sentinel noted,
Furthermore, according to the Foundation's 2011 form 990 (the latest available), its executives received the following total compensation from the foundation and its affiliated organizations:
Robert J Beall, CEO $1,073,725
C Richard Mattingly, COO $759,799
Preston W Campbell MD, Exec VP of Medical Affairs $736,031
Vera H Twigg, CFO $445,183
Ann Palmer, Senior VP $276,029
Daniel Klein, Senior VP $277,300
Gregory August, CIO $262,698
David McLoughlin, Senior VP $316,122
Glen Goldmark, VP $253,215
Amy DeMaria, Senior VP $241,672
Mary Dwight, Senior VP $246,232
These compensation amounts may be much lower than the gargantuan pay dealt out to for-profit health care corporate executives, but they are very high for those who are managing a supposed charity meant to help vulnerable patients.
A More Complex Web
While profiting from its underwriting of the development of Kalydeco, the CFF also sponsored guidelines about the treatment of cystic fibrosis, with not unexpected results.
Criticism of Venture Philanthropy
Merriam-Webster online suggests one definition of a charity is an institution funded by a gift for public benevolent purposes.
The Cystic Fibrosis Foundation appears to be such a charity, but now one that functions more as a venture capitalist. In this case, it did provide venture capital to develop a new drug for its disease of interest. However, the foundation appeared to have done so not to provide public benevolence, but to generate a return on its investment. It is using that return not for public benevolence, but to provide more venture capital to other drug companies, presumably with the goal of getting further returns. Meanwhile, its executives make generous compensation for people who are supposed to be running a charity. Finally, the drug has an astronomical price, and its pricing has helped make investors in and executives of the company supported by the CFF very rich.
This has not been lost on some dissidents, per the Journal Sentinel,
Vertex responded with seeming contempt,
The funding by the supposedly charitable CFF of guidelines that promote the drug it financed has also drawn criticism,
Also,
However, the very well paid CEO of the CFF pooh poohed concerns about conflicts of interest,
One wonders when patients would ever have the opportunity to voice any "concerns" to Mr Beall, who also disdained any restraints on the price of the drug,
Summary
We have discussed numerous cases in which non-profit health care organizations seem to put short term revenue ahead of their missions to further patients' and the public's health. In this case, a disease specific charity seems to have foregone its mission to directly support patient care, teaching, or research to provide venture capital, an action which lead to a profit for the organization, huge profits for a drug company, large rewards for the charity's executives, and even greater wealth for the drug company's executives. It did also lead to the marketing of a beneficial drug, but at a breathtaking price that no middle-class patient without exceedingly good insurance could afford.
Where is the public benevolence here? Where is the charity? How much is about patients and how much is about making insider executives wealthy?
As we have said until blue in the face, true health care reform would ensure health care organizational leadership that upholds the health care mission, not their personal finances.
Background
The background was,
What happens when a disease-fighting charity dives into venture capitalism?
In the first case of its kind, the results include one of the planet's most expensive pills, huge sales projections for a drug company and windfalls for executives who sold stock in the glow of enthusiastic news releases about the drug.
Kalydeco is a breakthrough drug designed from knowledge of the genetic roots of cystic fibrosis, a lung disease that kills most victims before they reach middle age. Developed by Vertex Pharmaceuticals with a $75 million investment from the Cystic Fibrosis Foundation, it is an early example of 'venture philanthropy,' where a nonprofit helps finance development of a treatment in return for a cut of sales.
Remember that while disease specific charities often sponsor basic and clinical research, in this case, the CFF sponsored drug development. In fact, much of the research on which this development was based was sponsored by charities and the US National Institutes of Health:
In the 1980s, Francis Collins, now director of the National Institutes of Health, was a researcher at the University of Michigan and on his way to becoming a renowned gene hunter.
Collins and a team headed up by Lap-Chee Tsui at the Hospital for Sick Children in Toronto collaborated to identify the gene responsible for cystic fibrosis. That breakthrough involved funding from the NIH, the Cystic Fibrosis Foundation and the Howard Hughes Medical Institute, said Collins.
Another decade of intense basic science followed, much of it funded by NIH.
The Price to Patients
Despite, or perhaps because of the funding provided by CFF, Vertex chose a stratospheric price for its new drug.
Yet it costs each patient $307,000 a year to take two Kalydeco pills a day - a price borne by taxpayers through Medicaid and other government programs and by the workers and companies who finance employee health insurance plans.
In 2012, with less than a full-year on the market, Vertex sold $172 million worth of Kalydeco....
To put that in perspective, the yearly cost of Kalydeco is approximately six times the median family income in the US.
Minimizing the Harms
To put it further into perspective, keep in mind that for the moment, the data from the single best published clinical trial on Kalydeco suggests that while the drug seems to help the average patient, it is not without risks, and it is certainly not a cure.
The largest published trial that followed patients for a reasonable amount of time appeared in 2011 in the New England Journal of Medicine. [Ramsey BW, Davies J, McElvaney G et al. A CFTR potentiator in patients with cystic fibrosis and the G551D mutation. N Engl J Med 2011; 365: 1663-1672. Link here.] The study followed 161 patients for 48 weeks. The patients treated with Kalydeco on average showed improved lung function (increase of FEV1 [forced expiratory volume in one second] of 10% compared to essentially no change (-0.2 percent) in the placebo group. Treated patients were less likely to have an exacerbation of their pulmonary disease requiring hospitalization (31% vs 49%). So the drug certainly seems to have benefits at least in the short term. The number needed to treat to prevent one exacerbation requiring hospitalization in one year is five, which seems quite respectable.
On the other hand, the drug may have significant harms, even thought the report of the study seems to have attempted to minimize them. The article stated
there was a lower rate of serious adverse events in the ivacaftor [Kalydeco] group than in the placebo group (24% vs 42%).
However, this statement depended on a rather peculiar definition of severe adverse events. In particular, pulmonary exacerbations of cystic fibrosis were included among severe adverse events. Yet these are, as the term suggests, manifestations of the disease that is being treated. Reduction of pulmonary adverse events should be and was considered a measure of efficacy. So placing exacerbations within the definition of adverse events essentially double counts these incidents.
Furthermore, the presence of these within the category of adverse events swamps out other events which may in fact be adverse results of the study drug. If one subtracts pulmonary exacerbations and hemoptysis from the counts of serious adverse events, what remains is that patients on Kalydeco were more likely to have a serious adverse event (10%) than those on placebo (4%). Thus the apparent number needed to harm was 17. Thus, using this peculiar definition of adverse events appears to be a way to manipulate the analysis to minimize the apparent harmfulness of the drug.
While the study did appear to show that more patient received the benefit of avoiding a hospitalization due to a pulmonary exacerbation of cystic fibrosis than had a serious adverse event, the study did not show that the drug had overwhelming efficacy, or tremendous safety. The study did not last long enough to show long term advantages, or to rule out rare but severe side effects.
This is the only drug available of this type, and it may well provide benefits that outweigh harms, at least over the short-term, but it is not a wonder drug, and the rationale to charge so much money for it, other than that is what the market will bear, is not obvious.
A Windfall for Corporate Executives, and a Question of Insider Trading
The drug's approval has lead to a lot of financial success for stock holders, particularly Vertex executives:
Last month, news about success of the drug sent Vertex stock soaring more than $6 billion in a single day. That surge and a similar one last May allowed top executives and directors of the company to sell stock and options worth more than $100 million.
The executives' lavish windfall occurred in somewhat questionable circumstances:
Vertex and its executives have benefited greatly from Kalydeco and foundation funding.
Last May, when Vertex and the foundation reported positive results from a clinical trial involving Kalydeco and whether it could be combined with another drug to treat more patients, the company's stock jumped more than 70%, from $37.41 to $64.85 a share.
Five executives and two directors sold off more than $35 million in shares, mainly at prices from about $55 to $64 a share. Many of the options were priced between $16 and $40 a share.
Three weeks later, the company said it overstated the effectiveness of the drug in that trial and the stock dropped about $7 a share, ultimately falling back under $40 by December.
Vertex spokeswoman Nikki Levy said in an email the company does not comment on individual stock sales. She said the executive stock sales either were part of pre-existing 10b5-1 plans or followed the company's internal stock trading policy. A 10b5-1 plan is an automatic trading tool in which executives specify timing or pricing of sales to avoid questions about inside information the seller had at the time.
U.S. Sen. Chuck Grassley (R-Iowa) wrote a letter to the U.S. Securities and Exchange Commission, saying it could appear that Vertex executives took advantage of the situation, knowing the overstated clinical trial results would eventually be made public and cause the stock price to drop.
The letter said the stock sales were troubling for industry investors and the federal government, which pays billions of dollars a year for drugs through Medicaid and Medicare.
Judith Burns, a spokeswoman for the SEC, declined to comment on the Grassley letter.
Last month, the company's stock shot up more than 60% again, from $52.87 to $85.60, after positive early data from a clinical trial of Kalydeco and another drug it is developing with funding from the foundation. On April 19, the day after the news was released, the company's market value jumped by more than $6 billion.
That same day, two company executives sold huge chunks of stock options. Executive Vice President and Chief Financial Officer Ian Smith alone sold 745,685 shares worth more than $60 million. Most shares were sold at $81.50, with options purchased from $29 to $39.
So at least Senator Grassley raised the question of whether Vertex executives may have taken advantage of their insider knowledge to personally profit even more from this useful but not miraculous drug meant to be used on vulnerable patients.
Keep in mind that those huge trading gains were layered on top of already lavish compensation. The 2013 Vertex proxy statement, the total compensation and stock holdings of its top executives in 2012 was:
Jeffrey M Leiden, CEO $5,656,684 441,160 shares
Ian F Smith, CFO $3,109,193 795,434
Stuart A Arbuckle, Chief Commercial Officer $4,808,697 66,477
Kenneth L Horton, Chief Legal Officer $2,802,735 41,161
Peter Mueller, Chief Scientific Officer $3,614,890 997,651
Matthew W Emmens, Former CEO $6,896,029 1,486,748
David T Howton Jr, Fomer Chief Legal Officer $3,447,898 3,105
So the top executives of Vertex, while their company got $75 million from an ostensible charity to develop what became an extremely expensive drug, got very rich in the process, although how they got rich may yet attract attention from the SEC.
A Windfall for the Cystic Fibrosis Foundation and its Executives
Furthermore, it appears that the supposedly charitable Cystic Fibrosis Foundation also made quite a bit of money, and its executives, while not getting quite as rich as their associates in Vertex, did not do at all badly.
As the Journal Sentinel noted,
the foundation cashed in by selling future royalties from the drug to an undisclosed firm for $150 million.Keep in mind that since the CFF was to receive royalties, the money it gave for drug development was not a grant, but an investment.
Furthermore, according to the Foundation's 2011 form 990 (the latest available), its executives received the following total compensation from the foundation and its affiliated organizations:
Robert J Beall, CEO $1,073,725
C Richard Mattingly, COO $759,799
Preston W Campbell MD, Exec VP of Medical Affairs $736,031
Vera H Twigg, CFO $445,183
Ann Palmer, Senior VP $276,029
Daniel Klein, Senior VP $277,300
Gregory August, CIO $262,698
David McLoughlin, Senior VP $316,122
Glen Goldmark, VP $253,215
Amy DeMaria, Senior VP $241,672
Mary Dwight, Senior VP $246,232
These compensation amounts may be much lower than the gargantuan pay dealt out to for-profit health care corporate executives, but they are very high for those who are managing a supposed charity meant to help vulnerable patients.
A More Complex Web
While profiting from its underwriting of the development of Kalydeco, the CFF also sponsored guidelines about the treatment of cystic fibrosis, with not unexpected results.
Last month, new treatment guidelines for doctors who handle cystic fibrosis patients strongly recommended use of Kalydeco. The guidelines were funded by the Cystic Fibrosis Foundation.These guidelines hardly look like they would be deemed trustworthy according to the Institute of Medicine's standards (look here). However, they certainly look like they might help sell ivacaftor, and hence help justify higher pay for the executives listed above.
Three of the 10 authors of the guidelines were employees of the foundation and four others worked for institutions that received grants from the foundation. The chairman, Peter Mogayzel, is a professor of pediatrics at Johns Hopkins University, which foundation tax records show received more than $2 million in grants from 2009 through 2011.
Criticism of Venture Philanthropy
Merriam-Webster online suggests one definition of a charity is an institution funded by a gift for public benevolent purposes.
The Cystic Fibrosis Foundation appears to be such a charity, but now one that functions more as a venture capitalist. In this case, it did provide venture capital to develop a new drug for its disease of interest. However, the foundation appeared to have done so not to provide public benevolence, but to generate a return on its investment. It is using that return not for public benevolence, but to provide more venture capital to other drug companies, presumably with the goal of getting further returns. Meanwhile, its executives make generous compensation for people who are supposed to be running a charity. Finally, the drug has an astronomical price, and its pricing has helped make investors in and executives of the company supported by the CFF very rich.
This has not been lost on some dissidents, per the Journal Sentinel,
'The concept of a charitable, not-for-profit taking on the role of a venture capitalist is new and difficult to digest at the moment,' said Paul Quinton, a cystic fibrosis researcher at the University of California, Riverside and the University California, San Diego.
Quinton, who has cystic fibrosis, is one of 28 doctors and scientists who sent a letter to Vertex calling the price of Kalydeco 'unconscionable.' A copy of the letter was provided to the Journal Sentinel and MedPage Today. Kalydeco, the doctors wrote, costs 10 times more than what a typical cystic fibrosis patient pays in total drug costs.
'This action could appear to be leveraging pain and suffering into huge financial gain for speculators, some of whom were your top executives who reportedly made millions of dollars in a single day,' the doctors wrote.
Vertex responded with seeming contempt,
Since receiving the letter last July, Vertex has raised the annual price of Kalydeco another $13,000.
The funding by the supposedly charitable CFF of guidelines that promote the drug it financed has also drawn criticism,
'It is definitely a conflict of interest,' said Eric Campbell, an associate professor at Harvard Medical School who has researched conflicts of interest in patient treatment guidelines.
In the past, drug companies have been criticized for funding treatment guidelines that recommend their drugs. It is no different if the guidelines are funded by a foundation that gets royalties from drug sales, Campbell said.
Also,
'It is concerning that the organization now stands to profit when patients choose to use the drug,' ... [Prof Lisa Schwartz of Dartmouth Medical School] said. 'Financial entanglement with industry, even with the best of intentions, creates a conflict of interest.'
However, the very well paid CEO of the CFF pooh poohed concerns about conflicts of interest,
Robert Beall, president of the Cystic Fibrosis Foundation, said that without its financial support, drugs such as Kalydeco would never get to patients. Neither insurance companies nor patients have voiced any concern to him about conflicts of interest, he said.
'They applaud the decision and our business model to the utmost,' Beall said. 'The patients are excited.'
He rejected the idea of using the royalty money to help patients pay for the medical care, noting that the foundation needed the money to entice drug companies to get involved in risky cystic fibrosis drug research.
One wonders when patients would ever have the opportunity to voice any "concerns" to Mr Beall, who also disdained any restraints on the price of the drug,
.Beall said the foundation did not ask Vertex to price the drug more affordably.That seems to put making money ahead of patients' needs. Was this venture philanthropy, or vulture philanthropy?
'That would have been a deal-breaker,' he said.
Summary
We have discussed numerous cases in which non-profit health care organizations seem to put short term revenue ahead of their missions to further patients' and the public's health. In this case, a disease specific charity seems to have foregone its mission to directly support patient care, teaching, or research to provide venture capital, an action which lead to a profit for the organization, huge profits for a drug company, large rewards for the charity's executives, and even greater wealth for the drug company's executives. It did also lead to the marketing of a beneficial drug, but at a breathtaking price that no middle-class patient without exceedingly good insurance could afford.
Where is the public benevolence here? Where is the charity? How much is about patients and how much is about making insider executives wealthy?
As we have said until blue in the face, true health care reform would ensure health care organizational leadership that upholds the health care mission, not their personal finances.
Subscribe to:
Posts (Atom)