Studies about medications published in the most influential medical journals are frequently designed in a way that yields misleading or confusing results, new research suggests.
Investigators from the medical schools at UCLA and Harvard analyzed all the randomized medication trials published in the six highest-impact general medicine journals between June 1, 2008, and Sept. 30, 2010, to determine the prevalence of three types of outcome measures that make data interpretation difficult.
In addition, they reviewed each study's abstract to determine the percentage that reported results using relative rather than absolute numbers, which can also be a misleading.
The findings are published online in the Journal of General Internal Medicine.
The six journals examined by the investigators— the New England Journal of Medicine, the Journal of the American Medical Association, The Lancet, the Annals of Internal Medicine, the British Medical Journal and the Archives of Internal Medicine — included studies that used the following types of outcome measures, which have received increasing criticism from scientific experts:
Surrogate outcomes (37 percent of studies), which refer to intermediate markers, such as a heart medication's ability to lower blood pressure, but which may not be a good indicator of the medication's impact on more important clinical outcomes, like heart attacks.
Composite outcomes (34 percent), which consist of multiple individual outcomes of unequal importance lumped together — such as hospitalizations and mortality — making it difficult to understand the effects on each outcome individually.
Disease-specific mortality (27 percent), which measures deaths from a specific cause rather than from any cause; this may be a misleading measure because, even if a given treatment reduces one type of death, it could increase the risk of dying from another cause, to an equal or greater extent.
"Patients and doctors care less about whether a medication lowers blood pressure than they do about whether it prevents heart attacks and strokes or decreases the risk of premature death," said the study's lead author, Dr. Michael Hochman, a fellow in the Robert Wood Johnson Foundation Clinical Scholars Program at the David Geffen School of Medicine at UCLA's division of general internal medicine and health services research, and at the U.S. Department of Veterans Affairs' Los Angeles Medical Center.
"Knowing the effects of a medication on blood pressure does not always tell you what the effect will be on the things that are really important, like heart attacks or strokes," Hochman said. "Similarly, patients don't care if a medication prevents deaths from heart disease if it leads to an equivalent increase in deaths from cancer."
Dr. Danny McCormick, the study's senior author and a physician at the Cambridge Health Alliance and Harvard Medical School, added: "Patients also want to know, in as much detail as possible, what the effects of a treatment are, and this can be difficult when multiple outcomes of unequal importance are lumped together."
The authors also found that trials that used surrogate outcomes and disease-specific mortality were more likely to be exclusively commercially funded — for instance, by a pharmaceutical company.
While 45 percent of exclusively commercially funded trials used surrogate endpoints, only 29 percent of trials receiving non-commercial funding did. And while 39 percent of exclusively commercially funded trials used disease-specific mortality, only 16 percent of trials receiving non-commercial funding did.
The researchers suggest that commercial sponsors of research may promote the use of outcomes that are most likely to indicate favorable results for their products, Hochman said.
"For example, it may be easier to show that a commercial product has a beneficial effect on a surrogate marker like blood pressure than on a hard outcome like heart attacks," he said. "In fact, studies in our analysis using surrogate outcomes were more likely to report positive results than those using hard outcomes like heart attacks."
The new study also shows that 44 percent of study abstracts reported study results exclusively in relative — rather than absolute — numbers, which can be misleading.
"The way in which study results are presented is critical," McCormick said. "It's one thing to say a medication lowers your risk of heart attacks from two-in-a-million to one-in-a-million, and something completely different to say a medication lowers your risk of heart attacks by 50 percent. Both ways of presenting the data are technically correct, but the second way, using relative numbers, could be misleading."
Still, the authors acknowledge that the use of surrogate and composite outcomes and disease-specific mortality is appropriate in some cases. For example, these outcomes may be preferable in early-phase studies in which researchers hope to quickly determine whether a new treatment has the potential to help patients.
To remedy the problems identified by their analysis, Hochman and McCormick believe that studies should report results in absolute numbers, either instead of or in addition to relative numbers, and that committees overseeing research studies should closely scrutinize study outcomes to ensure that lower-quality outcomes, like surrogate makers, are only used in appropriate circumstances.
"Finally, medical journals should ensure that authors clearly indicate the limitations of lower-quality endpoints when they are used — something that does not always occur," McCormick said.
The authors did not receive any internal or external funding for this research.
The Robert Wood Johnson Foundation focuses on the pressing health and health care issues facing our country. As the nation's largest philanthropy devoted exclusively to improving the health and health care of all Americans, the Foundation works with a diverse group of organizations and individuals to identify solutions and achieve comprehensive, meaningful and timely change. For more than 35 years, the Foundation has brought experience, commitment, and a rigorous, balanced approach to the problems that affect the health and health care of those it serves. When it comes to helping Americans lead healthier lives and get the care they need, the Foundation expects to make a difference in your lifetime.
General Internal Medicine and Health Services Research is a division within the Department of Medicine at the David Geffen School of Medicine at UCLA. It provides a unique interactive environment for collaborative efforts between health services researchers and clinical experts with experience in evidence-based work. The division's 100-plus clinicians and researchers are engaged in a wide variety of projects that examine issues related to access to care, quality of care, health measurement, physician education, clinical ethics and doctor/patient communication. The division's researchers have close working relationships with economists, statisticians, social scientists and other specialists throughout UCLA and frequently collaborate with their counterparts at the RAND Corp and Charles Drew University.
Cambridge Health Alliance is an innovative, award-winning health system that provides high quality care in Cambridge, Somerville, and Boston's metro-north communities. It includes three hospital campuses, a network of primary care and specialty practices, the Cambridge Public Health Dept., and the Network Health plan. CHA is a Harvard Medical School teaching affiliate and is also affiliated with Harvard School of Public Health, Harvard School of Dental Medicine, and Tufts University School of Medicine.
