A Trump administration decision to terminate the contract of one of the federal government’s top evidence clearinghouses, the National Registry of Evidence-based Programs and Practices (NREPP), has raised questions about its ongoing commitment to evidence. But the decision may have actually reflected the opposite — a stronger commitment to evidence amid concerns about the clearinghouse’s credibility and that of the medical research industry as a whole.
The clearinghouse, operated by the Substance Abuse and Mental Health Services Administration (SAMHSA), reviews studies of evidence-based mental health and substance use programs, including those related to opioids, an administration priority.
The contractor overseeing the clearinghouse received word of its termination from SAMHSA over the Christmas holidays. However, the decision may have been made much earlier in the year, according to a story yesterday in the Washington Post:
Agency officials froze the existing website in September, and no new postings have been added, according to mental health advocacy groups. As a result, about 90 new programs that have been reviewed and rated for their scientific merits since September are not available to the public, they said.
SAMHSA has been largely silent on the matter, saying only that the duties of the clearinghouse will now be shifted to a new Policy Lab that was created by federal law and that was launched earlier this month. [Edit: SAMHSA has issued a statement. See below.]
In describing the Lab’s duties, Elinore McCance-Katz, the agency’s new administrator, told Congress last year that:
[T]he primary focus of the Policy Lab is to “periodically review programs and activities” relating to the diagnosis, prevention, treatment and recovery from mental illness and substance use disorders. The lab will also identify programs or activities “that are duplicative and are not evidence-based, effective or efficient,” she said.
The Policy Lab will also play a role in awarding grants to state and local governments, educational institutions and nonprofits to develop evidence-based interventions, she said.
The new Policy Lab is headed by Christopher Jones, who is well-regarded in the opioids world. However, the Lab and NREPP are governed by different sections of federal law. While SAMHSA could theoretically assign oversight of the clearinghouse to the Lab, it must still maintain NREPP or something like it. The only real questions are whether it is run in-house or by a new contractor, and when it becomes fully operational again.
Questions Raised About Objectivity and Rigor
NREPP’s cancellation may have been about more than a reshuffling of SAMHSA’s internal duties, however. The clearinghouse had been the subject of serious criticism in a journal article published over the summer, which condemned it for insufficient rigor in its reviews.
Because evidence clearinghouses exist to give practitioners an objective third-party opinion of existing research, any doubts about their credibility can be devastating. According to the Washington Post:
In July, Dennis Gorman, a professor in Texas A&M University’s Department of Epidemiology & Biostatistics, published a paper in the International Journal of Drug Policy saying that the database was not “weeding out” ineffective programs.
Between September 2015 and January 2017, the paper notes, 113 interventions were added to the registry. Of these, the evidence for 14 percent of them consisted of a single, non-peer-reviewed report, and 45 percent of them relied primarily on a single peer-reviewed article.
The journal article, entitled Has the National Registry of Evidence-based Programs and Practices (NREPP) Lost Its Way?, also criticized the clearinghouse for failing to adequately deal with what it said were potential conflicts of interest. It concluded with a withering criticism:
NREPP can be added to the list of evidence-based initiatives that no longer serve the purpose for which they were created and contribute to a waste of valuable societal resources and a degradation of science (Ioannidis, 2016).
The linked article, entitled “Evidence-Based Medicine Has Been Hijacked” by John Ioannidis, is a broader critique of research in the medical community. Its author, a respected doctor and researcher at Stanford University (formerly at Johns Hopkins and the National Institutes of Health), has been an ardent critic of poor research, which he has variously attributed to poorly-designed studies and researcher bias. He is now the co-director of a center at Stanford that is devoted to promoting reproducibility in research.
In his 2016 article, Ioannidis leveled similar charges of bias against the medical research community writ large. According to Ioannidis:
Influential randomized trials are largely done by and for the benefit of the industry. Meta-analyses and guidelines have become a factory, mostly also serving vested interests. National and federal research funds are funneled almost exclusively to research with little relevance to health outcomes.
Such criticisms, if true, suggest that NREPP’s perceived problems may not have been solely attributable to the clearinghouse, but part of a much larger, systemic problem for medical research as a whole.
SAMHSA has been largely silent on the matter, so it is unclear what the reasons were for its decision to terminate NREPP. Whatever the justifications, the agency’s failure to articulate them has sparked confusion and concern in the provider community and on Capitol Hill, according to the Washington Post.
The decision also comes one month after the Trump administration was criticized for a ban on certain words in budget documents by the Centers for Disease Control, including the term “evidence-based.” Reports later emerged that suggested the CDC decision was driven by career staffers who were trying to avoid antagonizing congressional budget writers.
In this case, the administration’s decision may have reflected a stronger commitment to evidence, not a backing away from it as some had feared. The agency may nevertheless find itself having to explain that commitment in the coming months as its new Policy Lab comes online.
Update (1/11/18)
SAMHSA has today issued a press statement on NREPP from Assistant Secretary Elinore F. McCance-Katz that appears to confirm some of the outside criticisms of the clearinghouse. It reads (in part) as follows:
For the majority of its existence, NREPP vetted practices and programs submitted by outside developers – resulting in a skewed presentation of evidence-based interventions, which did not address the spectrum of needs of those living with serious mental illness and substance use disorders. These needs include screening, evaluation, diagnosis, treatment, psychotherapies, psychosocial supports and recovery services in the community.
The program as currently configured often produces few to no results, when such common search terms as “medication-assisted treatment” or illnesses such as ”schizophrenia” are entered. There is a complete lack of a linkage between all of the EBPs that are necessary to provide effective care and treatment to those living with mental and substance use disorders, as well. If someone with limited knowledge about various mental and substance use disorders were to go to the NREPP website, they could come away thinking that there are virtually no EBPs for opioid use disorder and other major mental disorders – which is completely untrue.
They would have to try to discern which of the listed practices might be useful, but could not rely on the grading for the listed interventions; neither would there be any way for them to know which interventions were more effective than others.
We at SAMHSA should not be encouraging providers to use NREPP to obtain EBPs, given the flawed nature of this system. From my limited review – I have not looked at every listed program or practice – I see EBPs that are entirely irrelevant to some disorders, “evidence” based on review of as few as a single publication that might be quite old and, too often, evidence review from someone’s dissertation.
This is a poor approach to the determination of EBPs. As I mentioned, NREPP has mainly reviewed submissions from “developers” in the field. By definition, these are not EBPs because they are limited to the work of a single person or group. This is a biased, self-selected series of interventions further hampered by a poor search-term system. Americans living with these serious illnesses deserve better, and SAMHSA can now provide that necessary guidance to communities.
We are now moving to EBP implementation efforts through targeted technical assistance and training that makes use of local and national experts and will that assist programs with actually implementing services that will be essential to getting Americans living with these disorders the care and treatment and recovery services that they need.
These services are designed to provide EBPs appropriate to the communities seeking assistance, and the services will cover the spectrum of individual and community needs including prevention interventions, treatment and community recovery services.
We must do this now. We must not waste time continuing a program that has had since 1997 to show its effectiveness.
But yet we know that the majority of behavioral health programs still do not use EBPs: one indicator being the lack of medication-assisted treatment, the accepted, life-saving standard of care for opioid use disorder, in specialty substance use disorder programs nationwide.
SAMHSA will use its technical assistance and training resources, its expert resources, the resources of our sister agencies at the Department of Health and Human Services, and national stakeholders who are consulted for EBPs to inform American communities and to get Americans living with these disorders the resources that they deserve.
Update (1/12/18)
During a January 11 press call, SAMHSA administrator McCance-Katz said she questioned NREPP when she was the agency’s chief medical officer from 2013 to 2015. She knew it would be one of the first things she would address as its new Assistant Secretary.
In a 2016 article, she wrote this:
I left SAMHSA after 2 years. It became increasingly uncomfortable to be associated with an agency that, for the most part, refused to support evidence-based psychiatric treatment of mental disorders. It was also quite clear that the psychiatric perspective I brought—inclusive of assessment, diagnosis of mental disorders, utilization of evidence-based treatments, including psychotropic medication and psychosocial interventions as integral components of recovery—was a poor fit for the agency. SAMHSA needs a complete review and overhaul of its current mission, leadership, and funded programs. Congress should quickly address this through legislative mandate.
McCance-Katz was a somewhat controversial choice to head the agency. Some have suggested that she prefers mental health and substance abuse treatments that emphasize medical solutions, not non-medical social supports.
This has concerned some members of Congress and providers who prefer the inclusion of such wrap-around social services. On the other hand, in her 2016 statement above she indicated that she supported “psychosocial interventions as integral components of recovery.”