Child welfare services that are evidence-based may see a significant jump in financial support under legislation that is being drafted by the Senate Finance Committee.
A bipartisan bill that is now being assembled by Sens. Orrin Hatch (R-UT) and Ron Wyden (D-OR) would expand federal support for evidence-based prevention and family service programs, including mental health, substance abuse prevention, and in-home parent skill-based services.
Under the proposal, to be eligible for federal support the new services would need to qualify under an increasingly stringent set of evidence guidelines. Starting on October 1, 2017, the services would need to be rated as promising practices to receive federal support. By 2020 they would need to be rated as evidence-supported and by 2023 they would need to be rated as well-supported. The legislation directs the Department of Health and Human Services to assemble a list of evidence-based services that qualify.
The proposal’s three tiers of evidence (described below) are loosely based on those of the California Evidence-based Clearinghouse for Child Welfare. They differ primarily because they do not require a randomized controlled trial to meet the highest definitions. As drafted, the proposed definitions appear less strict than those included in a major education bill that Congress enacted last year.
This may be due in part to the relatively nascent state of evidence in child welfare generally. As of February, 2014, only 27 of 325 child welfare programs (just 8 percent) catalogued by the California Clearinghouse met its criteria for being well-supported by research.
Moreover, many evidence-based practices in the field are borrowed or adapted from those originally used in other settings or with other populations. Ratings for mental health and substance abuse programs more broadly, for example, can be found at SAMHSA’s National Registry of Evidence-based Programs and Practices.
Nevertheless, child welfare experts appear to be welcoming the bill’s evidence provisions, arguing that creating greater demand will probably create increased supply. A 2015 report by Results for America and Bridgespan, for instance, found that some private purveyors were withholding their programs from review until they could be sure that they would earn top ratings. By creating financial incentives, the legislation may quicken the pace of evidence development.
Some experts caution, however, that more federal investment is needed to build out the evidence base. Current federal initiatives — including the Permanency Innovations Initiative (PII) and others studying family group decision-making and youth at risk of homelessness — tend to be small and often narrowly focused.
Moreover, state-initiated child welfare waivers, a major source of information about the effectiveness of new practices, have relied less on experimental research in recent years. Authority for approving new waivers has also expired.
“Restoring waiver authority as mandated in the original 1995 legislation, which incentivized randomized controlled trials, is a proven way to jump-start the development of evidence-based interventions in child welfare,” said Mark Testa, principal investigator for PII’s national evaluation and professor at the University of North Carolina at Chapel Hill.
Given these limitations, some experts warn that the field may not advance quickly enough to meet the legislation’s aggressive timetable. Moreover, where it occurs, the development of new evidence will likely be uneven.
“This timeline is fast,” said Becci Akin, Assistant Professor at the University of Kansas and a principal investigator for one of the PII projects. She noted that most of the PII projects have taken five years to complete.
“There are ways of building evidence through low-cost RCTs, which are faster, cheaper, and smarter to shorten the timeline from design to broad-scale roll-out,” said Testa. “The trick is that many trials of promising innovations have to be conducted simultaneously in many states to yield an adequate supply of evidence-based interventions.”
“If we do this, we will need to maintain the integrity of the list,” said Jennifer Noyes, Associate Director at the Institute for Research on Poverty at the University of Wisconsin at Madison. “Just because something was tested and found to be effective in one setting, even if through a randomized trial, does not mean it will be effective in another setting, so adjustments to the list may need to be made along the way.”
Some experts recommend that HHS be given sufficient latitude to delay full implementation of the standards for certain kinds of services if the field does not advance as quickly and uniformly as hoped.
Three Proposed Evidence Tiers
According to the Senate summary, the legislation would establish the following three tiers of evidence for prevention and family service programs:
Promising Practice
- There is no case data suggesting a risk of harm that: a) was probably caused by the treatment and b) the harm was severe or frequent.
- There is no legal or empirical basis suggesting that, compared to its likely benefits, the practice constitutes a risk of harm to those receiving it.
- The practice has a book, manual, and/or other available writings that specify the components of the practice protocol and describe how to administer it.
- At least one study utilizing some form of control (e.g., untreated group, placebo group, matched wait list study) has established the practice’s benefit over the control, or found it to be comparable to a practice rated a 1, 2, or 3 on this rating scale or superior to an appropriate comparison practice. The study has been reported in published, peer-reviewed literature.
- Outcome measures must be reliable and valid, and administered consistently and accurately across all subjects.
- If multiple outcome studies have been conducted, the overall weight of evidence supports the benefit of the practice.
Supported Practice – Must adhere to 1-3 and 5, 6 of Promising Practice criteria. In addition the practice must satisfy the following criteria:
- At least one rigorous randomized controlled trial (RCT) OR a control or comparison group with pre and post outcome assessments in usual care or a practice setting has found the practice to be superior to an appropriate comparison practice.
- In that same RCT or comparison study, the practice has shown to have a sustained effect of at least six months beyond the end of treatment, when compared to a control group. The trial or outcomes of the control or comparison group must be published in peer-reviewed literature.
Well Supported Practice – Must adhere to 1-3 and 5, 6 of Promising Practice criteria. In addition the practice must satisfy the following criteria:
- At least two rigorous randomized controlled trials (RCTs) OR control or comparison groups with pre and post outcome assessments in different usual care or practice settings —
- have found the practice to be superior to an appropriate comparison practice;
- have been reported in published, peer-reviewed literature; and
- in at least one of these RCTs or control or comparison group the practice has shown to have a sustained effect of at least one year beyond the end of treatment, when compared to a control group.
The proposed legislation directs HHS to issue guidance to states including a list of services and programs that meet the evidence-based policy standards.
Related
- Report Finds Evidence Clearinghouse Gaps and Challenges (April 11, 2015)
- Foster Care Innovation Initiative Charts a Different Path to Evidence (December 6, 2014)