Evidence-Based Policy Proponents Face Cost, Privacy, Political Hurdles

Posted on October 26, 2016 by Patrick Lester

After three initial meetings, early hints are beginning to emerge on what may eventually become a package of recommendations from the congressionally authorized Commission on Evidence-Based Policymaking.

Enacted in March, the bipartisan commission has been tasked with developing recommendations that would bolster evidence-based policy, principally by making federal data more widely available to program evaluators. Such evaluations have begun to influence funding decisions, both at the federal and state levels, although their direct influence is still small.

Statutorily, the commission’s mission appears to be somewhat narrow, but at its first meeting in July senior Democratic and Republican congressional staff urged it to interpret its mandate more broadly.

Continue reading SIRC’s column at Government Executive magazine.

Posted in Evidence

Teen Pregnancy Prevention Program Findings Show Benefits, Challenges of Evidence-Based Programming

Posted on September 22, 2016 by Suzanne Perry

When the federal government started a new grants program in 2010 to finance and evaluate programs to reduce teen pregnancies, Hennepin County decided to give it a shot.

The Minneapolis-area county already had a pilot project in two suburbs with high teen pregnancy rates that it had developed after intensive consultations with young people, parents, and community organizations. But an influx of federal money would allow it to expand its efforts to encourage young people to delay sex or use contraception.

The county won a five-year grant from the Teen Pregnancy Prevention Program, operated by the Office of Adolescent Health in the U.S. Department of Health and Human Services (HHS). The money paid for two efforts. One, the Teen Outreach Program, was on a list that HHS had compiled of “evidence-based programs” — that is, approaches that had already proven effective through rigorous evaluation.

“It was all the rage,” says Kathy Wick, initiative manager at Better Together Hennepin, the county’s teen-pregnancy program. ”It was really a youth-development program that supposedly had positive results in terms of impacting teen behaviors around pregnancy prevention and whether to have sex or not.”

The Teen Outreach Program (TOP), which had shown good results in a 1997 study,  involves weekly classroom sessions, community-service learning, and adult support. The county spent about $500,000 a year out of its annual $3.3 million grant to conduct a randomized controlled trial involving 1,644 students in 24 middle and high schools.

The results? The program had no impact.

Three months and 15 months after the intervention, participants were just as likely as students who followed the regular school curriculum to have had recent sexual activity as well as unprotected sex. They also scored no better in areas like school performance, school engagement, educational expectations, and civic responsibility.

“We spent five years very intensively building TOP with a whole lot of school partners,” says Katherine Meerse, former manager of Better Together Hennepin. “Results showed it didn’t have an impact, what do we do about that?”

There was only one answer, she says: “It wasn’t effective for our kids, we needed to move on to something that was.” Continue reading →

Posted in Children and Families, Evidence, Government Performance

CBO Says CMMI Will Save $34 Billion Over 10 Years

Posted on September 7, 2016 by Patrick Lester

In testimony before the House Budget Committee, the Congressional Budget Office (CBO) estimated that the Center for Medicare and Medicaid Innovation (CMMI) will save the federal government $34 billion over the next ten years (2017-2026).

CMMI was established by the Affordable Care Act to test “innovative payment and service delivery models to reduce program expenditures…while preserving or enhancing the quality of care” for individuals enrolled in Medicare, Medicaid, or the Children’s Health Insurance Program (CHIP). Congress authorized $10 billion for such test from 2011-2019.

CBO based its estimate on savings achieved so far by CMMI, which rest primarily on one demonstration project, called the Pioneer Accountable Care Organization model, that offered doctors, hospitals, and other providers shared financial incentives to improve quality and reduce spending. CBO is basing its estimate on a projection that CMMI will achieve similar successes in the years ahead and that HHS will roll successful demonstration projects out more widely.

During the hearing, Mark Hadley, CBO’s deputy director, said that his organization expects “many or most” of the CMMI demonstrations to fail, but that it will still produce the estimated savings.

The projection drew skepticism from committee Republicans and from at least one other witness, according to the Healthcare Financial Management Association.

Joe Antos, a scholar from the right-leaning American Enterprise Institute and a former CBO and Medicare agency official, said at the hearing that the future savings projection was a “shot in the dark.”

“They are extrapolating from information that is not current with CMMI’s activities and they have no information—as none of us have the information—about CMMI’s future activities,” Antos said.

Related

Posted in Health

More Data than Evidence in Evidence Commission’s First Meeting

Posted on July 24, 2016 by Patrick Lester

Will the new Evidence-based Policymaking Commission, created by Congress and signed into law by President Obama on March 30, actually further the cause of evidence-based policymaking?

With just one meeting under its belt, it is too early to know for sure, but the inaugural meeting on July 22 provided reason to worry.

Presentations by executive branch officials, including the Census Bureau (which is staffing the commission), focused almost entirely on improving the quality and usefulness of federal statistics. This focus was also reflected in the meeting’s attendees, most of whom represented various federal statistical agencies, with few (if any) representatives of the federal evaluation community.

Even the one presentation devoted to evaluation — by Raj Chetty of Stanford University — seemed to downplay the usefulness of randomized controlled trials, a gold standard evaluation methodology that was cited as something to support in the authorizing legislation. More broadly, the needs of local practitioners and evaluators, who are commonly on the front lines of evidence-based work, were almost entirely absent from a meeting that lasted over three hours.


Data Data Everywhere

It was a curious start for a commission that drew strong bipartisan support in Congress and from the Obama administration as a way to further the evidence-based policymaking agenda.  The commission is directed by law to issue a report within 15 months of its appointment which, according to an estimate by the chair, Katharine Abraham, will be September 6, 2017.

Continue reading →

Posted in Evidence

Evidence Guidelines Are Slated for U.S. Foreign Assistance Programs Under New Law

Posted on July 15, 2016 by Patrick Lester

On July 15, President Obama signed legislation into law that would establish federal guidelines for monitoring and evaluating U.S. foreign assistance programs.

Under the new law, called the Foreign Aid Transparency and Accountability Act,  the next administration would be directed to issue guidelines by January 15, 2018 that would establish goals, performance metrics, and monitoring and evaluation plans for foreign assistance programs.

Among other provisions, the required guidelines must include:

  • Rigorous Monitoring and Evaluation Requirements: The guidelines must apply rigorous monitoring and evaluation methodologies to these programs, including through the use of impact evaluations, ex-post evaluations, or other methods, as appropriate, that clearly define program logic, inputs, outputs, intermediate outcomes, and end outcomes.
  • Impact or Performance Evaluations for Pilot Programs: The guidelines will require pilot programs to have either: (1) an impact evaluation, or (2) a performance evaluation with a justification for why an impact evaluation was deemed inappropriate or impracticable.
  • Use of Professional Standards: The guidelines must ensure that standards of professional evaluation organizations are employed for monitoring and evaluation efforts, including ensuring the integrity and independence of evaluations, permitting and encouraging the exercise of professional judgment, and providing for quality control and assurance in the monitoring and evaluation process.
  • Public Reporting of Evaluations: Evaluation results must be publicly reported within 90 days, including an executive summary, a description of the evaluation methodology, key findings, appropriate context, including quantitative and qualitative data when available, and recommendations.
  • Clearinghouse: The guidelines will also include provisions for developing a clearinghouse for the collection, dissemination, and preservation of knowledge and lessons learned to guide future programs for United States foreign assistance personnel, implementing partners, the donor community, and aid recipient governments.

The guidelines will apply to the following foreign aid programs:

  • USAID programs operated under part I of the Foreign Assistance Act
  • Overseas Private Investment Corporation
  • Economic Support Fund
  • Millennium Challenge Corporation
  • Office of Food for Peace in USAID
Posted in Evidence, Foreign Assistance

The Moneyball Debate

Posted on July 14, 2016 by Patrick Lester

There is an emerging debate over the “Moneyball for Government” concept. Two perspectives in the Stanford Social Innovation Review.

Srik Gopal & Lisbeth B. Schorr, Getting “Moneyball” Right in the Social Sector (June 2, 2016)

Patrick Lester, Defining Evidence Down (July 14, 2016)

Lisbeth Schorr & Srik Gopal, Broadening the Evidence Base Without “Defining Evidence Down” (August 3, 2016)


Related

Posted in Evidence

Promise Zones Program Ramps Up Amid Uncertainty About Post-Obama Era

Posted on July 7, 2016 by Suzanne Perry

When the Obama administration announced the third and final round of Promise Zone communities in June, perhaps no applicant was more surprised to find itself on the list than South Los Angeles.

“We were kind of in disbelief when we got the news,” says Heddy Nam, director of the South Los Angeles Transit Empowerment Zone (SLATE-Z), a coalition of more than 50 nonprofit groups, educational institutions, public officials, and others that applied for the Promise Zone designation.

The award gives high-poverty communities easier access to some federal grants along with technical assistance – and SLATE-Z had worked hard to earn it. With Los Angeles Trade Technical College as the lead organization and strong support from city officials and Congresswomen Karen Bass and Lucille Roybal-Allard, it developed a plan to boost economic development and educational achievement along major transit lines in an area of almost 200,000 residents.

But a Central Los Angeles area had won a Promise Zone designation in the first round of awards, and what were the chances that one city would get two “zones”?

Furthermore, SLATE-Z had lost its bid to operate a Promise Zone in the second round of applications in 2015. So it was prepared for another rejection. But, Nam says, the coalition decided that it was worthwhile to draw up a strategy to boost South L.A. even if it was passed over again.

The attitude, she says: “We’re probably not going to get this thing, but it’s a historic opportunity.”

Communities across the country have made similar calculations since President Obama announced the Promise Zone Initiative in his State of the Union address in 2013, flooding the administration with more than 230 applications for just 22 designated zones (the number includes some applicants who tried more than once). What is the appeal?


Promise Zones Benefits

In announcing Promise Zones, Obama said the federal government would partner with high-poverty communities to boost economic development, improve education, fight crime, enhance public health, attract private investment, and achieve other goals.

Continue reading →

Posted in Collective Impact, Government Performance

House Moves Bill With Career and Technical Education Innovation Program

Posted on July 7, 2016 by Patrick Lester

A House education committee today unanimously approved legislation (HR 5587) that would reauthorize the Department of Education’s career and technical education programs. The bipartisan legislation would also create a new innovation program that would support evidence-based career and technical education initiatives, including through the use of pay-for-success strategies.

The overall bill (detailed summary) reauthorizes the Carl D. Perkins Career and Technical Education Act, which was last updated in 2006. The proposed innovation program (bill language for the proposal) is part of a larger set of national program activities that are authorized at $7.5 million per year starting in fiscal year 2017, which begins October 1 of this year.

The overall bill has drawn broad bipartisan support in Congress and support from over 200 groups that focus on career and technical education. Results for America was among the supporters. It worked with members of the Invest in What Works coalition to support the new innovation program, according to a statement.

One organization — the Center for Law and Social Policy (CLASP) — said it was concerned about the overall bill’s proposed rollback of state accountability measures. However, it expressed support for the proposed innovation program.

The bill must still be approved in the full House and in the Senate, but experts believe it has a realistic chance of being enacted this year.

Related

Posted in Education, Workforce Issues

No More “Deals”

Posted on July 4, 2016 by Patrick Lester

One of the strengths of pay for success is that it brings together several distinct worlds in pursuit of a common goal, including nonprofits, government, and the private sector.

As one might expect, however, the cultures of these different actors can also be very different.  One of these differences can be seen in the use of the word “deal,” whose roots can be traced to the world of finance.

At best, this usage can grate on those who come from a nonprofit or public sector background.  At worst, it suggests a certain detachment from the very people whose lives we are working to improve.

The world of social innovation includes many good people with highly diverse backgrounds and skill sets.  This diversity of talent is one of its central advantages.  However, this is one piece of the social innovator’s lexicon that needs to be put to rest.

Posted in Social Impact Bonds / Pay for Success