The following is an excerpt from SIRC’s report on the i3 program. In it, national experts and former Obama administration officials weigh in on lessons learned and the program’s possible future.
EPILOGUE
Viewed in isolation, the i3 program – while imperfect – appears to have been generally effective. But the newly-renamed Education Innovation and Research (EIR) program does not exist in a silo.
The program exists in a larger policy context – one that includes other federal programs, state and local education agencies, and perhaps most importantly, a political context that has become very different in the aftermath of the 2016 elections.
How does EIR fit into this larger context? As one administration exits the stage, what fate awaits it in the next one? If the new Republican administration and GOP Congress choose to keep it, how might it change?
New Administration, New Priorities
In late 2015, Congress reworked i3 as part of the Every Student Succeeds Act, bipartisan legislation that replaced No Child Left Behind. The new EIR program now bears a bipartisan imprint.
Still, as a program originally associated with the Obama administration, its fate under the incoming administration is uncertain. The Trump administration may decide to eliminate it, but it may also see it as useful tool for furthering its school choice and accountability agendas. Moreover, support for evidence-based policy and tiered evidence initiatives (like EIR) more generally has been growing among Republicans on Capitol Hill.
When asked, Rick Hess, the Director of Education Policy Studies at the American Enterprise Institute, began by making the pessimistic case. “I don’t know what will be done in the new administration,” he said, “but I assume the new administration and Congress will take a hard look at the full array of Obama initiatives, including this one.”
“If i3 had happened outside the context of Race to the Top and had not been locked arm-in-arm with foundations on Common Core, I think I would have looked upon it differently, because historically the idea of public-private partnerships has a lot of appeal.”
“On the other hand, there are folks who are interested in school choice. They might see it as a vehicle for encouraging more choice programs and expanding efforts to study their impact,” he said.
The president-elect has pledged $20 billion for school choice. His Education Secretary-designee, Betsy DeVos, is a strong school choice and accountability advocate. Federal support for charter schools is overseen by the Office of Innovation and Improvement, the same division that runs EIR. As a program that provided support for charter school initiatives like KIPP and New Schools for New Orleans, EIR could be useful to the incoming administration.
Others point to potential support in Congress. “Betsy DeVos will be extremely important, but there is also more appetite on the congressional side than there used to be,” said Grover “Russ” Whitehurst, a former director of the Institute of Education Sciences under President George W. Bush.
“There is still bipartisan momentum to increase evidence use within federal policy,” agreed Martin West, associate professor at the Harvard Graduate School of Education and a former senior education policy advisor to Sen. Lamar Alexander (R-TN). “I think the status of programs like EIR hinges on that broader momentum.”
Support for evidence-based policy has been growing among Republicans in recent years. In the summer of 2016, House Speaker Paul Ryan and other members of the House Republican leadership made evidence a central component of a policy plan that it ran on in the fall, called A Better Way. Although the plan did not mention EIR specifically, it endorsed tiered-evidence initiatives in general.
The broader focus on evidence has also drawn cautious support from analysts at conservative organizations like the American Enterprise Institute, Manhattan Institute, and Heritage Foundation. In November, the Heritage Foundation endorsed the increased use of evidence in the federal budget process.
Given these varied sources of potential support, EIR’s fate is unclear. If it were to be kept in place, however, how might it be changed?
Where Does EIR Belong?
One issue that has emerged is the program’s organizational placement. EIR is partly an education research program. Does it belong in the Office of Innovation and Improvement (OII), where it is located now, or at the Institute of Education Sciences (IES)?
“I think it belongs on the program side at the Department,” said Nadya Dabby, the last Obama appointee to oversee OII. “Many of i3’s greatest contributions extend beyond using or generating evidence.”
“The thing I heard over and over from grantees is that it changed how their organizations think about data, improvement and evidence—beyond their i3-funded project. You only get that kind of organizational change if the work at its core is led by practitioners and not the researchers,” she said.
“IES did the evidence reviews, but we put the mechanics in OII. That made sense to us,” said Robert Gordon, who previously served at both the Department of Education and OMB during the Obama administration and helped design the program. “We still got tons of expertise from IES.”
What were the benefits of putting the program in OII? “It is no knock on IES – what they are doing is incredible and really pushing the evidence movement forward,” said Shane Mulhern, the program’s most recent director. “However, EIR is about evidence in practice. We want those grantees to be in a continual cycle of learning and improvement.”
“It shouldn’t matter where the evidence comes from, but that’s not the way human behavior works – people leave evidence on the shelf unless they have some personal connection to it,” said Dabby. “i3 has changed practices and efforts in schools and education nonprofits across the country. You only get that if the primary ‘clients’ of the program are schools and nonprofits.”
Others think the program should be moved. “I was the director of IES. You need a wall between you and politics,” said Whitehurst.
“EIR ought to be run by IES. OII could easily be politicized and I don’t think anyone wants these funds to be compromised by the sense that someone has their thumb on the scale,” he said. “That has not been true so far, but it would be better in IES.”
“Perception of politicization also matters,” said West, the former advisor to Sen. Lamar Alexander. “Within the Department there are strong pressures to align everything with the overall policy priorities. This is supposed to be bottom-up and field generated. What you want is a more open-ended process, with no absolute priorities.”
“I would argue that i3 and EIR are research programs and belong in IES because it is non-partisan and non-political,” said Ruth Neild, the most recent director of IES. “The lesson of i3 is that there is a hunger for evaluation among organizations that do not have the research expertise to compete successfully for IES research grants. They need an easier on-ramp – but it should be housed in a scientific agency with protections for independence and a staff with research training,” she said.
“I don’t have strong opinions about this as long as attention is paid to lessons we learned about what works well and is successful,” said Jim Shelton, a former Deputy Secretary of Education who also oversaw the program in its early years.
“If they can build that at IES that’s good. If it’s at OII, that’s fine. I am less concerned about where and more concerned about capacity,” he said.
Innovation
As discussed elsewhere in this report, one of i3’s main goals was to stimulate and leverage innovation in education. So far, however, it appears to have been less effective at this than its other goals. How might its work on innovation be improved?
“I’m not shocked that validation and scale-up grants were more successful. We were criticized for the same old, same old,” said Gordon. “My reaction always was that this is a government program. Identifying initiatives with strong evidence and building them is a relatively straightforward task for government to perform well.”
“Identifying early stage brilliant ideas is less objective and requires a different kind of insight. That’s not to say government can’t do it well, but it is more difficult,” he said.
Others were more critical. “Yes, research is a business you want government to be in,” said Hess of the American Enterprise Institute. “It is a public good and it is an appropriate role for the national government.”
“But there is a difference between that and development. In medical research, NIH spends $40 billion on bench science. That’s people in labs figuring out the building blocks that later gets monetized by other actors who turn it into drugs. We don’t want government doing the second part,” he said.
“The danger is that i3 was intended to leverage philanthropy. You wind up having the feds shoulder-to-shoulder with specific philanthropic agendas,” he said. “It is easy to talk about programs that were supported. What isn’t noticed was what did not happen or was discouraged.”
“Research is an appropriate federal role, it is a public good” said Whitehurst, the former IES director under President Bush. “But the priorities should not be exclusively or primarily the administration’s priorities, they should come from the research and practice communities along with political players among which a presidential administration is but one.”
“The people closer to the work have a lot more knowledge than career bureaucrats,” he said. “A state department of education is not going to turn around schools, and federal officials aren’t either. If anything is top-down, it should come from legislation, not bureaucrats.”
“I share Russ’s concern about innovation being a poor competency for the feds,” said West. “Deciding which innovations are worthy of encouragement is not a natural role for them. But I am reluctant to recommend that it should be eliminated altogether. The development-validation-scale process should be more closely linked. There should be more of a pipeline approach.”
Obama officials overseeing the program were keenly aware of how difficult spurring innovation can be. “One of the things I struggled with was when I was looking for expert reviewers,” said Shelton. “I leaned toward putting them on scale-ups. In hindsight, I should have put them on the development grants. There was already a significant track record on the scale-ups. Recognizing something that doesn’t have a track record is harder.”
“We were forced to make decisions based on reading reports. It is difficult to find an early stage investment firm that does not deeply engage with an organization they are planning to invest in,” he said. “In some cases, we need to test using intermediaries. They are in the business of finding and supporting those organizations.”
One other idea proposed by the Obama administration, but never approved by the Republican-controlled Congress, was a new initiative within i3 modeled after the Defense Advanced Research Projects Agency (DARPA). Called ARPA-ED, the program would have focused on a few high-value research projects using technology.
The proposal drew early qualified support from Hess. It may draw greater support from the GOP Congress if a variant is proposed by the incoming administration. “On the researcher side, we need a DARPA that moves quickly and allows risky investment, not five year grants,” said Whitehurst. “We need that on the researcher side of things.”
Evidence
“I think on the evidence building, we got a lot of things right,” said Shelton. “I think that we were smart to arrange for technical assistance to help the grantees structure their evaluations to get the most rigorous, appropriate evidence. There were a number of folks who had consultants, but they found they were putting together a study that would not have served them well in the end.”
“Evidence-building is a boutique endeavor,” said Dabby. “There are a small number of organizations that do it well. So far, most of the evidence that has been generated under i3 has been specific to the project that they took on. Did the project work? It is a little binary: red light, green light.”
“If we really want to democratize evidence the way ESSA envisions, we need different and better evidence,” she said. “We still need to know if it works, but also what about it was most impactful and why.”
“I think if I had it to do over, I would have spent more of the scale up money on implementation studies, not impact studies,” said Shelton. “Because one of the questions is when you move to scale, given the evidence is already high, the real question is: can you scale with fidelity and produce results?”
What about the pace of research? The first grants from i3 were made in 2010, but it was not until the end of the administration that those grants began to produce results.
“How do we speed up the process?” asked Dabby. “On i3 and EIR, you can apply for a three-year grant. It’s the same amount of money, so you get more per year if you apply for a shorter grant. But few people do it and the ones that do have all needed extensions.”
“We didn’t have to wait until now to know that those early grants were showing success,” said Shelton. ”We could see what was happening in a lot of the interim studies.”
“But these are questions you only ask at the beginning,” he said. “No one thinks about the most important breakthroughs that started 25 years ago at NIH. What we really need to do is keep going, so there is a constant pipeline of new things that are coming out.”
Scale
Scale was a central feature of i3 and it continues to be under EIR. In interviews, there were different ideas about how well this has worked or how it could be improved.
“I don’t see i3 as successfully scaling,” said Whitehurst. “Why do you need to scale up Success for All and Teach for America? They are widely-scaled already. The notion that the feds should pay for that doesn’t make sense. Spending most of the money to create wider implementation of programs that are already well-established doesn’t support innovation.”
“What is the theory of action behind scaling?” he asked. “What motivates education administrators writ large? i3 is a bribe. They compete, they agree to do things, they do it. My guess is that most of that goes away once the money isn’t on the table anymore.”
“You need a different rationale,” he said. “One is accountability. It’s fine to prime the pump, but if you don’t have backup to make the local principals care, then you are spitting into the wind.”
“The peak funding was $650 million. There is only much scale you can accomplish with that,” said West. “Scale is a misnomer for describing anything that EIR can do. The right way to think about it is that the scaling grants are part of the evidence building process – testing something in multiple settings.”
“What we were trying to do in i3 was innovation, evidence, and scale,” said Shelton. “But at a higher level, I think we were trying to create a marketplace for evidence where some people are looking for evidence to make choices and other people are providing evidence to be chosen. By creating that marketplace, the incentives are aligned. If people can keep that kind of framing in mind, regardless of the program, that is the thing that is most important,” he said.
“It is ironic that the highest evidence thresholds in the Department are attached to competitive programs of a few hundred million dollars, while billions are going out with no evidence requirements at all,” he said. “If even a small portion of Title I dollars prioritized the top two evidence tiers, even just 10 percent, you would have a $1.4 billion marketplace. That would significantly change the incentives for people seeking Title I funding.”
He saw some of that coming from regulations being developed under ESSA, but he urged legislators not to be too restrictive. “The regulations need to be allowed to adapt quickly based on what we learn to continually become more effective,” he said. “The notion that we would design something this complicated and be expected to get everything right the first time is flawed.”
Even if the right incentives were put in place, he still saw a need to help states and local school districts.
“Think about how corporates decide whenever they are trying to do something new,” he said. “If they are implementing a new software system in HR, most of the time they will bring in outside organizations who have done it many times before to help with implementation.”
Dabby thought that the Regional Education Laboratories at IES could help states and school districts with this. “They could leverage the i3 and EIR grants,” she said. “That also keeps people in their lane. Supporting states and districts to build and leverage research is part of their role.”
She also thought that state-focused external organizations should provide support to states and districts that want to engage deeply with ESSA’s evidence provisions for formula funds.
Outside the Bubble
The national experts interviewed for this report were brimming with ideas. For these ideas to take hold, however, they must be widely embraced by educators, policymakers, and the public. So far, there is only modest evidence of that.
Is the concept of evidence too esoteric? How can decision makers and the broader public come to understand its importance?
“It is important for us to build evidence,” said Shelton, who has given the topic a lot of thought, “but what is most important is that it is conveyed with actual stories of how that evidence reflects improvements in real people’s lives.”
“We need to move the needle in a visible way, not just in a statistical way,” said Whitehurst. “We need something where Aunt Sarah can see it and get it.”
–
Baltimore is only 40 miles from the halls of power in Washington, DC, but it feels like it is worlds away.
Deep in the heart of one of its toughest neighborhoods is Franklin Square, a K-8 school that is one of the city’s rare gems. The neighborhood suffers high rates of teen pregnancy and recidivism. Boarded up row houses are a common sight. But the story inside the school is very different.
A few years ago, it began working with Success for All as part of its i3 grant. “Our scores were not the best in reading and we were looking for a vehicle that would help our scholars reach success and do it quickly,” said Terry Patton, the school’s hard charging principal.
How well is it working? This time the answer did not come from an evaluation, but from a young woman for whom that answer meant everything.
Tyria is the mother of two children at Franklin Square. She knows too well how tough the city can be because she was homeless once and she lived it. Now she is a school volunteer.
“I like being part of the school,” she said. “Everybody at Franklin is family. If you need a haircut, whatever you need, you can get it from Franklin Square.”
Her two children attended three other schools before they came to Franklin. They were behind when they arrived, but have made enormous progress since then.
“My son is in third grade now. His reading has gotten better. There are so many things they didn’t know,” she said. Her daughter is now a year ahead of grade-level reading. “From where she came from to now is so different.”
So is her life trajectory. “I want her to graduate, go to college, get a good job,” she said softly.
“She says she’s going to be a teacher or a nurse.”
Related
- Report – Investing in Innovation (i3): Strong Start on Evaluation and Scale, But Greater Focus Needed on Innovation (January 19, 2017)
- ED Announces First Round of Grants Under i3’s Replacement (December 15, 2016)
- K-12 Education Bill Advances Evidence-based Policy, Replaces i3 (December 7, 2015)