Evaluation of science, technology and innovation policies assists long-term programs

Experts discuss the impact of STI policy evaluation on the effectiveness and efficiency of policies in Latin America.

By Maria Fernanda Ziegler  |  Agência FAPESP – Science, technology and innovation (STI) policies are extremely important to any country’s social and economic development, but evaluation of their effectiveness and impacts is essential to comply with society’s demands, remedy their weaknesses and reinforce their strengths.

Researchers affiliated with the University of Campinas (UNICAMP) in São Paulo State, Brazil, in partnership with the Science and Innovation Policy Evaluation Repository (SIPER), developed by the Manchester Institute of Innovation Research (MIoIR) under the aegis of a broader European Union initiative, evaluated 143 documents comprising analyses of STI policies of Latin American countries such as Argentina, Brazil, Chile, Colombia, Mexico and Uruguay.

MIoIR is part of the University of Manchester.

The initiative and the results of the study were presented to an international workshop held on July 5, 2019, at FAPESP in São Paulo City as part of the Innovation Systems, Strategies and Policy (InSySPo) project conducted at UNICAMP with support from FAPESP’s São Paulo Excellence Chair (SPEC) program.

The workshop on STI policy evaluation and feedback was a follow-up to the International Symposium on Innovation Systems, Strategies and Policy held by InSySPo on July 3-4. Both the workshop and the symposium were chaired by Nicholas Vonortas, a professor of economics and international affairs at George Washington University in Washington, DC (USA).

“We all understand the need to ensure the continuity of long-term STI policies and to evaluate these policies in a structured manner. But it’s even more challenging to ensure the use of these evaluations as feedback into policy formulation and implementation,” said FAPESP President Marco Antonio Zago.

The study identified three categories of STI policy evaluation. “The evaluations analyzed were Latin America’s input to SIPER, an existing platform, so they extended the base of the evaluated STI policies. The study found that the Inter-American Development Bank (IDB) significantly influenced the design of these evaluations because it funds some of the policies in question. It also identified three categories of STI policy evaluation in these countries,” said Adriana Bin, a professor in the University of Campinas’s School of Applied Sciences (FCA-UNICAMP) and principal investigator of the study.

According to Bin, evaluations in the first category, present in parts of Brazil and Argentina, are more “summative” (judgment oriented) and are performed above all by institutions independently of the government.

“These evaluations focus on results, quasi-experimental design, and econometrics. They follow the IDB pattern to some extent but without in-depth investigation of aspects relating to impact,” she said.

The second category, most common in Mexico and Chile, is “formative,” meaning that assessments of this type are mainly concerned with policy design and the appropriation of targets and objectives. “These are nonexperimental evaluations and basically not concerned with results or impact. They tend to be more descriptive,” Bin told Agência FAPESP.

The third category can be found in several Latin American countries. “It focuses on content and process as well as results and impact. This category uses quasi-experimental and nonexperimental methods and deploys the largest number of instruments for assessment purposes,” Bin said.

The study, which is supported by FAPESP and the National Council for Scientific and Technological Development (CNPq), a Brazilian government agency, will analyze the impact of these evaluations on public policies.

“The reports don’t offer a great deal of information on the impact of these evaluations on policies because decisions are made after the evaluations are delivered. Therefore, as part of the project, we plan to develop strategies to help policymakers understand the role of evaluation in political decision making,” Bin said.

Big data in policy decisions

The workshop held at FAPESP marked the completion of five years of work by InSySPo, which aims to bring top-tier researchers from abroad to set up research centers at universities in São Paulo State.

“Studies have shown a relative lack of knowledge exchange among Latin American countries in terms of STI policy. Hence, the importance of a platform like SIPER, which has now been expanded to countries in the region. It’s important for a simple reason: learning,” Vonortas told Agência FAPESP. “We don’t need to reinvent the wheel. We need a better wheel instead of reinventing it all the time. In all these years, InSySPo has succeeded in doing this by creating collaborative networks that span Latin America, as well as Europe, the United States and Asia.”

In addition to evaluating STI programs and policies, InSySPo analyzes innovation systems, strategic partnerships and networks for innovation, and knowledge-intensive entrepreneurship.

“InSySPo has been very positive, and we intend to expand it. We’ve created a very good group, and now, we want to include the University of São Paulo as a partner. We also want to include big data and its influence on policy decision making. A new revolution is coming, and it’s related to data,” Vonortas said.

More information: www.ige.unicamp.br/spec.