FAPESP evaluates impact of programs

Three FAPESP programs – Fellowships, BIOTA and Multiuser Equipment (EMU) – will be evaluated in a study conducted by the Foundation’s Scientific Department with the support of the Study Group on Organization of Research and Innovation (GEOPI), housed under Universidade Estadual de Campinas’ Department of Scientific and Technological Policy.

By Fábio de Castro

Agência FAPESP – Three FAPESP programs – Fellowships, BIOTA and Multiuser Equipment (EMU) – will be evaluated in a study conducted by the Foundation’s Scientific Department with the support of the Study Group on Organization of Research and Innovation  (GEOPI), housed under Universidade Estadual de Campinas’ Department of Scientific and Technological Policy.

FAPESP will invite all applicants to EMU and BIOTA programs from 1998 to 2009 and any individuals applying for fellowships from 1995 to 2009. 
 
The questionnaires on EMU programs were sent out in the first week of October. In the second week of October, the Foundation sent out questionnaires on fellowships. The BIOTA-FAPESP program has completed its data collection process.

According to Sérgio Salles-Filho, coordinator of the study, the objective of the initiative is to obtain a snapshot of the perceptions of researchers funded by the Foundation, along with the opinions of those submitting unsuccessful applications.

“The evaluation will provide FAPESP fundamental information so that the Foundation can better direct its support lines and better serve the public, showing society that it is worthwhile to invest in science. We know that the programs have a positive impact, but it is important to measure them to show what they are,” explains the director of Applied Sciences at Unicamp in an interview with Agência FAPESP.

Salles-Filho highlights that the evaluation has three main objectives: evaluating the programs’ impacts on scientific production and society; providing a reference for FAPESP’s planning; and offering some accountability to society for the investments made over the years.

In the case of fellowships, one of the objectives is to help understand in what manner FAPESP Fellowships influence the professional trajectory and production of science, technology and innovation.
    
“We evaluated the effects that FAPESP’s support has afforded Brazilian Society in offering the expressive number of fellowships that range from scientific initiation to doctoral. We can estimate not only the impacts in producing knowledge, but also economic and social developments and the impact on the professional trajectory of the people,” he said.

In the case of EMU, the main objective is evaluating the extent to which the program offered benefits for scientific and technological production, in addition to, once again supplying estimates on its economic and social impact.

“In addition to measuring the impacts of programs in several dimensions, the second objective is to collect information that can improve FAPESP’s actions. Knowing the results, we can correct and improve the Foundation’s actions so that its impacts are more positive. In this manner, the evaluation will also be an instrument of planning,” he explained.

The third major objective of the evaluation, according to Salles-Filho is to be accountable to society for the public resources invested. “We want society to know that it is worth investing resources in these programs,” he says.

According to Salles-Filho, the participation of researchers whose proposals were approved or rejected is important because the study’s methodology demands a basis of comparison between the professional trajectory of those applicants that were funded and those that did not have support.

“The participation of those that had projects denied is fundamental. The evaluation study is like an experiment for which we need a control group. Furthermore, these applicants that were not funded have much to tell FAPESP. Their contribution will help the Foundation to better plan their investments. We want FAPESP to know the results of its investments so that it can improve its service to society as a whole,” he said.

Program evaluation

Each researcher invited to participate in the study will receive a link by e-mail that will give access to an individual questionnaire. The questionnaires open up with the researcher’s information and the project already filled out, based on the Lattes curriculum, in order to facilitate participation. The questionnaires present a set of roughly 20 questions that can be answered in 30 minutes. The data is absolutely confidential.

“There will 700 questionnaires for the EMU program and roughly 52,000 for the fellowship program including improved and denied projects. Each questionnaire will be available for one month. After this, creation of a database and analysis will take more than three months. The results, however, will be available in the month of March in 2012,” explains Salles-Filho.

The research coordinator highlights that FAPESP has experience with similar procedures in other program evaluation. As of 2004, for example, FAPESP analyzed and mapped the equipment park financed by the Foundation, and in 2008, concluded an evaluation of the results of the exchange program, analyzing the professional trajectory of its former fellows.

The two studies were released, respectively, in following publications: Parque de Equipamentos de Pesquisa  [Research Equipment Park] Perfil e trajetória acadêmico-profissional de bolsistas da FAPESP [Profile and Academic Professional Trajectory of FAPESP Fellows].

According to Salles, the results were evaluated in those studies. In the current study, we are looking for an impact evaluation. “This time, the study is focused on the effects that the results of research have had on society from the environmental, social and economic points of view,” he affirmed.

In 2008, the results of program evaluation of four programs: Innovative Research in Small and Micro Companies (PIPE), Partnership in Research for Technological Innovation (PITE) and Young Researchers in Emerging Centers.