Researchers and university officials discuss metrics for social impact of science

The meeting at FAPESP was attended by Youngsuk ‘YS’ Chi, Chairman of Elsevier.

By Heloisa Reinert  |  Agência FAPESP – There are no recipes for successful academic research, and it is extremely difficult to produce an objective metric that shows the world what universities do. Even “research impact” is a concept under construction. Academia and research funding agencies are striving to develop better ways to measure the impact of research on society and to determine what constitutes research excellence. Science is not just a combination of facts, and establishing metrics that accurately reflect the value of research excellence and social development is a major challenge. Moreover, the cost of publishing papers on findings in scientific journals is high for researchers based in countries like Brazil.

To think collectively about these complex issues, representatives of São Paulo state’s universities and FAPESP met on May 29 at FAPESP with Youngsuk ‘YS’ Chi, Chairman of Elsevier. He wanted to listen to the participants’ concerns and see what kind of decision could be made with the help of the data analysis tools offered by Elsevier. It was a constructive dialogue, but many questions remained open.

What does Elsevier think about the social impact of a publication and how its journals can improve the metrics of social impact? Chi offered the widely accepted examples of job creation and a reduction in the number of deaths from disease, adding that more shared ideas will emerge from discussion, so that a consensual definition of impact can be agreed. “There’s been a shift of this kind in the environmental field, although there are still people who deny the existence of climate change,” he said.

Even if impact is defined, the question of what to measure remains. “Let’s measure the impact that can be measured. We should be practical,” he said. In his view, one of the problematic aspects of defining impact is the nature of scientific output, which often includes results that will be transformative only in future.

More data

Another challenge, Chi continued, is that the data available from universities, research institutions and funding agencies is insufficient. Many producers and users of scientific findings work outside academia. He advocated a new metric, which he called “contributive data”, to extend the statistical universe. This could cover data produced by economists, sociologists and professionals in the third sector, for example. In any event, the difficulties of collecting “totalized data” are considerable, he said.

Many questions need to be answered, he went on. How to identify the data that is really important? How to ask for it? Will it be necessary to pay for it or can it be shared? He also mentioned the rising cost of processing and of the technology involved, especially the very expensive hardware.

There is data that is available and data that is not yet collected on researchers, research projects, and collaboration between groups and institutions. Considering that collection, availability and access to all this data requires labor and infrastructure, what kind of analysis would justify this investment? Insufficient collection of data on past and present university students and faculty is a point that deserves attention, according to Chi.

“Universities are designed to be permanent in our society and should therefore collect this data,” he said. The reasoning behind this recommendation is straightforward: knowing all about the careers and connections of researchers funded here and abroad, and whether they return to their alma maters, would help make decisions on which knowledge areas to invest in.

People will agree to share such data if they are persuaded of all this, Chi said. In practice, however, it is far from easy to change mindsets. According to Carlos Américo Pacheco, CEO of FAPESP, foreign researchers supported by FAPESP often ask why certain kinds of information are included in its Virtual Library. 

Difficulties can be surmounted, Chi replied. Permission is needed to collect sensitive data such as religious demographics, for example. Another possible solution entails the use of codes that can be interpreted only by machines. This point, however, aroused disagreement among the participants in the discussion, as some did not take the view that technology can surmount all difficulties.

The huge amount of data available raises the question of interpretation. It is not sufficient for computers to analyze the data. Furthermore, there are doubts about how artificial intelligence (AI) will benefit society. Pacheco asked about next steps in AI regarding planning, evaluation and data intelligence. Chi said AI would be predictive in an ideal scenario. “I’m not sure this will be the next stage in AI, but I’d like it to anticipate what will probably happen with a high degree of certainty,” he said. 

Open science and payment by authors

Paulo Nussenzveig, Pro-Rector for Research at the University of São Paulo (USP), said payment by authors and their funders was unfair, benefiting the rich countries and penalizing the rest, especially middle-income countries like Brazil. The policy prospered with the advance of open access for readers to content on the internet. Publishing in prestigious scientific journals has become an obstacle for researchers in developing countries.

According to Chi, when research was conducted by the rich countries after world war two, it was a good idea for readers to pay because to a degree they participated in the same academic environment. The situation has changed, and other models have emerged. Many countries with less purchasing power now do research and want to publish. Chi said China, which now accounts for a significant proportion of the papers published, needs to take part in the debate about the funding of publications. The same goes for India, which has rapidly increased its share.

Asked by Nussenzveig and Márcio de Castro, FAPESP’s Scientific Director, about the possibility of a policy to waive or discount fees for Brazil, Chi replied in the negative. “We can refine a discount policy and create more specific categories, but it will never be satisfactory,” he said. “It won’t answer the fundamental question of who in the scientific community should be responsible for which part of the publication of scientific papers.”

Universities should choose the model that best meets their needs, he said – payment by readers, by funders or by authors – and can rely on Elsevier to offer a solution.

Sergio Salles, a professor at the State University of Campinas (UNICAMP), and Nussenzveig highlighted the ongoing movement to improve the ways in which the outputs of scholarly research are evaluated, as a consequence of the DORA Declaration, which calls for an end to the practice of correlating the journal impact factor (the number of citations of articles published in a given period) to the merits of a specific scientist’s contributions.

Fátima Nunes Marques, who heads USP’s Office for Management of Economic Performance Indicators (EGIDA), asked whether assessment of journals is the best way to go. Chi replied that publishers are not the main actors in this case, adding that universities should decide on the key criteria. “Our role is to present one or two solutions that can help with this. We can be part of the solution,” he said. 

The other participants in the meeting were Carlos Gilberto Carlotti Júnior, Rector of USP; Edson Cocchieri Botelho, Pro-Rector for Research at São Paulo State University (UNESP); João Marcos Travassos Romano, UNICAMP’s Pro-Rector for Research, and Angela Christina Lucas, a professor at that university and advisor to Romano; Lia Rita Azeredo Bittencourt, Pro-Rector for Graduate Studies and Research at the Federal University of São Paulo (UNIFESP); Pedro Sérgio Fadini, Pro-Rector for Research at the Federal University of São Carlos (UFSCar); Wagner Alves de Carvalho, Pro-Rector for Research at the Federal University of the ABC (UFABC); Adalton Ozaki, a professor at the Federal Institute of São Paulo (IFSP); and Fernando Menezes, FAPESP’s Chief Administrative Officer, as well as other representatives of Elsevier.