Students admitted under inclusion programs are just as qualified as their colleagues

A study compares the grades of over a million Brazilian students in the national university exam and concludes that beneficiaries of the quota system and student loan program do not underperform their classmates.

By José Tadeu Arantes  |  Agência FAPESP – Brazilian undergraduates admitted to university under diversity and inclusion programs are just as qualified as their peers, if not better qualified, according to a study that compared the grades of more than a million university students in the National Undergraduate Performance Exam (ENADE) during the period 2012-14. 

The study was carried out by Jacques Wainer, Full Professor at the University of Campinas’s Computer Science Institute in São Paulo State, Brazil, and Tatiana Melguizo, Associate Professor at the University of Southern California’s Rossier School of Education (USC Rossier). FAPESP supported the study by means of a research scholarship that allowed Wainer to work abroad, at USC, on the project “Two results in education – computers and primary education and comparative outcomes of social actions in Brazilian universities”.

The results were published in the article “Inclusion policies in higher education: evaluation of student performance based on the ENADE exam from 2012 to 2014”.

The diversity and inclusion programs considered were the quota-based affirmative action system, which reserves places in undergraduate courses at public (free) universities for minorities (black people, indigenous people, people with special needs) and public-school students from low-income households; the University For All Program (ProUni), which awards full or partial scholarships to low-income students at private (paid) higher-education institutions; and the Student Loan Fund (FIES), which supports undergraduates in attending paid courses.

“Our aim was to find out whether the performance of students who benefited from federal diversity and inclusion programs was equivalent to that of their classmates who did not,” Wainer told Agência FAPESP. “We used data for more than 1m students who sat the ENADE university exam, or a third of the total population of university students who graduated between 2012 and 2014.”

The methodology was based not on a statistical sample but on an exhaustive survey because the subjects covered by the exam are repeated only every three years, such that the subjects covered in 2012 were not covered in either of the next two years (2013-14). The exam focused on technology and applied social sciences in 2012, health sciences in 2013, and exact sciences and humanities in 2014.

“By working with the complete dataset for the period 2012-14, therefore, we were able to analyze the performance of a third of all students who graduated during the period. The result was that the grades of students who had been admitted under the quota system or received FIES loans didn’t differ significantly from those of their classmates, “ Wainer said. “As for students who received scholarships from ProUni, their grades were much better than those of their classmates.”

The different categories of student could be distinguished in terms of whether a student participated in a diversity or inclusion program because when degree earners sit the exam they must complete a form that includes a question asking whether they benefited from the quota system, ProUni or FIES. The answers, which are anonymous, constitute the dataset that the researchers used.

Grade standardization 

To standardize the grades and to make them comparable so that performance could be measured across the entire survey contingent, regardless of the relative difficulty of the papers set in each subject, the researchers subtracted the average of all the grades in the course concerned from each individual’s grade and divided the result by the standard deviation. “This told us by how many multiples of the standard deviation a given student’s grade was higher or lower than the average,” Wainer said.

Thus, however difficult or easy the exam may have been in each case, performance could be compared across all courses and subjects. This was an improvement on previous research of the same kind.

Previous studies have obtained important results, especially in comparing the performance of students admitted under the quota system to the Federal University of Bahia (UFBa) or the University of Brasília (UnB) with that of students who were not admitted under the system, but these studies could not draw significant conclusions because they used data from only one university. Thus, generalizations could be based only on inference. The new study takes a step toward a complete picture by using nationwide data from the ENADE exam.

With regard to quotas, previous studies showed that beneficiaries of the system performed as well as non-beneficiaries, except in “high-prestige” courses, i.e., medicine, law and engineering, among others.

“Our data didn’t allow us to distinguish between students based on whether they had taken high-prestige courses, but we used the following criterion: by combining the variables ‘course,’ ‘university’ and ‘city,’ we defined the ‘classes’ with averages in the top 10% for the specific ENADE exam. Our assumption was that these ‘classes’ corresponded to high-level courses since the students’ grades were well above average. Even in these ‘classes,’ we found no significant differences between quota beneficiaries and non-beneficiaries,” Wainer said.

Statistical significance

“We defined statistical significance as follows. Taking the 5% immediately above average and the 5% immediately below, we decided that the difference between these two groups was not significant in terms of showing whether one was better trained. For practical purposes, such as employability, both groups can be considered indistinguishable.

“This gave us a metric for the statistical significance of these differences. The 5% above or below average refers not to grades but to student numbers – the first 5% on either side of the average grade. The difference between the average grades for these two sets was 0.13. We therefore defined upward of 13% as a statistically significant difference in grades. Any smaller difference than that was interpreted as an equivalent performance. When we say students who received ProUni scholarships performed better than their colleagues, it’s because the difference between grades in this case was far more than 13%. In fact, it was almost 40%.”

The explanation for this difference is the selection procedure used by ProUni, which, among other criteria, requires applicants for scholarships to score at least 450 on the National Secondary School Exam (ENEM) and also enforces strict rules requiring the termination of a scholarship if the undergraduate concerned fails 25% of the subjects taken in a semester.

The quota system, which is designed to include minorities and the poor, has always been highly controversial. Despite the extensive literature now available on this topic, few people who express an opinion about quotas take the trouble to ground their view in research findings, according to Wainer. The idea of reparation for a historic debt, the main justification for affirmative action programs, is fervently supported by some and equally fiercely attacked by others.

“Views on either side rarely change in response to numerical arguments, but for those who want more objective criteria, this study’s conclusions are important,” Wainer said.

“One of the arguments used by the first opponents of the quota system was that it caused a loss to society because students admitted as part of a quota took university places away from more gifted students and, on graduating, became less qualified professionals than the rest. Even I was surprised to find this so resoundingly refuted by our study. If you consider the ENADE exam a reliable gauge of graduate qualifications, you have to agree that the systematization of these data shows that beneficiaries of the quota system are as qualified as non-beneficiaries.”