Paper Title: Incentives and services for college achievement: Evidence from a randomized trial
Principal investigator: Viviane Pires Ribeiro
Authors: Joshua Angrist, Daniel Lang and Philip Oreopoulos
Location of the Intervention: to Canada
Sample Size: 1600 students
Main theme: Education
Main Variable of Interest: Academic results
Type of Intervention: Evaluation of the main strategies used to improve academic performance.
Methodology: Randomized study
Angrist, Lang, and Oreopoulos (2009) report an experimental evaluation of strategies aimed at improving academic performance among college freshmen. One treatment group received academic support services. Another received financial incentives for good grades. A third group combined both interventions. Service utilization was higher for women and for individuals in the combined group. The combined treatment also raised grades and improved the academic standing of women. These differences persisted until the end of the sophomore year, although the incentives only occurred in the freshman year.
Evaluation Context
In recent decades, there has been a growing interest in interventions aimed at increasing the frequency and completion of higher education in the United States, especially for low-income students. This interest has resulted in efforts to increase enrollment, and among these efforts, student aid based on need and merit, tax exemption programs, tuition grants, employment assistance, and infrastructure improvements stand out. Thus, the resulting expenditures are justified in part by empirical evidence suggesting that there are substantial economic returns to college education and graduation.
Student academic performance is not always satisfactory. Many students take longer to complete their courses than expected – and one possible reason for this poor performance is a lack of preparation. In particular, many students have poor study skills. However, motivated by the view that the return on these skills is high, the traditional response to academic performance problems has been a series of academic service strategies (such as time management workshops, goal setting, academic guidance, etc.). Furthermore, in some specific cases, student support services are combined with psychological support services.
Another strategy aimed at improving student performance is merit-based scholarship programs – which have primarily focused on a small number of high-achieving students. Seeking to broaden the number of students benefiting from these programs, an attempt has emerged in the scholarship field to use awards and financial incentives to motivate students with good academic performance, not just those with spectacular achievements. These programs are, in part, an effort to attract better students to public institutions. But they are also motivated by the view that merit-based aid increases interest in school and makes students more willing to develop good study habits.
Non-experimental evidence on the effectiveness of student services is mixed. More rigorous studies with designs Experimental and quasi-experimental studies, especially for high school students, present a more promising picture. However, as far as the literature has shown, neither academic support strategies nor financial incentives have been the subject of large-scale evaluations using Design research on random assignment in a traditional university setting. So this is the gap that Angrist, Lang and Oreopoulos (2007) seek to fill.
Intervention Details
Angrist, Lang, and Oreopoulos (2009) report on a large randomized field experiment designed to evaluate key strategies used to improve students' short- and long-term academic performance. Specifically, the study seeks to identify whether students with low academic performance show better outcomes when offered additional university services, merit scholarships, or both, and whether the effects of these interventions extend beyond the period during which they were provided.
The study reports on the Student Achievement and Retention Project (STAR), a randomized evaluation of academic services and incentives conducted at one of the satellite campuses of a large Canadian university. In American terms, this institution could be considered a large state school with heavily subsidized tuition. Approximately 1.600 first-year students participated in the project, most of whom were from the local area and had a typical high school education.
The STAR demonstration involves three types of treatment: a service strategy known as the Student Support Program (SSP), an incentive strategy known as the Student Scholarship Program (SFP), and an intervention that offers both strategies, called SFSP. The SSP offered 250 students access to peer counseling and supplemental instruction in the form of Facilitated Study Groups (FSGs). Peer counselors are students trained by the study program to offer academic suggestions to support the student in their first year of school. Meanwhile, FSGs are specific class sessions designed to improve students' study habits and learning strategies, without focusing solely on specific course content.
The SFP offered 250 students the opportunity to earn merit scholarships for maintaining good, but not necessarily the best, grades in their first year of school. Participants in this program received $5.000 in cash for a B (an average grade of 3,0) or higher, or $1.000 in cash for a C+ or B- (an average grade of 2,3 to 2,9). However, to be eligible for a scholarship, students had to take at least 4 courses per term and enroll to attend the second year of their program.
The third group of 150 students was offered both the SSP and the SFP. Finally, the STAR demonstration included a control group of 1.006 students, with whom the program operators had no contact.
Methodology Details
The study conducted by Angrist, Lang, and Oreopoulos (2009) reports a randomized field experiment involving two strategies designed to improve academic performance among full-time undergraduate students at a large Canadian university. One treatment group (“services”) received peer counseling and organized study groups. The other group (“incentives”) was offered merit scholarships to help first-year students maintain good, but not necessarily excellent, grades. A third treatment group combined both interventions.
For the purposes of this analysis, all first-year students who entered in September 2005, except those with grade point averages in the top quartile, were randomly assigned to one of three treatment groups or a control group. One treatment group received a range of support services, including access to mentoring by other students in more advanced classes and supplementary instruction to promote critical thinking. A second group received cash prizes for achieving a certain grade point average. In addition, a third treatment group received a combination of services and incentives, an intervention that had not been previously analyzed. The control group was eligible for standard university support services but received nothing extra.
Results
The first finding of the study is that young women use the services much more than young men. The second finding related to service use is the emergence of a strong interaction between the offer of scholarships and service uptake; students in the combined group were more likely to use the services than those offered no opportunity to win scholarships. The incentives, therefore, had the immediate short-term effect of increasing the rate at which students sought academic support.
The effects of the STAR intervention on students' academic performance are more varied. Alongside the gender differential in acceptance rates, the results on academic performance show a significant impact on grades only for women. The effects on women are greater in the combined group (scholarship and services). The combined group also earned more credits and had a significantly lower rate of academic internships at the end of the year.
The authors highlight that women in the combined group continued to outperform the rest of the STAR population in the second year – despite scholarships and services being available only in the first year. This result suggests that study skills or study habits acquired during the first year helped to increase subsequent performance.
Student interest in support services was lower than expected. On the other hand, interest in services, reflected in service uptake and usage rates, was markedly higher in the group that also received monetary incentives. Interest in and usage of services was also much higher for young women than for young men. Peer counseling was considerably more popular than supplementary instruction for both sexes. The peer counseling intervention clearly deserves further exploration, as does the use of achievement incentives to increase interest in services.
Several patterns emerge from the STAR results. First, students who received peer counseling and supplemental instruction services without scholarships did not perform better than students in the control group. This may be because adherence rates were relatively low in the service treatment groups, as low consent rates dilute the intention-to-treat effects. On the other hand, a 2SLS analysis adjusting the intention-to-treat effects for non-participation reveals a level of precision sufficient to detect theoretical service effects equal to 0,25σ in the combined sample of men and women. The results, therefore, suggest that the benefits of services alone are relatively modest.
Lessons in Public Policy
Based on the study results, it is possible to raise some questions, such as: why were the effects greater in the combined group; and why did women show better responses to the programs than men.
Angrist, Lang, and Oreopoulos (2009) suggest two possible reasons for the greater impact of the combined program. The significant impact of fall grades on both the SFP and SFSP groups suggests that the scholarships alone were a strong motivating force. The SFSP group had the advantage, however, of ongoing mentoring and support. These students received more frequent contact in the form of bi-weekly emails from peer advisors. Greater interaction with senior peers may have facilitated adaptation and integration into the new university environment. The service-only (SSP) group had similar access, but acceptance rates in the SSP were low. For the mentoring service, acceptance rates for women in the SFSP were almost double those for women in the SSP. Thus, the SFSP appears to have successfully combined increased motivation with a well-defined channel for sustained improvement in outcomes.
Although there is no simple explanation for gender differences in response to incentives and services, the authors point out that women generally appear more committed to post-secondary study than men. Part of this difference seems to be due to better study habits among women, suggesting that they may be more motivated to perform well in school and therefore take advantage of programs like STAR.
Previous studies have also confirmed gender differences in response to incentives and services, with women showing better responses to the programs.
References
ANGRIST, Joshua; LANG, Daniel; OREOPOULOS, Philip. Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, v. 1, no. 1, p. 136-63, 2009.
