For well over a decade, universities, governments, and accrediting agencies around the world have been discussing new ways to assess and interpret the basic functions of higher education, including faculty productivity and student learning. In 2006, at the request of the University of North Carolina system’s general administration, the UNC statewide Faculty Assembly put together an Assessment Task Force. One part of the Assessment Task Force was charged with thinking about how best to assess and report information about student learning outcomes. Specifically, we were asked to evaluate two particular assessment instruments, the National Survey of Student Engagement (NSSE), and the Collegiate Learning Assessment (CLA).
A commission appointed by U.S. Secretary of Education Margaret Spellings recommended in September 2006 that:
The results of student learning assessments, including value-added measurements that indicate how much students’ skills have improved over time, should be made available to students and reported in the aggregate publicly. Higher education institutions should make aggregate summary results of all postsecondary learning measures, e.g., test scores, certification and licensure attainment, time to degree, graduation rates, and other relevant measures, publicly available in a consumer-friendly form as a condition of accreditation. (http://www2.ed.gov/about/bdscomm/list/hiedfuture/reports/pre-pub-report.pdf, p. 23)
What we found was that, although the idea of comparing student learning outcomes across universities sounds simple and straightforward, in reality such comparisons are difficult and potentially misleading.
Follow this link to our final report to the UNC General Administration, which includes both our thinking about assessment more generally, and our evaluation of the utility of the NSSE and CLA instruments: Best Practices for Student Assessment. The full report is 22 pages long, but there’s a useful two-page executive summary up front.
UPDATE 1 September 2012:
There is progress in the world. In late June, Chris Jackson of the Council for Aid to Education (http://www.cae.org/content/about.htm) wrote to me about changes in some of the technical features of CLA. The changes he outlines in the instrument are welcome, and many of them answer the criticisms the UNC Faculty Assembly articulated in 2006 (sadly, this was not a direct causal relationship!) Here is what Mr. Jackson wrote:
- At the writing of the report, the assembly indicated that longitudinal data was not yet available. Findings from our longitudinal project may now be found here.
- Your assembly found that peer group comparisons were not available through the CLA which, at the time, was true. Since then we have introduced peer group comparisons by: Institution size, Minority-serving status, Institution Type (Doctoral, Masters, Bachelors), and Sector (Public vs. Private). Please see this sample 2011-2012 report for examples (specifically pages 12-13).
- In that same report (pages 9-10), you’ll also note that we’ve introduced subscore reporting in the areas of: Analytic Reasoning and Analysis, Writing Effectiveness, Writing Mechanics, and Problem Solving. The goal here being the provision of information that will assist participating institutions define specific areas for improvement. All reports that we provide to institutions are accompanied by an underlying data file, so that they have the opportunity to run local analyses to determine whether there are specific patterns (demographics, program participation, course-taking patterns, etc.) that lead to better attainment of the skills measured by the CLA.
- The report notes—correctly—that in 2006, the CLA value-added model did not control for institutional effects. We have since moved to a hierarchical-linear equation, which does control for student characteristics that, to some extent, define an institution.
- Finally, it is true that—at the institutional-level—CLA scores do correlate quite highly with the SAT and ACT. This, of course, does not mean that they measure the same thing (only that groups of students that tend to do well on one assessment also tend to do well on others, which may be a contribution of other factors, socioeconomic status not the least among them). Still, though raw cohort scores do correlate well, institutional value-added scores have no correlation with SAT or ACT, meaning that all institutions (highly selective and less-highly selective) have equal opportunity to contribute to student growth in the skills assessed by the CLA.
It’s worth noting that the increased sophistication of the data analysis and reporting may help universities make better sense of the CLA instrument if they choose to use it. But they do not affect the general recommendations in the UNC Faculty Assembly Best Practices report regarding the kinds of issues institutions should take into account when developing assessment policies.
Nor can they address the issue of student motivation to do well on the test in the first place. I’ve heard from some students that they regard it as just another hoop to jump through with as little effort as possible, on the way to priority registration or a gift card, two of the many sorts of inducements institutions now use in order to get students to spend three hours of their time on a test that doesn’t affect their individual academic record.
Nor, of course, can even the best testing instrument control for the irresponsible media coverage and political use of findings based on test results, as evidenced by the narrow, sensationalistic and utterly misleading coverage of the work of sociologists Richard Arum and Josipa Roksa (Academically Adrift: Limited Learning on College Campuses, University of Chicago Press, 2011) last year.
UPDATE 9/24/2013:
For an explanation of what Arum and Roksa actually said about their CLA data, see my subsequent post, “Academically Adrift with Molly Broad,” at http://pages.charlotte.edu/gregory-starrett/2013/09/23/academically-adrift-with-molly-broad/