For well over a decade, universities, governments, and accrediting agencies\u00a0around the world have been discussing new ways to assess and interpret the basic functions of higher education, including faculty productivity and student learning. In 2006, at the request of the University of North Carolina system’s general administration,\u00a0the UNC\u00a0statewide Faculty Assembly put together an Assessment Task Force.\u00a0One part of the Assessment Task Force was charged with\u00a0thinking about how best to assess and report information about student learning outcomes.\u00a0Specifically, we were asked to evaluate two\u00a0particular assessment instruments, the\u00a0National\u00a0Survey of Student Engagement (NSSE),\u00a0and the Collegiate Learning Assessment (CLA).<\/p>\n
A commission appointed by U.S. Secretary of Education Margaret Spellings\u00a0recommended in September 2006 that:<\/p>\n
The results of student learning assessments, including value-added measurements that indicate how much students\u2019 skills have improved over time, should be made available to students and reported in the aggregate publicly. Higher education institutions should make aggregate summary results of all postsecondary learning measures, e.g., test scores, certification and licensure attainment, time to degree, graduation rates, and other relevant measures, publicly available in a consumer-friendly form as a condition of accreditation. (http:\/\/www2.ed.gov\/about\/bdscomm\/list\/hiedfuture\/reports\/pre-pub-report.pdf<\/a>, p. 23)<\/p><\/blockquote>\n
What we found was that, although the idea of comparing student learning outcomes across universities sounds simple and straightforward, in reality such comparisons are difficult and potentially misleading.<\/p>\n
Follow this link to our final report to the UNC General Administration, which includes both our thinking about assessment more generally, and our evaluation of the utility of the NSSE and CLA instruments:\u00a0 Best Practices for Student Assessment<\/a>. The full report is 22 pages long, but there’s a useful two-page executive summary up front.<\/p>\n
UPDATE<\/strong> 1 September 2012:<\/p>\n
There is progress in the world. In late June, Chris Jackson of the Council for Aid to Education (http:\/\/www.cae.org\/content\/about.htm<\/a>) wrote to me\u00a0about\u00a0changes in some of the technical features of CLA. The changes he outlines in the instrument are welcome, and many of them\u00a0answer the criticisms the UNC Faculty Assembly articulated in 2006 (sadly, this was not a\u00a0direct causal\u00a0relationship!) Here is what Mr. Jackson wrote:<\/p>\n
Greetings Dr. Starrett:<\/span><\/span><\/div>\n<\/div>\nI wanted to bring you up to speed with information related to the CLA which was either unavailable at the time or has changed since you all wrote your report.<\/span><\/span><\/div>\n<\/div>\n\n
- At the writing of the report, the assembly indicated that longitudinal data was not yet available. Findings from our longitudinal project may now be found <\/span>here<\/span><\/a>.<\/span> <\/span><\/li>\n
- Your assembly found that peer group comparisons were not available through the CLA which, at the time, was true. Since then we have introduced peer group comparisons by: Institution size, Minority-serving status, Institution Type (Doctoral, Masters, Bachelors), and Sector (Public vs. Private). Please see this <\/span>sample 2011-2012<\/span><\/a> report for examples (specifically pages 12-13).<\/span> <\/span><\/li>\n
- In that same report (pages 9-10), you\u2019ll also note that we\u2019ve introduced subscore reporting in the areas of: Analytic Reasoning and Analysis, Writing Effectiveness, Writing Mechanics, and Problem Solving. The goal here being the provision of information that will assist participating institutions define specific areas for improvement. All reports that we provide to institutions are accompanied by an underlying data file, so that they have the opportunity to run local analyses to determine whether there are specific patterns (demographics, program participation, course-taking patterns, etc.) that lead to better attainment of the skills measured by the CLA.<\/span> <\/span><\/li>\n
- The report notes\u2014correctly\u2014that in 2006, the CLA value-added model did not control for institutional effects. We have since moved to a <\/span>hierarchical-linear equation<\/span><\/a>, which does control for student characteristics that, to some extent, define an institution. <\/span><\/span><\/li>\n
- Finally, it is true that\u2014at the institutional-level\u2014CLA scores do correlate quite highly with the SAT and ACT. This, of course, does not mean that they measure the same thing (only that groups of students that tend to do well on one assessment also tend to do well on others, which may be a contribution of other factors, socioeconomic status not the least among them). Still, though raw cohort scores do correlate well, institutional <\/span>value-added <\/em><\/span>scores have no correlation with SAT or ACT, meaning that all institutions (highly selective and less-highly selective) have equal opportunity to contribute to student growth in the skills assessed by the CLA. <\/span><\/span><\/li>\n<\/ul>\n
<\/div>\nAgain, I realize that this response was unsolicited. However, we do want you (and others) to know that we listen to participating institutions and are constantly working to refine and better our offerings to ensure that they have genuine utility.<\/span><\/span><\/div>\n<\/div>\nBest,<\/span><\/span><\/div>\nChris<\/span><\/span><\/div>\n<\/div>\n\nChris Jackson<\/span><\/span><\/div>\nDirector of Business Development, CLA & CWRA<\/span><\/span><\/div>\n<\/div>\n<\/p>\n
It’s worth noting that the increased sophistication of the data analysis and reporting may help universities make better sense of the CLA instrument if they choose to use it. But they do not affect the\u00a0general recommendations in the\u00a0UNC\u00a0Faculty Assembly Best Practices report regarding\u00a0the\u00a0kinds of issues institutions should take into account when developing assessment policies.<\/p>\n
Nor can they address the issue of student motivation to do well on the test in the first place. I’ve heard from some students that they regard it as just another hoop to jump through with as little effort as possible,\u00a0on the way to priority registration or a gift card, two of the many sorts of inducements institutions now use in order to get students to spend three hours of their time on\u00a0a test that doesn’t affect their individual academic record.<\/p>\n
Nor, of course, can even the best testing instrument control for the irresponsible media coverage and political use of findings based on test results, as evidenced by the narrow, sensationalistic and utterly misleading\u00a0coverage of the work of sociologists\u00a0Richard Arum and Josipa Roksa (Academically Adrift: Limited Learning on College Campuses<\/em>, University of Chicago Press, 2011) last year.<\/p>\n
More on this later.<\/p>\n
UPDATE 9\/24\/2013:<\/p>\n