The Teagle Liblog

At the recent meeting of the National Forum on the Future of Liberal Education, I promised to put together some references to important work concerned with assessment and other approaches to improving liberal education. As I drafted this, I realized that many of the topics raised practical questions about how one might exercise low key but effective leadership on campuses. Hence the “leadership questions” in the text below:

1. In recent years, many institutions have reformulated their understanding of “liberal education” as the cumulative development of cognitive and personal capacities such as critical thinking, analytical reasoning, cogency of written and oral expression, moral reasoning, civic engagement, etc.

Background: Under the auspices of its “Liberal Education and America’s Promise” program (LEAP), the Association of American Colleges and Universities (AAC&U) has articulated one widely followed set of “essential learning outcomes,” which are commonly known as LEAP outcomes.

Many colleges and universities have similar learning outcomes. An AAC&U poll found that about four out of five institutions have specified such learning outcomes for undergraduate education. See page 3.

Leadership question: Does this match my understanding of “liberal education” and that of my department or institution?

2. While there is broad agreement among faculty members, business leaders, and others about the importance of such capacities, students often have little idea that these are intended outcomes of their education

Background: The same AAC&U poll surveyed senior administrators and found only about 5% were confident that their students understood the institution’s learning goals. See page 5.

Leadership question: How can I best convey the learning goals that I regard as most important with my students?

3. Most of the goals in point 1 above are cumulative, that is, they are not likely to be achieved in a single course or project; but progress toward many of them can to some degree be assessed by currently available tools and instruments.

Background: There are many kinds of assessment instruments. Here are some frequently used types:

• Surveys of student responses: The most widely used of these is the National Survey of Student Engagement or NSSE, which your institution, which your institution may already be using.

• Direct measures of student performance: While perhaps not the most widely used, one instrument that is frequently discussed is the Collegiate Learning Assessment or CLA, which uses well-crafted performance tasks to gauge the quality of student critical thinking, problem solving, analytical reasoning, and writing.

Because the CLA is designed to measure student performance at an institutional level, translating results to the individual or classroom level is challenging (and is not without problems). CLA in the Classroom is an “on the ground” alternative that provides faculty with the know-how to adapt the CLA performance tasks—or even to develop their own—for use in their own courses.

Another way to directly assess student learning at the individual or course (or even departmental) level is through the use of rubrics. Through its Valid Assessment of Learning in Undergraduate Education or VALUE initiative, the AAC&U has developed a set of rubrics that evaluate student accomplishment of LEAP outcomes.

Worth knowing about as well is the Wabash National Study of Liberal Arts Education, which uses a combination of student surveys and direct measures of performance to get a holistic sense of students over their four years of college.

Leadership question: Does my institution use these instruments and do faculty learn from them?

4. Data from these instruments, when aggregated and analyzed, can show which practices are most effective, as well as which are not.

Example: Using data and research on student learning, George Kuh of Indiana University has documented a number of “high impact educational practices,” that is, practices that have been proven to benefit many college students. An excerpt of Kuh’s findings, as well as a link to purchase the whole document, can be found here.

Leadership questions: Are all these “high impact practices” in place on my campus and what percentage of students take advantage of them? In general, does the development of new academic programs or practices on my campus start with data? That is, are data used to determine the kinds of programs or practices that need to be implemented to genuinely improve student learning?

5. Course evaluations can be transformed from “student satisfaction surveys” to student self-assessment of their progress toward learning goals.

Background: Robert J. Thompson and Matt Serra. “Use of Course Evaluations to Assess the Contributions of Curricular and Pedagogical Initiatives to Undergraduate General Education Learning Objectives.” Education 125, no. 4, Summer 2005, pp. 693-701. Not available on line but see also: http://www.teagle.org/liblog/entry.aspx?bid=1&id=58.

Also worth checking out is the work of the IDEA Center, and in particular, its “student ratings of instruction” initiative, which is focused on “learning and curricular objectives.”

Leadership question: When was our course evaluation questionnaire last revised?

6. Good analyses of assessment data are under way at Wabash and at other centers, including the University of Virginia. Fellow Josipa Roksa (UVA) can bring us up to date on this work; in the meantime see http://teis.virginia.edu/files/doc/highered/Quarterly_Summer_07.pdf and http://www.web.virginia.edu/iaas/reports/subject/competencies/overview.htm .

Also worth knowing about is the work of the New England Consortium on Student Learning and Accountability.

Leadership question: Who on my campus is best informed about such matters? Is there any campus-wide venue for dialogue about such matters?

7. Cognitive science is yielding some insights about student learning at the college level. Fellow Elise Temple can help on this matter, I am sure, but for a start, see also the list of “Findings” from the Teagle Collegium at Columbia University.

Leadership question: Are cognitive psychologists or neuro scientists at your institution doing any work along these lines?

8. Among the many other promising approaches to increasing student learning is the Reacting to the Past pedagogy project run by Mark Carnes at Barnard College.

Leadership question: Has anyone I know tried any of these simulations, or role- playing games and with results?

9. There are many websites that provide resources and forums for discussion of assessment material. Here are just a few:

• AAC&U

• Duke University

• New Leadership Alliance for Student Learning and Accountability

• North Carolina State University

• The Reinvention Center

• The Teagle Foundation

• The University of Michigan

Leadership question: As I learn more about student learning where can I best share my insights to others?

10. For an overview of assessment based projects on various campuses see

• The National Institute for Learning Outcomes Assessment (NILOA).

Post a Comment | Email This Article | Subscribe to E-Updates

Countless sessions at January’s CHEA conference and the AAC&U annual meeting were devoted to great work on assessing student learning. One of those sessions—at AAC&U—was led by the good folks at the National Survey of Student Engagement. It got me thinking about the parallels between student engagement in learning, and faculty and staff engagement in assessment. Our NSSE colleagues tell us that students learn best when they are challenged, when they engage in active and collaborative learning; when they interact frequently with faculty; when they take advantage of enriching educational experiences; and when they are supported. It dawned on me that these same five benchmarks are associated with robust practices in assessment—which is, after all, essentially a giant exercise in organizational learning. For assessment to grow and thrive, faculty and staff need to be challenged, to create and collaborate; to communicate; to be coached, and to be celebrated.

Of all these benchmarks of effective assessment practice, the one that I believe has my benefited my institution the most is collaboration. We have been involved in a terrific array of partnerships both within and beyond St. Olaf, and each and every one of them bears some responsibility for the award we recently received from CHEA for “institutional progress in student learning outcomes.” First, we have benefited tremendously from inter-institutional collaboration. Several years of grant support from the Teagle Foundation, and from the National Institute for Technology and Liberal Education, have enabled assessment partnerships with Carleton, Macalester, and Grinnell Colleges that have helped us invent, implement, interrogate, and institutionalize attention to evidence of student learning. One of our projects has now grown to include 50 other liberal arts institutions. Other invaluable opportunities for “active and collaborative learning” about assessment have been provided by AAC&U; for example, our current approach to assessing General Education was inspired by what we heard from other schools during a round-table discussion at an AAC&U Network conference. As we said about these inter-institutional collaborations in our most recent report to the Higher Learning Commission, “Together we have exchanged—and acted on—ideas for designing assessment instruments, recruiting students to participate in assessment projects, conducting further analysis of our findings, identifying appropriate audiences for assessment results, presenting findings in ways that engage our audiences, and making meaning out of the results. We have supported each other in the inevitable difficult moments that attend innovation and organizational learning, and we have celebrated one another’s successes as well. Our inter-institutional assessment projects have built interpersonal and inter-institutional bridges in ways we never could have imagined.”

Second, our assessment efforts have benefited from intra-institutional collaboration involving not only faculty, but also staff and students. For example, last summer, we assembled a faculty-staff-student team to develop a new assessment instrument focused on the essential learning outcomes of a college education, as articulated in AAC&U’s LEAP initiative, which we administered to incoming first-years during their fall orientation week, and which we’re about to administer to graduating seniors as well. The team was a great illustration of the power of intra-institutional partnership; it included two members of our faculty assessment subcommittee (a philosopher and a chemist); myself (a political scientist); the College Registrar (a professor of French); the Associate Dean for First-Year Students; the faculty Director of Writing; the staff Director of Institutional Research; our staff Senior Research Analyst; our staff office coordinator (who also happens to be a recent graduate); the president of our student government association; and a student research assistant. Thanks to their work, we’ve gained valuable insight into how our incoming students conceptualize key outcomes such as “critical thinking,” “effective writing,” and “global perspective,” and what they think they need to do to develop those outcomes. We learned about their priorities for college learning and their prior experiences (or lack thereof) in high school. We discovered that they are much more likely to look inward rather than outward in tracking their progress as learners, relying on their own sense of ease and engagement with the subject matter rather than on extrinsic indicators like grades or comments on papers. We found out what they were most worried about and what they were most looking forward to as they anticipated their first year of college. Because our “Essential Learning Outcomes Assessment” questionnaire resulted from a faculty-staff-student collaboration, it reflected the “whole person” that we aim to educate—and many of the students told us they found it a welcome opportunity to remember why they had come to college in the first place. We can hardly wait to see what we learn from our seniors!

These are just a few of the ways in which “active and collaborative learning” among St. Olaf faculty, staff, and students—in the company of many colleagues at other institutions—have contributed to the assessment practices highlighted in the NSSE presentation and affirmed by our CHEA award. We are grateful for these partnerships in support of the learning that all our students need and deserve. If you make friends, you can assess; if you assess, you’ll make friends!

Many times a good teaching idea comes our way, and we are able to work it into the mix of strategies we include in our own teaching. But it is a greater challenge to promote adoption of good strategies in a more systemic way, so that the improvements in learning and teaching are not limited to a few individuals, but are employed in multiple courses and programs. With support from the Teagle and Spencer Foundations, a group of KU faculty members are using collaborative course design to enhance students' reading of scholarly research and their writing of integrative analyses of that research, and our work includes a plan to help the practices spread beyond a few faculty members.

The idea for collaborative course design came from a project at UC Berkeley, as described by the then Associate Librarian, Patricia Ianuzzi. Rather than waiting for some students to need help with writing or library research skills, the Berkeley teams designed staged assignments and feedback to scaffold those skills for all students. Jennifer Church-Duran of KU libraries was familiar with this model, and she led a pilot redesign program in which library and writing center colleagues worked with five professors to build staged assignments into their courses. The faculty members were delighted to see an immediate and noticeable jump in the quality of term papers developed through this collaboration, and we all wondered how we could spread this practice at KU.

We recognized immediately the challenge of scaling up the effort, for there simply would not be enough people in the library instruction and writing center programs to meet even a modest increase in requests to participate, especially in our large lower-division classes. When Bob Thompson from Duke University offered us a chance to be part of a cluster of grants on undergraduate writing and thinking (funded by the Teagle and Spencer Foundations), we initiated graduate student fellowships as a way to accomplish well-informed and collaborative course redesign on a larger scale. The Graduate Student Fellows (GSFs) are teaching assistants already assigned by their departments to work in courses with enrollments of 100 or more, so they bring knowledge of the specific goals and content of their courses. With added support from the grant, they spend time working with librarians and with writing consultants learning about the key skills those groups teach and about designing assignments that help students gradually acquire those skills.

After 18 months of this project, the initial results are very encouraging. The first two courses with GSF participation found that both synthesizing research and crafting written reports were done better than prior semesters, and they also showed some improvements in comparison with other courses in the same fields that did not have GSF design assistance. We continue to replicate that basic finding while adding a GSF program and support to four additional courses in each subsequent year of the grant. We are using the AACU Value Project meta-rubrics on written communication and critical thinking as guides for describing the quality of student work, and we also look at course-based grades and a standardized test of college level writing and thinking. Among other benefits of the project, our colleagues demonstrate continuous inquiry into the effectiveness of teaching based on the quality of understanding demonstrated in students' course projects.

While that glass is certainly half full (or filling steadily, anyway), it requires a lot of time and attention to promote even modest adoption of teaching approaches that demonstrably enhance important features of students' intellectual development. Why is it that higher education generally has not embraced many available evidence-based practices that would ramp up student retention and completion? At our center, I often point to our very small library and suggest that there are more good ideas in those few books, journals, and video reports than any of us could implement in a lifetime of teaching.

Many in higher education (and our critics) point out that academic values favor research over teaching, leading faculty members (even at smaller, teaching-oriented campuses) to cap their efforts in teaching to preserve time for continued discovery in their fields. While that may be true, legislating or altering the economic and intellectual forces that sustain those priorities is beyond any individual's power. Without deciding on the best mix of discovery and teaching, there is another aspect of our culture that may be easier to adjust: what do we mean by good teaching and how do we recognize it?

A consensus has emerged that "scholarship of teaching and learning" should be valued at all campuses, but that emphasis sometimes contributes to the problem. When well-meaning chairs, deans, and provosts call for their faculties to publish about their teaching, they inadvertently divert limited faculty time and energy from teaching into discovery. Many journals, funding sources, and professional societies prefer to support and present research about teaching, and give less attention to the challenges and accomplishments of implementing effective teaching. Often, promotion and tenure committees only recognize publicly visible excellence in promoting learning when it is presented as discovery scholarship.

If we want colleagues across the spectrum of our higher education system to learn about, use, and refine the best practices known to our profession, then visible evidence of excellence in generating learning must be recognized as the highest form of teaching, not as a weak version of discovery. If, for example, we define excellent teaching as publishing survey results about learning, then faculty members will spend their creative energy being amateur members of the faculty of education. We need instead to honor our colleagues' public inquiry into student success in their own courses as an achievement in its own right and the highest form of intellectual teaching.

Randy Bass (Georgetown University) and I shared in AAUP's Academe a dialogue we have had for years, working to find a balance among the benefits of shared discovery and shared success in the implementation of others' discoveries in challenging circumstances. I think our Teagle/Spencer project at KU attempts to make excellence in learning a visible and honored example of the finest in teaching. It is not research into principles of learning and motivation, but it is difficult and a very public intellectual work conducted without the carefully constructed conditions of a laboratory enterprise. I think we need to follow Boyer's lead and honor public treatments of such inquiry into learning as wonderfully scholarly teaching; only then will larger numbers of our colleagues give valuable time to learning about and incorporating known good practices into their own teaching.

Any nuanced conversation about assessment in higher education recognizes many layers of performance and of measurement, ranging from the knowledge, skills, and understanding typical of any individual course to the general skills characteristic of a well-educated person such as critical thinking, communication, integrative analysis, and quantitative literacy.

I have found two frameworks for meta-evaluation very generative for how I go about teaching and measuring the impact of my teaching. One is articulated in the book Teaching for Understanding, edited by Martha Stone Wiske. Based on Project Zero in Boston, the authors argue that students demonstrate deep understanding when they remember ideas and material without re-memorizing and when they can use what they have learned in novel contexts. The chapter by David Perkins makes a compelling case for finding understanding in contextualized performance rather than in largely symbolic representations of relations among ideas and variables.

In 1998, I reframed my examinations from essays about conceptual systems and abstract representations to word problems in which students had to use those systems and representations to analyze and solve problems presented in vignettes of hypothetical situations. To my dismay, many students who were quite capable of reproducing my conceptual language were largely incapable of using the ideas on a real problem. I spent the next ten years learning how to teach differently so that students could use conceptual tools, as well as represent them formally.

A second big idea was a model of intellectual development initially put forward by William Perry, who was describing the informal ways of knowing of typical students. A substantial literature and a richly nuanced intellectual position have evolved from the original writing; see the Center for the Study of Intellectual Development for a comprehensive bibliography. I have a simplified variant on the theme, and it has proved useful in thinking about my teaching and my students' learning. Students typically arrive in college having largely learned that assignments have correct answers, known by teachers, and their task is to identify and correctly use that answer. After a few college courses, students recognize that there are often many answers and perspectives brought forth to illuminate an assignment or problem, and they fall into an uncritical relativism that functions as epistemological anarchy--all answers are in some way correct. When education works well, some students come to recognize that evidence, analysis, and argument can differentiate among alternatives, and some answers are more equal than others. In the Perry community, this is a form of commitment that embraces multiple perspectives and invites learners to the difficult task of critical evaluation of evidence and reasoned construction of their own positions.

In assessing my students' growth along these lines, I first followed the traditional line of using an "instrument" to get directly to the essence of students' thinking, and I was guided by the published work of Marcia Baxter Magolda. Lacking the resources to implement her full protocol of measuring tools, I worked with students to fashion a crude but reliable set of questions that ask students about their ways of knowing and their preferences for learning from authority or from discovery. After several years of teaching, I finally found a few methods that nudge students toward some commitment to critical evaluation. Now I am following Perkins' ideas in Teaching for Understanding by observing the development of critical evaluation and synthesis within course assignments such as term papers and research-based writing. No results as yet, but I derive some satisfaction from examining genuine samples of intellectual work rather than answers to my decontextualized prompts.

All this attention to students' thinking about knowledge has sensitized me to observing thinking among my faculty colleagues. We are masters of critical analysis, being well prepared and even eager to question and challenge the evidence, arguments, and conclusions of scholars in our own fields. Many of us, however, back away from a commitment to constructing a complex understanding when we read outside our primary field of scholarship. I may read and cite cultural anthropologists in my teaching, but I often present the work packaged exactly as I received it. I presume that I cannot bring sufficient knowledge and sophistication to challenge the conclusions I am offered, behaving in this regard very much like a beginning student. I notice this often among colleagues as well, as for example when I hear natural science or humanities professors cite Piaget in support of their teaching methods.

And coming full circle to assessment, I occasionally see familiar patterns in the way faculty members talk about measuring education. Some remain in the early undergraduate mode, citing studies that demonstrate the validity of this instrument or that. Without knowing how to ask questions about what observed predictions and associations demonstrate the utility of proxy measures, faculty simply accept the authority of testing specialists without questioning or understanding their methods or assumptions. Some more adventurous colleagues find multiple opinions among a range of testing researchers and claim that measurement cannot be done at all. A variant on that position is the notion that education is an ineffable thing that cannot in principle be measured, but both positions result in the same intellectual paralysis we observe in students trapped in uncritical relativism.

If Perry's original observations provide a lesson for assessment, it is that we need to commit ourselves to the harder task of learning enough about measurement that we can identify and/or construct the best possible ways to understand what our students are learning. There is a path through the unreliability of measures and the challenges of identifying appropriate intellectual work as a sample of complex cognition and affect, but it will take effort and openness to evidence and argument. We are very good at these skills within the confines of our own fields of study, and we need to learn to use those skills in new contexts in which we were not taught. That is the lesson to be taken from Teaching for Understanding; we enjoy seeing transferable learning in our students and we can revel in accomplishing it ourselves.

Is there an easy way to move our own community further in this intellectual development? Probably not, but perhaps we can start with the same interdisciplinary and integrative opportunities many call for in our teaching. One of Boyer's four scholarships is integration, and it is a less common form of scholarship than disciplinary discovery. Integration may also be undervalued. If we practice what we teach about integrative understanding, then perhaps we can break through our paralysis on measurement and begin to enjoy constructing representations of what our students know, can do, and understand.