|Faculty Resources - A Model for Evaluating Student in Academically Based Service-Learning|
Printed in Connecting
Cognition and Action:
Why is evaluation necessary?
Instructors who use academically based service (service learning) have found it difficult to evaluate student performance. There is no clearly defined academic content which can be easily tested for mastery as in a mathematics course. There are no major texts about which a student can reflect and write a paper as in a literature course. There is no widespread agreement on the appropriate skills students should be mastering as in an applied music course. Furthermore much of what instructors hope that students learn - the development of attitudes and values - is typically regarded as too subjective for effective measurement.
Nevertheless, evaluation is essential for at least two reasons:
Thus, difficult or not, we have to develop effective means of evaluating students. This paper is a presentation of one possible solution to this problem.
What makes for effective evaluation?
We would like to suggest four components of an effective evaluation system:
Goals and objectives
We have adopted the distinction between goals and objectives often made in business - that goals are broad, general, overarching statements of what one hopes to accomplish and objectives are specific statements about what one hopes to accomplish in limited domains. The selection of goals and objectives and their careful expression can be a difficult exercise for the instructor! Service experiences can vary widely and can involve aspects of which the instructor is totally unaware. The formation of goals is not intended to stifle this creative aspect of service, nor to force a rigid conformity. Rather this formation is intended to make it a purposeful activity designed to enhance, enrich, and support the achievement of the academic goals of a course. Without explicit academically oriented goals, students typically have "meaningful experiences," but are unlikely to relate these experiences clearly to the academic content of the course. Also, as we shall see below, reflective thinking is very much a goal directed activity. Thus well-formed goals can greatly encourage reflective thinking on the part of students. While the process of goal selection is unique to each situation and requires too much creativity to be routinized, some principles of effective goal selection can be identified:
An example of a well-formed collection of goals for an economics Course is presented in Table 1. Note under item II. B. how explicit the instructor has been in showing students how to relate academic content to their service experiences.
TABLE I SERVICE-LEARNING PROJECT FOR ECONOMICS 335, LABOR ECONOMICS
To increase students' understanding of and concern for:
outcomes of labor markets which are harmful to individuals and families
ways to alleviate these harmful outcomes Some examples of such harmful outcomes are long-term unemployment, underemployment, and discrimination
I I. Objectives:
A. Knowledge Objectives:
1. To learn about an organization which is working to help those who are having difficulty in the labor market: its goals and objectives, its structure, its activities, its sources of support, its impacts
2. To learn about at least three cases of unemployment or underemployment: education /employment history of the person, attempts to find employment, effects on family, present situation, what is being done to help them.
B. Skill Objectives:
1. Improve ability to apply economic theories of labor markets (e.g., neoclassical models and institutional models of wage determination: discrimination, and unemployment) to actual situations, using them to help understand the situation
2. Improve ability to use actual situations and experiences to evaluate the adequacy of economic theories
3. Improve expository writing skills.
C. Attitude/Value/Commitment Objectives:
1. Form or deepen a concern for those for whom the labor market does not give good outcomes
2. Form or deepen a commitment to use available opportunities to help those for whom the labor market does not give good outcomes.
D. Service Objective:
1. Provide assistance to an organization which is helping people who are having difficulty in the labor market.
III. SERVICE Activities
Spend 15-20 hours over the course of 8-10 weeks working for an organization helping the unemployed or underemployed to find jobs which fit their needs. The work could vary according to the organization's needs. For example, it could involve doing a literature search on some topic of importance to the organization, office work (as long as a means to learn about the organization is provided), or providing transportation to job interviews. In some cases such service may not be a direct source of much learning but would be a form of "payment" for the time the organization spent helping the student learn about it and its clients. Each student should have the opportunity to learn about the organization and its activities and to learn about at least three people the organization is working with. It's best if the people could be met in person, but if this is impossible, confidential briefings are acceptable.
Please note: Entire text of the syllabus is published in another Campus Compact publication, Redesigning Curricula, pp. 4248.
Communicating what is learned
Requiring the student to communicate what he or she has learned has two objectives: to provide the instructor something concrete on which to base the evaluation and to provide the student an opportunity to synthesize what was learned.
Note that we are not directly assessing students' performance but are indirectly assessing it through their written work. Direct assessment is ultimately impossible - the only truly direct measure would be one that summed the impact of the service experience on the student, the client, the agency, the instructor, and the academic institution over the lifetimes of all of them. A semi-direct technique that is sometimes employed is observation of the student in the service setting. This is widely used in some situations, for example, in student teaching. However, it has some significant disadvantages which render it unsuitable for most academically based service:
We see three primary means for students to communicate their service experiences: group discussion, journals, and theory-to-practice papers, although other means such as exams and such as one-on-one discussions between instructor and student are possible .3 Another means of communication - feedback from the client - is also an important component of evaluation.
Group discussion. is an essential part of the academically based service experience. It provides an opportunity for students to compare their experiences with others, to sort out which aspects of their experience are unique and which are general, and to assimilate their learning. However, group discussion does not lend itself well to evaluation as the instructor's attention is too fully given to filling the mentor role to allow for evaluation and there is little opportunity to stop and assess a contribution as it is being made. Group discussion, therefore, won't be pursued further here.
Perhaps the most widely used method for evaluating service experiences is the evaluation of student journals. Journals provide the opportunity for students to reflect on their experiences while they are still fresh in their mind and they provide for somewhat more freedom of form than does a formal paper. Theory-to-practice papers also provide opportunities for reflection. While not as immediate, they typically expect a more thorough and careful analysis. Both are valuable, serving somewhat different purposes. However, some recent research suggests that the quality of reflection occurring in theory-to-practice papers depends a great deal on the choice of question on which students are asked to write.' Thus such questions need to be constructed quite carefully.
Reflection and Measurement
Much research has been done in recent years on the reflection process.5 We will draw heavily on that research here in developing a technique for evaluating student journals and theory- to-practice papers. Writing about teacher education, Dorene Doerre Ross provides the following definition of reflection :6
At a general level, reflection is defined as a way of thinking about educational matters that involves the ability to make rational choices and to assume responsibility for those choices ... The elements of the reflective process include:
It seems that the process of reflection Ross describes for student teaching is generalizable to any service situation. Ross does not explicitly define "dilemma;" however, her context suggests she is thinking of a student teacher who finds himself or herself in a somewhat unfamiliar situation in which appropriate action must be selected. Thus the student must recognize his or her uncertainty, compare and contrast the situation to other more familiar situations, develop a framework within which to understand the situation, and consider alternatives and their possible consequences. This is no different from what a student in a service learning situation must do. In order to provide an effective technique of evaluation, we reformulate Ross' outline of the reflective process using a simple, three-step rubric:
Note that Ross' last step (examining and evaluating) cycles back to the first two steps of our three step process. The same rubric can then be applied to subsequent experiences in the service setting with the student's actions as part of the material Subsequently to be reflected upon.
Those who have studied reflection intensively are eager to point out that reflection is more than a learning technique; rather it is a life-long habit of thought characteristic of highly skilled professionals who work in complex situations. If they are correct, the development of reflective skills is of great importance. Thus the academically based service experiences can make a valuable contribution to a student's professional life by helping students develop this skill. However, we need to acknowledge from the beginning that we can only make a small contribution to this process. Furthermore our evaluations are poor attempts to measure such a rich aspect of the students' thinking. However, some feedback is better than none. It seems advisable then that before Students are sent out to their service experiences and before they are assigned the writing of journals, they be given the above rubric, Ross' definition of reflection, and some opportunity to discuss them. As for the actual evaluation of the journals or papers, we are suggesting a three-factor measure using the three steps of the rubric as the factors. The factors should not be consolidated into a single grade as the measurement is not quantitative - the numbers used are ranks, not quantities.
Evaluation of each step needs to be done with the service goals in mind. That is, the goals provide a frame within which both the student and the instructor can reflect on the service experience. Also students should think about the goals in writing their journals or papers. Student journals are extraordinarily unique! Reading a journal while looking for evidence of how the student has reflected on his or her efforts to achieve the goals can give a great deal of insight into the student's thinking.
We recommend that the three factors be evaluated using a simple three level scoring system, followed by a verbal explanation of why the particular level was chosen. Table 2 presents the levels .7
TABLE 2 CRITERIA FOR ASSESSING LEVELS OF REFLECTION
Note that a single journal excerpt is unlikely to show all of the items listed as characterizing a particular level. Rather they serve as typical representatives of thinking at each level. Furthermore, while the levels seem clear and intuitively appealing, it is sometimes difficult to see which level best describes a student's journal.
The theoretical foundation behind this scale is a model of the development of reflective judgment developed by Kitchener and King and adapted by Ross.' Ross' adaptation is presented in Table 3. Note that we have collapsed her five stages into three levels in order to make the levels easily understandable by both the instructor and student. An instructor who wished to provide a more discriminating scale could use all five stages.
TABLE 3 STAGES IN THE DEVELOPMENT OF REFLECTIVE JUDGMENT THE INDIVIDUAL:
Views the world as
of differences of viewpoints
differences of viewpoint
Views knowledge as contextually based
Exhibits all stages
listed in Stage 4
For example, consider the evaluation of a journal submitted in the economics course cited earlier. Suppose a student has submitted a journal after an initial service learning experience of providing transportation for Mr. Moore, an unemployed person, to a job interview. The journal retells Mr. Moore's story, shows some empathy, discusses his educational background, and suggests that Mr. Moore needs some education to improve his marketable skills. As noted earlier, the goals and objectives for the service learning experience serve as the framework within which the student acts and reflects and the instructor evaluates. Thus the instructor turns back to Table I and notes that the students recognizes the harmful outcome of the labor market for Mr. Moore and shows concern. Hence some encouragement for achieving this goal is in order. However, there is no mention of the sponsoring organization, only one client is discussed, there is no application of economic theory, no evaluation of theories, concern is developing but there is no evidence of commitment yet, but some assistance was provided to the organization. After making these observations, the instructor compares the journal to Table 2 and identifies reflective levels. A possible evaluation might be as follows:
Observation -I- "You're off to a good start. You told Mr. Moore's story well and you have some genuine empathy for him. You need to look more closely at the organization and meet a couple more clients."
Analysis -I- "You are beginning to analyze the causes of Mr. Moore's chronic unemployment. What are some other factors besides education? How do the theories we have discussed in class help us understand his situation?"
Synthesis -I- "Your suggestion about the possibility of Mr. Moore improving his marketable skills is a reasonable one. How feasible is that for him? What obstacles might there be? What steps could he take? If you drive him to an interview again, are there practical ways you could be of help to him?"
Client feedback could be from the person or persons directly helped or from the agency. Ideally it would involve both, but that may not always be possible. It needs to be simple and brief so as not to consume much of the client's or agency's time. For Instance, Mr. Moore who was provided transportation in the previous example might be asked to complete a form which consists of just three questions (or even just the first two)':
In what ways was the student helpful? Were there ways the student was not helpful? How could the student have been even more helpful?
These evaluations need to be shared with students as soon as possible and students should be expected to respond to them, either in some form of discussion or in their journal. The same questions could be used by an agency staff member at one or more points in the student's service.
How can students be given opportunity to improve?
The evaluation system provides for a cycle of improvement. That is, as students observe, analyze, and synthesize, they can observe the effect of their actions and can repeat the cycle. The instructor can enhance students' improvement by giving feedback as frequently as possible and calling students' attentions to changes in their reflective process over time. Feedback must be given early enough that students have the opportunity to change. Hence, a minimum evaluation would be at midterm and at the end of the term. More often would be better. Furthermore, discussion groups after the first evaluation should provide students who wish to participate a chance to discuss their evaluations-much can be accomplished by talking through nuances in situations and by allowing students to help each other see multiple aspects of situations.
In summary, then, our solution to the problem of how to evaluate student performance in academically based service is:
Note that we are not primarily measuring the student's performance against the goals themselves, in the sense of ticking off items accomplished on a checklist. Rather we are assessing the level of reflection evidenced by the student in the process of attempting to accomplish those goals.
The model presented here has been primarily worked out as a theoretical solution to the problem of student evaluation. Much research is needed on the model. First and foremost, it needs to be tested by instructors in several different types of courses. Based on the results of such testing, improvements will need to be made.
Also, measurements of changes in student's reflective levels over the duration of an academically based service experience would be of great value. Many questions could be asked, for instance: How much change can an instructor reasonably expect? Are there any patterns to when the change occurs? Do different types of experiences lead to more change than others? To what extent are changes retained after completion of the experience? If students are given explicit instruction in the nature of the reflective process, does that yield higher reflective levels?
Dr. James Bradley is a Professor in the Department of Mathematics and Computer Science at Calvin College in Grand Rapids, Michigan.
Campus Compact: The Project for Public and Community Service is a coalition of over 500 college and university presidents who believe that institutions of higher education hold a primary responsibility to foster students' sense of civic responsibility and to contribute to the welfare of their communities. The Compact works on a national level to cultivate discourse and support for issues of public service; develop resources materials, grant programs, workshops and institutes; and support a network of state and specialized offices. to provide targeted programs for campuses. Campus Compact, Box 1975, Brown University, Providence, RI 02912-1975, 401-863-1119.
Campus Compact is a project of the Education Commission of the States, a nonprofit, nationwide interstate compact formed in 1965. The primary purpose of the commission is to help governors, state legislators, state education officials and others develop policies to improve the quality of education at all levels. It is the policy of the Education Commission of the States to take affirmative action to prevent discrimination in its policies, programs and employment practices. Education Commission of the States, 707 17th Street, Suite 2700, Denver, CO 80202-3427, 303-299-3600.
The Project on Integrating Service with Academic Study (SAS) provides training, advice on strategy and technical assistance to colleges and universities that are working to build community service into their teaching and research. While it provides some resources for individual faculty (such as a database of courses and syllabi), the project works primarily with campuses attempting to institutionalize service-based teaching and research. Campuses are typically motivated to integrate service into teaching and research by some mixture of concerns regarding citizenship, diversity or building community. Much of the project's actual work focuses on integrating service with the curriculum, because we recognize that the curriculum is the core institutional structure around which most campuses are organized.
Project on Integrating Service with Academic Study Staff
Sandra Enos, Project Director
Marie Troppe, Project Associate
Kate Jackson, Project Intern
Aileen Keenan, Project Intern
1995 ECS/Campus Compact