Printed in Connecting Cognition and Action:
Evaluation of Student Performance in Service-Learning Courses
Published by Campus Compact, 1995

Why is evaluation necessary?

Instructors who use academically based service (service learning) have found it difficult to evaluate student performance. There is no clearly defined academic content which can be easily tested for mastery as in a mathematics course. There are no major texts about which a student can reflect and write a paper as in a literature course. There is no widespread agreement on the appropriate skills students should be mastering as in an applied music course. Furthermore much of what instructors hope that students learn - the development of attitudes and values - is typically regarded as too subjective for effective measurement.

Nevertheless, evaluation is essential for at least two reasons:

• Without it, the student does not derive the benefit of the mature reflections of a faculty mentor. Students may (or may not) discuss their experiences with peers. Evaluation, however, provides a structured, systematic way for students to get helpful feedback on their activities; such feedback can be a major contributor to learning.'
• Evaluation provides for student accountability. The sponsoring institution has an ethical responsibility to the client with whom the student works and perhaps an agency as well. Evaluation provides a means by which the institution can at least attempt to fulfill this responsibility.

Thus, difficult or not, we have to develop effective means of evaluating students. This paper is a presentation of one possible solution to this problem.

What makes for effective evaluation?

We would like to suggest four components of an effective evaluation system:

• Well-chosen, clearly articulated goals and objectives on the part of the instructor
• A means for students to communicate their experiences to the instructor
• A measurement technique
• Opportunity for the student to improve through feedback

Goals and objectives

We have adopted the distinction between goals and objectives often made in business - that goals are broad, general, overarching statements of what one hopes to accomplish and objectives are specific statements about what one hopes to accomplish in limited domains. The selection of goals and objectives and their careful expression can be a difficult exercise for the instructor! Service experiences can vary widely and can involve aspects of which the instructor is totally unaware. The formation of goals is not intended to stifle this creative aspect of service, nor to force a rigid conformity. Rather this formation is intended to make it a purposeful activity designed to enhance, enrich, and support the achievement of the academic goals of a course. Without explicit academically oriented goals, students typically have "meaningful experiences," but are unlikely to relate these experiences clearly to the academic content of the course. Also, as we shall see below, reflective thinking is very much a goal directed activity. Thus well-formed goals can greatly encourage reflective thinking on the part of students. While the process of goal selection is unique to each situation and requires too much creativity to be routinized, some principles of effective goal selection can be identified:

• The selection of goals goes hand-in-hand with the selection of the service activities themselves. If the goals are selected solely on basis of the academic objectives of the course, they maybe unsuited to the particular service experiences available; on the other hand, if the experiences are selected and/or structured without reference to the academic goals of the course, students may find little relationship between academic content and service. Thus creativity is required of the instructor to select and structure service experiences and to select goals for the experiences which comport well with the academic goals for the course and which are achievable within the particular service setting available.
  
• The objectives need to be quite explicit in showing students how to relate service experiences and academic course content. Without such direction, many students will not make the connection at all, some will see the connection vaguely, and only a few will see the connection clearly.
  
• The goals and objectives need to be expressed simply and clearly. Some writers advocate that all objectives should be measurable. Such a requirement may or may not be helpful. Sometimes writing an objective so that it is quantified can add clarity. For instance, the objective "Get to know people who are underemployed or unemployed." can be clarified by writing it as "Spend at least one hour with three distinct individuals who are underemployed or unemployed." However, some valuable objectives do not lend themselves well to quantification. Requiring measurability forces people to either neglect such goals or to replace them with measurable surrogates which don't state what is actually intended. Thus it seems preferable to encourage instructors to state their objectives clearly and to use quantification whenever it helps clarity.
  
• Goals and objectives need to be written so that the instructor as well as the student can tell when they have been achieved.
  
• If an agency is involved in the service experience it should at least be informed of the goals and objectives; if the agency wishes to be, it should be involved in goal and objective selection as well.
  
• Goals and objectives need to be selected with consideration of the well-being of the service client as well as the student.

An example of a well-formed collection of goals for an economics Course is presented in Table 1. Note under item II. B. how explicit the instructor has been in showing students how to relate academic content to their service experiences.

TABLE I SERVICE-LEARNING PROJECT FOR ECONOMICS 335, LABOR ECONOMICS

I. GOAL:

To increase students' understanding of and concern for:

• outcomes of labor markets which are harmful to individuals and families

• ways to alleviate these harmful outcomes Some examples of such harmful outcomes are long-term unemployment, underemployment, and discrimination

I I. Objectives:

A. Knowledge Objectives:

1. To learn about an organization which is working to help those who are having difficulty in the labor market: its goals and objectives, its structure, its activities, its sources of support, its impacts

2. To learn about at least three cases of unemployment or underemployment: education /employment history of the person, attempts to find employment, effects on family, present situation, what is being done to help them.

B. Skill Objectives:

1. Improve ability to apply economic theories of labor markets (e.g., neoclassical models and institutional models of wage determination: discrimination, and unemployment) to actual situations, using them to help understand the situation

2. Improve ability to use actual situations and experiences to evaluate the adequacy of economic theories

3. Improve expository writing skills.

C. Attitude/Value/Commitment Objectives:

1. Form or deepen a concern for those for whom the labor market does not give good outcomes

2. Form or deepen a commitment to use available opportunities to help those for whom the labor market does not give good outcomes.

D. Service Objective:

1. Provide assistance to an organization which is helping people who are having difficulty in the labor market.

III. SERVICE Activities

Spend 15-20 hours over the course of 8-10 weeks working for an organization helping the unemployed or underemployed to find jobs which fit their needs. The work could vary according to the organization's needs. For example, it could involve doing a literature search on some topic of importance to the organization, office work (as long as a means to learn about the organization is provided), or providing transportation to job interviews. In some cases such service may not be a direct source of much learning but would be a form of "payment" for the time the organization spent helping the student learn about it and its clients. Each student should have the opportunity to learn about the organization and its activities and to learn about at least three people the organization is working with. It's best if the people could be met in person, but if this is impossible, confidential briefings are acceptable.

Please note: Entire text of the syllabus is published in another Campus Compact publication, Redesigning Curricula, pp. 4248.

Communicating what is learned

Requiring the student to communicate what he or she has learned has two objectives: to provide the instructor something concrete on which to base the evaluation and to provide the student an opportunity to synthesize what was learned.

Note that we are not directly assessing students' performance but are indirectly assessing it through their written work. Direct assessment is ultimately impossible - the only truly direct measure would be one that summed the impact of the service experience on the student, the client, the agency, the instructor, and the academic institution over the lifetimes of all of them. A semi-direct technique that is sometimes employed is observation of the student in the service setting. This is widely used in some situations, for example, in student teaching. However, it has some significant disadvantages which render it unsuitable for most academically based service:

• observation requires more instructor time than is usually available
• in observing the service, the instructor disturbs it (what the instructor sees is not what happens when the instructor is absent)
• the sample of observations is normally too small to be reliable
• the observation itself is filtered through the instructor's frame of reference and is highly vulnerable to instructor preconceptions of how the service should be done. Thus the focus of the evaluation can easily shift from "What is the student learning?" to "Is the student performing in the way the teacher expects?"

We see three primary means for students to communicate their service experiences: group discussion, journals, and theory-to-practice papers, although other means such as exams and such as one-on-one discussions between instructor and student are possible .3 Another means of communication - feedback from the client - is also an important component of evaluation.

Group discussion. is an essential part of the academically based service experience. It provides an opportunity for students to compare their experiences with others, to sort out which aspects of their experience are unique and which are general, and to assimilate their learning. However, group discussion does not lend itself well to evaluation as the instructor's attention is too fully given to filling the mentor role to allow for evaluation and there is little opportunity to stop and assess a contribution as it is being made. Group discussion, therefore, won't be pursued further here.

Perhaps the most widely used method for evaluating service experiences is the evaluation of student journals. Journals provide the opportunity for students to reflect on their experiences while they are still fresh in their mind and they provide for somewhat more freedom of form than does a formal paper. Theory-to-practice papers also provide opportunities for reflection. While not as immediate, they typically expect a more thorough and careful analysis. Both are valuable, serving somewhat different purposes. However, some recent research suggests that the quality of reflection occurring in theory-to-practice papers depends a great deal on the choice of question on which students are asked to write.' Thus such questions need to be constructed quite carefully.

Reflection and Measurement

Much research has been done in recent years on the reflection process.5 We will draw heavily on that research here in developing a technique for evaluating student journals and theory- to-practice papers. Writing about teacher education, Dorene Doerre Ross provides the following definition of reflection :6

At a general level, reflection is defined as a way of thinking about educational matters that involves the ability to make rational choices and to assume responsibility for those choices ... The elements of the reflective process include:

• Recognizing an educational dilemma

   

• Responding to a dilemma by recognizing both the similarities to other situations and the unique qualities of the particular situation

   

• Framing and reframing the dilemma

   

• Experimenting with the dilerrima to discover the consequences and implications of various solutions

   

• Examining the intended and unintended consequences of an implemented solution and evaluating the solution by determining whether the consequences are desirable or not

   

It seems that the process of reflection Ross describes for student teaching is generalizable to any service situation. Ross does not explicitly define "dilemma;" however, her context suggests she is thinking of a student teacher who finds himself or herself in a somewhat unfamiliar situation in which appropriate action must be selected. Thus the student must recognize his or her uncertainty, compare and contrast the situation to other more familiar situations, develop a framework within which to understand the situation, and consider alternatives and their possible consequences. This is no different from what a student in a service learning situation must do. In order to provide an effective technique of evaluation, we reformulate Ross' outline of the reflective process using a simple, three-step rubric:

• observation (recognizing a dilemma)
• analysis (responding, framing, reframing)
• synthesis (experimentation and strategy selection)

   

Note that Ross' last step (examining and evaluating) cycles back to the first two steps of our three step process. The same rubric can then be applied to subsequent experiences in the service setting with the student's actions as part of the material Subsequently to be reflected upon.

Those who have studied reflection intensively are eager to point out that reflection is more than a learning technique; rather it is a life-long habit of thought characteristic of highly skilled professionals who work in complex situations. If they are correct, the development of reflective skills is of great importance. Thus the academically based service experiences can make a valuable contribution to a student's professional life by helping students develop this skill. However, we need to acknowledge from the beginning that we can only make a small contribution to this process. Furthermore our evaluations are poor attempts to measure such a rich aspect of the students' thinking. However, some feedback is better than none. It seems advisable then that before Students are sent out to their service experiences and before they are assigned the writing of journals, they be given the above rubric, Ross' definition of reflection, and some opportunity to discuss them. As for the actual evaluation of the journals or papers, we are suggesting a three-factor measure using the three steps of the rubric as the factors. The factors should not be consolidated into a single grade as the measurement is not quantitative - the numbers used are ranks, not quantities.

Evaluation of each step needs to be done with the service goals in mind. That is, the goals provide a frame within which both the student and the instructor can reflect on the service experience. Also students should think about the goals in writing their journals or papers. Student journals are extraordinarily unique! Reading a journal while looking for evidence of how the student has reflected on his or her efforts to achieve the goals can give a great deal of insight into the student's thinking.

We recommend that the three factors be evaluated using a simple three level scoring system, followed by a verbal explanation of why the particular level was chosen. Table 2 presents the levels .7

TABLE 2 CRITERIA FOR ASSESSING LEVELS OF REFLECTION

Level One

1. Gives examples of observed behaviors or characteristics of the client or setting, but provides no insight into reasons behind the observation; observations tend to be one dimensional and conventional or unassimilated repetitions of what has been heard in class or from peers
2. Tends to focus on just one aspect of the situation
3. Uses unsupported personal beliefs as frequently as "hard" evidence
4. May acknowledge differences of perspective but does not discriminate effectively among them

Level Two

1. Observations are fairly thorough and nuanced although they tend not to be placed in a broader context
2. Provides a cogent critique from one perspective, but fails to see the broader system in which the aspect is embedded and other factors which may make change difficult
3. Uses both unsupported personal belief and evidence but is beginning to be able to differentiate between them
4. Perceives legitimate differences of viewpoint
5. Demonstrates a beginning ability to interpret evidence

Level Three

1. Views things from multiple perspectives; able to observe multiple aspects of the situation and place them in context
2. Perceives conflicting goals within and among the individuals involved in a situation and recognizes that the differences can be evaluated
3. Recognizes that actions must be situationally dependent and understands many of the factors which affect their choice
4. Makes appropriate judgments based on reasoning and evidence
5. Has a reasonable assessment of the importance of the decisions facing clients and of his or her responsibility as a part of the clients' lives.

Note that a single journal excerpt is unlikely to show all of the items listed as characterizing a particular level. Rather they serve as typical representatives of thinking at each level. Furthermore, while the levels seem clear and intuitively appealing, it is sometimes difficult to see which level best describes a student's journal.

The theoretical foundation behind this scale is a model of the development of reflective judgment developed by Kitchener and King and adapted by Ross.' Ross' adaptation is presented in Table 3. Note that we have collapsed her five stages into three levels in order to make the levels easily understandable by both the instructor and student. An instructor who wished to provide a more discriminating scale could use all five stages.

TABLE 3 STAGES IN THE DEVELOPMENT OF REFLECTIVE JUDGMENT THE INDIVIDUAL:

Stage 1:

Views the world as simple
Believes knowledge to be absolute
Views authorities as the source of all knowledge

Stage 2:

Acknowledges existence of differences of viewpoints
Believes knowledge to be relative
Sees varying positions about issues as equally right or wrong
Uses unsupported personal belief as frequently as "hard"
evidence in making decisions
Views truth as "knowable" but not yet known

Stage 3:

Perceives legitimate differences of viewpoint
Develops a beginning ability to interpret evidence
Uses unsupported personal belief and evidence in making
decisions but is beginning to be able to differentiate
between them
Believes that knowledge is uncertain in some areas

Stage 4:

Views knowledge as contextually based
Develops views that an integrated perspective can be
evaluated as more or less likely to be true
Develops initial ability to integrate evidence into a coherent point of view

Stage 5:

Exhibits all stages listed in Stage 4
Possesses ability to make objective judgments based on reasoning and evidence
Is able to modify judgments based on new evidence if necessary

For example, consider the evaluation of a journal submitted in the economics course cited earlier. Suppose a student has submitted a journal after an initial service learning experience of providing transportation for Mr. Moore, an unemployed person, to a job interview. The journal retells Mr. Moore's story, shows some empathy, discusses his educational background, and suggests that Mr. Moore needs some education to improve his marketable skills. As noted earlier, the goals and objectives for the service learning experience serve as the framework within which the student acts and reflects and the instructor evaluates. Thus the instructor turns back to Table I and notes that the students recognizes the harmful outcome of the labor market for Mr. Moore and shows concern. Hence some encouragement for achieving this goal is in order. However, there is no mention of the sponsoring organization, only one client is discussed, there is no application of economic theory, no evaluation of theories, concern is developing but there is no evidence of commitment yet, but some assistance was provided to the organization. After making these observations, the instructor compares the journal to Table 2 and identifies reflective levels. A possible evaluation might be as follows:

Observation -I- "You're off to a good start. You told Mr. Moore's story well and you have some genuine empathy for him. You need to look more closely at the organization and meet a couple more clients."

Analysis -I- "You are beginning to analyze the causes of Mr. Moore's chronic unemployment. What are some other factors besides education? How do the theories we have discussed in class help us understand his situation?"

Synthesis -I- "Your suggestion about the possibility of Mr. Moore improving his marketable skills is a reasonable one. How feasible is that for him? What obstacles might there be? What steps could he take? If you drive him to an interview again, are there practical ways you could be of help to him?"

Client feedback

Client feedback could be from the person or persons directly helped or from the agency. Ideally it would involve both, but that may not always be possible. It needs to be simple and brief so as not to consume much of the client's or agency's time. For Instance, Mr. Moore who was provided transportation in the previous example might be asked to complete a form which consists of just three questions (or even just the first two)':

In what ways was the student helpful? Were there ways the student was not helpful? How could the student have been even more helpful?

These evaluations need to be shared with students as soon as possible and students should be expected to respond to them, either in some form of discussion or in their journal. The same questions could be used by an agency staff member at one or more points in the student's service.

How can students be given opportunity to improve?

The evaluation system provides for a cycle of improvement. That is, as students observe, analyze, and synthesize, they can observe the effect of their actions and can repeat the cycle. The instructor can enhance students' improvement by giving feedback as frequently as possible and calling students' attentions to changes in their reflective process over time. Feedback must be given early enough that students have the opportunity to change. Hence, a minimum evaluation would be at midterm and at the end of the term. More often would be better. Furthermore, discussion groups after the first evaluation should provide students who wish to participate a chance to discuss their evaluations-much can be accomplished by talking through nuances in situations and by allowing students to help each other see multiple aspects of situations.

Conclusions

In summary, then, our solution to the problem of how to evaluate student performance in academically based service is:

• To use students' writing about their experience as the artifact which is actually evaluated
• To evaluate the writing by assessing the level of the reflective process expressed in the writing (the level is determined by comparison to a model developed by Kitchener and King)
• To use the goals of the course as the frame within which the student's reflective process is evaluated -that is, as providing the purpose of the service learning activity, the context within which the student places his or her experiences, and hence, the problem or "dilemma" the student is trying to solve. The goals are used this way since reflection (as conceptualized here) is inherently goal directed - it is part of the process of making choices.

   

Note that we are not primarily measuring the student's performance against the goals themselves, in the sense of ticking off items accomplished on a checklist. Rather we are assessing the level of reflection evidenced by the student in the process of attempting to accomplish those goals.

Future Directions

The model presented here has been primarily worked out as a theoretical solution to the problem of student evaluation. Much research is needed on the model. First and foremost, it needs to be tested by instructors in several different types of courses. Based on the results of such testing, improvements will need to be made.

Also, measurements of changes in student's reflective levels over the duration of an academically based service experience would be of great value. Many questions could be asked, for instance: How much change can an instructor reasonably expect? Are there any patterns to when the change occurs? Do different types of experiences lead to more change than others? To what extent are changes retained after completion of the experience? If students are given explicit instruction in the nature of the reflective process, does that yield higher reflective levels?

Dr. James Bradley is a Professor in the Department of Mathematics and Computer Science at Calvin College in Grand Rapids, Michigan.

1. For example, see chapter 1 of Alexander Astin's Assessment for Academic Excellence: The Th
2. From the syllabus for Economics 335, offered fall 1992 at Calvin College by George Monsma.

    

3. "Theory-to-Practice paper" is a term used to describe a type of writing in which students are explicitly asked either to use specified theories to explain their observations in a particular setting or to use their observations to critique a theory.

    

4. Ross, Dorene Doerre (1989), First Steps in Developing a Reflective Approach, Journal of Teacher Education, 40, 22-30.

    

5. See, for example, two books by D.A. Schon, The Reflective Practitioner, Basic Books, NY, 1983 and Educating the Reflective Practitioner, Jossey-Bass, San Francisco, 1987.

    

6. Ross, op. cit

    

7. This is a modified version of a chart from the paper by Ross cited above.

    

8. Kitchener, K.S. (1977) Intellectual development in late adolescents and young adults: reflective judgment and verbal reasoning, unpublished doctoral dissertation, University of Minnesota, Minneapolis. Also, King, P. (1977) The development of reflective judgment and formal operations in adolescents and young adults, unpublished doctoral dissertation, University of Minnesota, Minneapolis.

    

9. It does not seem to us that a quantitative evaluation is suitable in this setting. Quantitative measures require the establishment of an underlying theory of the entity being measured which is sufficiently well-developed to allow meaningful comparisons between entities and to justify the numerical assignment. (See Bradley & Schafer, forthcoming.) We don't even have such a theory and even if we did, it doesn't seem to us reasonable to ask that clients and agencies learn enough about it to use a quantitative measure.

Campus Compact: The Project for Public and Community Service is a coalition of over 500 college and university presidents who believe that institutions of higher education hold a primary responsibility to foster students' sense of civic responsibility and to contribute to the welfare of their communities. The Compact works on a national level to cultivate discourse and support for issues of public service; develop resources materials, grant programs, workshops and institutes; and support a network of state and specialized offices. to provide targeted programs for campuses. Campus Compact, Box 1975, Brown University, Providence, RI 02912-1975, 401-863-1119.

Campus Compact is a project of the Education Commission of the States, a nonprofit, nationwide interstate compact formed in 1965. The primary purpose of the commission is to help governors, state legislators, state education officials and others develop policies to improve the quality of education at all levels. It is the policy of the Education Commission of the States to take affirmative action to prevent discrimination in its policies, programs and employment practices. Education Commission of the States, 707 17th Street, Suite 2700, Denver, CO 80202-3427, 303-299-3600.

The Project on Integrating Service with Academic Study (SAS) provides training, advice on strategy and technical assistance to colleges and universities that are working to build community service into their teaching and research. While it provides some resources for individual faculty (such as a database of courses and syllabi), the project works primarily with campuses attempting to institutionalize service-based teaching and research. Campuses are typically motivated to integrate service into teaching and research by some mixture of concerns regarding citizenship, diversity or building community. Much of the project's actual work focuses on integrating service with the curriculum, because we recognize that the curriculum is the core institutional structure around which most campuses are organized.

Project on Integrating Service with Academic Study Staff

Sandra Enos, Project Director

Marie Troppe, Project Associate

Kate Jackson, Project Intern

Aileen Keenan, Project Intern

1995 ECS/Campus Compact