The provision of good literacy practice is the goal of everyone involved in learner-centred community based adult literacy programs. It is well accepted that instruction which recognizes the complex nature of literacy and focuses on the strategies and processes involved in reading, writing and learning is a fundamental ingredient in good literacy practice. What is not so well recognized is the role that evaluation should play in the provision of good literacy practice.

The following story illustrates some of the issues associated with evaluation in community based, adult literacy programs.

A student in a community-based literacy program was discussing her learning in a small group.

She said, " I don't always feel like I'm doing better, so it's hard for me to keep coming to the program each day."

One of the tutors in the program said reassuringly, "But you are making progress, I can tell."

"You may think I'm making progress, but I need to feel it; and I don't always. So I have to make myself come even though I'm not sure things are improving" the student answered.

It should be apparent from the above story that:

• No student should ever be left feeling frustrated and uncertain about their progress.

  All instructors need to have a meaningful and useful way to evaluate, document and communicate information about learning progress to their students.

• Although the story only reveals the difficulties that the student and instructor have with evaluation, it also implies the difficulties that other stakeholders must have in feeling confident that the program is meeting its objectives and serving its clients well.

  If both instructor and student are so unclear about the progress that is being made, how can program managers, community supporters and funders of the program be convinced of the effectiveness of provision?

Evaluation tools and procedures must provide meaningful and useful data that can be communicated in ways that satisfy the needs of everyone involved in supporting, delivering and receiving literacy provision. Such tools are as important to good literacy practice as is good instruction.

In fact it can be argued that good evaluation is more important. Without good assessment tools and procedures, there can be no real foundation on which to plan the appropriate instruction required for good practice. In any case, what is certain is that without good evaluation and good instruction there can not be good practice.

Careful consideration of the following 5 questions should help clarify some of the factors that need to be considered when implementing an effective evaluation plan for learner-centred, community-based adult literacy programs. These questions are:

1. Who needs evaluation information?
2. Why do they need it?
3. What kind of evaluation information does everyone want?
4. How can they best get the information they need?
5. When should evaluation take place?

1. WHO NEEDS EVALUATION INFORMATION?

The learner, the instructor, the program managers, the community supporters and the program funders all need evaluation information.

Colleges, other training institutions or programs, potential employers, government agencies and the media are some examples of others who may want information about the program and/or the learners' achievements.

2. WHY DO THESE INDIVIDUALS OR GROUPS NEED EVALUATION INFORMATION?

Learners want to know whether or not they are achieving their goals. They need to be able to celebrate their successes and develop confidence in themselves as learners. They have to know what they have already achieved so they can make decisions about setting new goals and planning the next stages of their learning.

Instructors need to know where instruction should start and what direction it should take. They need to know whether or not their planning and instruction has been effective. This information will enable them to make decisions about adjusting their instructional plans, setting new goals and/or selecting different materials or approaches. Finally, they also need to be able to celebrate their good practice and develop confidence in themselves as diagnostic and prescriptive teachers.

Program managers, supporters and funders need to know if the program is providing good literacy provision. They will want to know if learners are achieving their goals. They will also need to know if the program is fulfilling its mandate and meeting its objectives.

This kind of information will enable them to make decisions about program delivery and management, as well as about whether or not they should continue to support the program. They also need to be able to celebrate the positive results of their support and be reassured that their time, effort and financial contributions have made a difference.

Colleges, other "outside" institutions, and employers will want to know if students meet their requirements. They will need to know that the evaluation data provided by the program is reliable. Only then can they be confident of a smooth transition from literacy program to their institution or workplace.

Even though stakeholders in the literacy program have their individual reasons and purposes for wanting evaluation data, the following reasons are common to all of them:

• Everyone wants to have the opportunity to celebrate success, to experience the joy of a job well done. Having concrete evidence of what has been accomplished allows everyone to grow in confidence and inspires them to set higher and higher goals.

• Everyone wants to make plans for the future. Having information about what was achieved in the past, what worked well and what didn't work well provides a solid foundation on which to build plans for the future.

• Everyone wants to know whether or not the program will continue to exist. Having information about what goals were met and what changes if any need to be implemented enables everyone to feel more confident that support for the program can be sustained.

In summary, it can be said that for everyone the goal of evaluation is to allow for the examination of past performance and the initiation of future action.

3. WHAT KIND OF EVALUATION INFORMATION DOES EVERYONE WANT?

A study of current evaluation research suggests that the only way to ensure that the above goals of evaluation are achieved is to choose evaluation tools and procedures that will provide meaningful and useful data. To clarify what is meant by meaningful and useful data, it is helpful to define these words as they relate to evaluation and then look at specific examples that illustrate the collection of this kind of data.

Meaningful evaluation data is information that relates directly to one's objectives.

The following examples illustrate how evaluation data can relate directly to objectives:

If a learner's objective was to improve his comprehension of text by using self monitoring strategies, having the learner tape himself thinking aloud while reading would be an effective evaluation procedure. The tape would provide meaningful data because it would reveal exactly what he is and is not doing in relation to his goal.

If a teacher's instructional objective was to teach writing using a process approach, ongoing evaluation of students' writing at various stages in the writing process would provide her with meaningful data. The samples of work in progress would provide concrete evidence of the strategies and techniques students were using to arrive at a finished piece of writing. Such evidence would be meaningful to the teacher because it would explicitly reflect where instruction had been effective for students and where it had not been effective.

Useful evaluation data is information that provides a springboard for action.

Some researchers suggest that if no action will result from the data collected through a particular evaluation tool or procedure then the evaluation should not be carried out.

Celebrating progress and good practice, setting new learning goals, rethinking instructional objectives and strategies, seeking out specific materials, inviting an expert into the classroom, renewing a commitment to support the program are all examples of actions that could follow from useful evaluation information.

To summarize, meaningful and useful evaluation information is data that relates specifically to the objectives that have been set. It is data that clearly shows what has been accomplished and what still needs to be done. An evaluation process that provides data that evaluates the past and directs the future would be a key element of good literacy practice.

4. HOW CAN MEANINGFUL AND USEFUL EVALUATION DATA BE OBTAINED?

In order to collect evaluation data that tells both where you are and where you need to go in relation to your objectives, careful consideration must be given to the selection of evaluation tools and procedures.

Only those evaluation tools and procedures that are aligned with both the broad goals and the specific learning, instructional and management objectives of the program will provide data that can reveal what progress has been made in achieving these goals and objectives and what future actions need to be taken.

It is widely accepted that in learner-centred, community-based adult literacy programs, good literacy practice occurs when the goals, objectives and delivery of literacy provision mesh with the philosophy that defines this kind of programming.

If evaluation tools and procedures are selected on the basis of their alignment with instructional practices and the program goals and objectives, then they will be congruent with the beliefs about the nature of literacy and literacy learning that make up this philosophy. Evaluation will then be woven together with philosophy, objectives and instruction and play an integral part in good literacy provision.

Good literacy provision depends on the alignment of instruction and evaluation with philosophy. To select appropriate evaluational and instructional tools and procedures, we need to have a clear understanding of the basic beliefs that define learner-centred, community-based, adult literacy programming.

These beliefs and the objectives, methodologies, and evaluation procedures that follow from them can best be understood by considering each of the following three questions:

1. What is it important for adult learners to know and be able to do?
2. What is valued in good literacy instruction?
3. How can literacy learning best be demonstrated?

1. WHAT IS IT IMPORTANT FOR ADULT LITERACY LEARNERS TO KNOW AND BE ABLE TO DO?

A. According to the current philosophy of learner-centred, community-based literacy programs learners need to know how to be active learners. As active learners they need to be able to set their own learning goals, participate in determining their own curriculum, and monitor their own progress.

As active learners, they also need to be able to take risks, ask questions, solve problems, engage in critical thinking, interact with authentic texts, and with "real" people. They need to discuss, debate, listen and reflect.

B. Learners need to know strategies, processes and skills for reading, writing and learning. They need to be able to use these independently and be able to transfer them so that they can handle a variety of genuine tasks for a variety of purposes.

C. Furthermore learners need to know how the processes of reading and writing work. Then they will be able to comprehend and create text. Learners need to monitor their comprehension and employ fix-up strategies when necessary.

D. Finally adult literacy learners need to know how it feels to have a positive attitude towards reading and writing. They need to be comfortable using reading and writing for their own pleasure and to serve their own purposes.

An example of a specific instructional objective following from these beliefs would be the decision to teach the pre-reading strategies of skimming, scanning, activating previous knowledge and making predictions. An evaluation procedure that is congruent with this objective would be one that allows the learner the time and opportunity to use these strategies with a variety of texts and in a variety of situations. It would also be one that provides the learner with a way of documenting and reporting on his/her use of the strategies.

Examples of evaluation tools that are congruent with these beliefs and objectives are:

• Interviews with the student.
• Observations of his/her interacting with various texts.
• Records of predictions made.
• Journals that show how previous experience was related to the text.

When evaluation tools are congruent with beliefs and objectives they provide data that is meaningful and useful to both the instructor and the learner.

In contrast, a timed reading assignment that required students to demonstrate their understanding of the text by answering multiple choice questions on the content, would not be congruent with this instructor's objective or with his/her beliefs about what learners need to know and be able to do.

Such an evaluation tool would not provide any explicit information about the learner's use of prereading strategies. Not only would it not provide any data related to the objective but it would also inhibit the use of the very strategies that the instructor was trying to promote.

Misaligned evaluation tools not only fail to provide useful data, but they can also have a very negative impact on learners. They can, for example, result in students feeling confused and frustrated. They can also cause learners to lose confidence in themselves as learners, in the instructor as a reliable and helpful teacher and even in the program as a worthwhile place to spend their time and energy.

All results that are inconsistent with beliefs that the program holds about the need for learners to know how it feels to have a positive attitude to learning and to themselves as learners.

What is valued in literacy instruction?

According to the current philosophy of learner-centred, community-based adult literacy programming, valued literacy instruction is instruction that is participatory and interactive.

Such instruction is directed by the individual learner's goals, shaped by his or her learning style, needs and interests and driven by his or her learning pace. It builds on all facets of the learner; his or her strengths and weaknesses, previous knowledge and experience, and motivation.

Valued literacy instruction is instruction that engages the learner in a broad range of authentic tasks presented in real situations and involving "real" people.

Valued instruction is instruction that recognizes the multi-dimensional nature of literacy learning and emphasizes the processes and metacognitive aspects involved in reading and writing rather than on the mere production of finished pieces.

Above all, valued literacy instruction is instruction that empowers learners by fostering independence, self-esteem and positive attitudes towards learning.

Looking at an evaluation tool that is not congruent with the objectives and methodologies that will follow from these beliefs will best serve to highlight what appropriate tools and procedures would be like.

Formal standardized tests would not be an appropriate choice for learners and instructors who are setting goals and engaging in the kind of practices that would be aligned with these beliefs about valuable literacy instruction.

Why are these inappropriate evaluation tools? They are inappropriate because:

• They ignore the functional nature of literacy by removing the literacy tasks from real purposes and uses.

• They ignore the dynamic, multi-dimensional nature of literacy by testing skills in isolation.

• They focus on the products of knowledge and not on the processes and strategies of learning.

• They inhibit interaction and the use of strategies and processes the involve consultation, reflection, and revision; all valued goals of literacy instruction.

• They fail to take into account the central role the learner plays in good I literacy instruction and practice. By the very nature of their standardization, they cannot accommodate individual starting points, individual learning styles, and individual learning goals. All of these factors are fundamental to the beliefs about good literacy instruction in a learner-centred adult literacy program.

• They often intimidate adult learners and remind them of former failures in school thus counteracting the efforts of literacy providers to help their adult learners develop positive self-esteem and confidence in their ability to achieve their learning goals.

Formal standardized tests are so misaligned with the instructional objectives and methodologies that originate in the philosophy of learner-centred, community-based, adult literacy provision that they disrupt and counteract good literacy provision and result not in meaningful and useful assessment information but rather in lost instructional time, confused students, and frustrated teachers.

In contrast to formal standardized tests, appropriate evaluation tools and procedures would:

• be collaborative in nature.

• compliment the participatory, interactive aspects of valued literacy instruction.

• engage the learner in assessing and evaluating his/her own strengths, weaknesses, needs and progress.

Interviews, conferences, journals, observations, and portfolios are all examples of appropriate evaluation tools.

These tools would provide individualized information that reflects the dynamic, functional, multi-dimensional nature of literacy. They would also promote the instructional objective of learner empowerment.

As learners identify and discuss the knowledge and skills they have acquired, as they reflect on what they have achieved and what they need to do next, as they analyze and evaluate their work, they will be assuming ownership of their learning and thus be empowered.

How can literacy learning best be demonstrated?

Since the philosophy of learner-centred, community-based, adult literacy programming accepts the belief that literacy is functional, then the skills that have been learned and the conceptual understandings that have been developed need to be demonstrated in authentic situations for real purposes.

Observations of learners reading writing, speaking and listening while engaged in genuine tasks for real purposes would be an appropriate evaluation procedure.

Since this philosophy also accepts the belief that literacy is multi-dimensional, then an appropriate evaluation process must provide opportunities for the learner to demonstrate a broad range of performances and behaviours.

Using a variety of evaluation tools and procedures would allow for the collection of this kind of data.

Furthermore, since this philosophy accepts that literacy learning is a process of growth and change that occurs over time, then an appropriate evaluation procedure must allow the learner to demonstrate that growth and change has occurred over the time that he/she has been in the program. Using portfolios is an example of this kind of evaluation procedure.

In summary, meaningful and useful evaluation information can be obtained if evaluation tools and procedures are:

• aligned with the beliefs that determine the goals and objectives of learner-centred adult literacy provision.

• grounded in authentic tasks.

• designed to collect information about a broad range of performances and behaviours.

• designed to allow for the collection of information over a length of time.

If the evaluation process meets these criteria then it will enhance instruction and provide information that can evaluate progress as well as initiate future actions.

Knowing what evaluation tools and procedures are appropriate to use is one of the indicators of good literacy practice. Knowing when to use them is another.

5. WHEN SHOULD EVALUATION TAKE PLACE?

The practitioners in learner-centred, community-based, adult, literacy programs generally recognize the need to have procedures in place to gather initial and final assessment information. However, the need to have a plan to collect evaluation data on a regular, ongoing basis is not as widely accepted. Yet regular, ongoing evaluation is critical to good literacy practice. Why? Because regular, ongoing evaluation:

• is most congruent with the beliefs that constitute the philosophy of learner-centred, community-based, adult literacy programs.

• allows for the assessment of the process of learning rather than simply the evaluation of a particular skill or product of learning.

• facilitates the blending of evaluation with instruction so that evaluation can be indistinguishable from instruction.

• allows evaluation to grow out of authentic reading, writing and learning tasks and makes evaluation congruent with the belief that literacy is functional and multi-dimensional.

• provides opportunities for reflection and timely adjustments in objectives, methodologies, curriculum, program scheduling, etc.

• allows for the celebration of learning all along the way. This enhances students' self-esteem, motivation and commitment to learning. All of these outcomes are valued in the philosophy of learner-centred, community-based adult literacy programs.

CONCLUSION:

Evaluation can be a key element in good literacy provision if practitioners implement an evaluation plan that:

• includes tools and procedures that are congruent with the philosophy of learner-centred, community-based, adult literacy programs.

• provides data that is directly related to the desired outcomes.

• provides data that can direct future actions.

• provides measures of a wide variety of indicators of learning.

• allows for the collection of data over time and across many different situations.

• enhances instruction.

• enables learners to assume ownership for their own learning.