Evaluation

What Does It All Mean?

Evaluation is Step 2 of the strategic planning process. In this step, you will answer the question: what does the information mean?

Essentially, this step helps you make conclusions based on the information you gathered during the assessment phase. Although you won’t have begun an evaluation of the assessment data yet, you will probably have a good handle on the particular strengths and weaknesses, opportunities and threats facing your agency. During the evaluation you will bring your professional judgment to bear on the assessment data and identify:

1. The degree to which the various internal strengths and weaknesses affect the success of your agency.
2. The significance of the external opportunities and threats facing your agency and its continued success in the future

Evaluating your data will support the decision-making process that comes next by identifying those areas that require urgent attention and those areas that can be left as is. Evaluation will also identify opportunities that are just too good to pass up, and it will highlight red flags which indicate that now is not a good time to move forward.

For example, the assessment results might reveal that the agency has no marketing strategy and, in fact, does very little in the way of outreach in the community. If the agency has been struggling to meet contact hours, then you might decide this finding has a serious impact on success. You might determine to make marketing a priority for the near future. If, on the other hand, there is a constant stream of inquiries and a waiting list for the program, then you may not consider the lack of a marketing strategy as a significant finding.

You may, however, want to instead look deeper into this situation for other significant issues. For example: Why is there a waiting list? Is it an unusual year? Has there always been a lack of learners but suddenly that has changed? Has there been a change in the employment rates? Is a new business coming to town that people are training for?

Next steps in evaluation

In order to evaluate the assessment data, you will need two things:

• A document that summarizes all the data from the assessment phase
• A scale or tool by which you can measure or assign a value to the findings (i.e. rank the findings according to each item’s status or need for attention to ensure ongoing success.) This tool is included in the next section.

The evaluation activity itself involves:

• Reviewing the items from the assessment phase
• Making a judgment call on the need for attention for each item
• Marking each item according to its status as thriving, safe, stable, at risk, or in crisis

Although this could be a collaborative activity, probably the most time-and-energy efficient way is to carry out an evaluation activity yourself first and then present the results to key stakeholders for discussion, revision and final validation. It is crucial to get the support and general agreement of stakeholders at this stage before moving on to decision-making.

How you go about evaluation?

In the assessment phase you have collected and summarized information on your agency’s mission, values and mandates as well as on its internal and external environment. This gives you critical information about your agency and its guiding purposes and principles, current strengths and weaknesses, and opportunities and threats.

The next step is to create a chart that shows:

• The items that you have examined
• The tools and methods you used to gather information on this item
• The significant findings for each item
• The ranking of the relative importance of each item

Fortunately, all this information should already be written up in your notes from the assessment phase so cutting and pasting should be quite simple and quick! For a clear example, just see the sample chart below.

Sample evaluation chart

Using a template that worked well for Community Literacy of Ontario, here is a sample of the evaluation chart filled out:

Creating your own evaluation chart

1. Cut and paste the collected data into an evaluation report as follows:
   • Column one: Name the item you assessed.
   • Column two: Indicate the assessment methods and tools you used. This could be one-to-one interviews, focus group discussion, larger group activities, surveys, questionnaires, guided questions, etc.
   • Column three: Summarize your findings from the assessment and put them here. Try to capture all the important information but keep it as simple as possible using headings and bullets.
2. For each item you identified in your evaluation chart, ask yourself the following questions:
   • What does the research show about this item?
   • Are there some strategic issues here? (i.e. fundamental challenges that affect our agency’s mandates, mission, service level, clients, costs, financing, organization, or management)
   • How should I rate this item with regard to its effect on our agency’s success in the short- and long-term?
3. For each item, establish a ranking of thriving, safe, stable, at risk or in crisis and paste it into column four.

The Rating Scale

A rating scale has been developed that allows you to identify different levels of need for action and response. The colour code was added to allow you to pick out particular information at a glance.

EVALUATION

How to use the colour code

With markers in hand (or the computer’s highlighting tool turned on) follow the steps listed above and then mark/colour each item according to the code. Remember, for each assessed item you want to ask, “How important is this, and how urgently does it require attention?” At the end, you want to be able to look back at everything and have all the really important items that demand attention jump out at you.

Sharing your evaluation results with key stakeholders

Once you have completed the evaluation chart for all items identified in the assessment phase we strongly recommend that you set aside some time to talk over your evaluation results with key stakeholders. Since they are already well-invested in the process, they will likely value this opportunity for input. It will also provide a way for you to validate your interpretation of the data or, if necessary, give you an opportunity to reassess a judgment call.

Once you have identified and agreed upon all assessment findings as either thriving, safe, stable, at risk or in crisis, you are ready to move on to the next step in the process: making actual decisions for the future.

Questions and Activities for Reflection

1. What items do you think will be identified as “thriving” in your literacy agency and why?
2. What items do you think will be identified as “at risk” or “in crisis” in your agency and why?
3. In your agency, what would be the most effective way to share your initial evaluation results with key stakeholders?
4. How could this evaluation process help you make decisions about the direction your agency is moving in?