What?
What do you need to know for the Ministry of Training, Colleges and Universities (MTCU) and what does your agency itself want to know?
For MTCU
We know from the Ontario Transfer Payment Agreement that you have to complete the LBS Exit and Follow-Up Form. The LBS Exit and Follow-Up Form also has sections to complete at exit and at 3, 6 and 12 months after the learner exits your program.
What questions does your agency need answered?
Designing your follow-up is about asking effective questions, then coming up with ways to get useful answers. Take time to determine which questions are the “right” ones. Consider the “who”, “what”, “where”, “why”, “when” and “how” and develop appropriate questions. There is no point asking questions if you do not intend to react to or use the responses. Inappropriate or unrealistic questions will produce ineffective and useless answers. You should ask questions that will provide positive quantitative or qualitative results to help your program improve, grow or develop.
In order to get the best information for comparison and tallying, you should design the answers to the questions on a weighted scale (as in a scale of 1 to 5 with 1 indicating complete dissatisfaction and 5 indicating complete satisfaction)
Community Literacy of Ontario’s (CLO) and Literacy Link South Central’s (LLSC) Developing a Culture of Evaluation website has about everything you could want to know about evaluation. In its Collecting Data module there is a section devoted to Asking the Right Question.
Community Literacy of Ontario’s (CLO) SmartSteps to Organizational Excellence has samples of both learner and community partner surveys in its Program Evaluation section. These may assist you in developing a suitable survey for your agency needs. CLO’s Capacity Plus: Organizational Capacity Resource Guide for Ontario’s Community Literacy Agencies has sample customer satisfaction questions for learners, volunteers and other community stakeholders in its Customer Service Management chapter. You can download these valuable resources from CLO’s Publications webpage.
Asking the Questions that Enable You to Act (YouTube) with Dr. Andrew Taylor (United Way of Peel and Region of Peel) challenges you to consider creating “powerful questions.” He encourages you to look past your regular satisfaction surveys and obvious questions to ask braver questions. He maintains that “The heart (of evaluation) isn’t in the asking. It’s about putting yourself out there to learn…”
Where?
Where will you record the results?
Once you get the facts and feedback from your Follow-Up Plan, where will you record the results?
The results of the Ministry required information on the LBS Exit and Follow-Up Form will be added to EOIS-CaMS. Also, every closed learner paper file must include one of these forms. (For convenience, you can download the LBS Exit and Follow-Up Form from the Employment Ontario Partners’ Gateway (EOPG) Tools link, Forms section as a Microsoft Word document and print it.)
It is up to you to decide where you will keep the follow-up results that are specific to your agency. Sheets kept in learner files are not so easy to refer to when it comes time to complete your program’s evaluation. Many programs use a spreadsheet or database to compile all or some of the results. Others keep all the results in one file, while still others use tracking forms for recording information to which they will respond later.
Again, we recommend referring to CLO and LLSC’s Developing a Culture of Evaluation website, especially the module on Analyzing Data that discusses how to prepare and organize the different types of data for analysis. Also, you can view one of their Webinar #3 – Collecting Data: Beyond Survey Monkey in the site’s Webinars Section.
Imagine Canada’s Project Evaluation Guide for Nonprofit Organizations has a section on Managing Data Collection that you may find helpful.
Where will you use the results?
Where you use the results of the follow-up facts and feedback depends on the type of information you have collected. Some results will be positive, while others may require changes or call for improvement. Imagine Canada’s Project Evaluation Guide for Nonprofit Organizations suggests ways you can use the results of project evaluation. Since a good part of the Follow-Up service is project evaluation, we have selected ways from Imagine Canada’s list that are transferable to LBS service provider options:
- identify ways to improve or shift your project activities
- facilitate changes in the project plan
- prepare project reports (e.g., mid-term reports, final reports)
- inform internal and external stakeholders about the project
- plan for the sustainability of the project
- learn more about the environment in which the project is being or has been carried out
- learn more about the target population of the project
- present the worth and value of the project to stakeholders and the public (e.g., 98% of learners who use our programs are extremely satisfied with our programming)
- compare projects to plan for their futures
- make evidence-based organizational decisions
- demonstrate your organization’s ability to perform evaluations
- demonstrate your organization’s accountability concerns for implementing its plans, pursuing its goals and measuring its outcomes
Information generated in MTCU’s Report 60D, LBS – All Data – Outcomes/Follow-ups, can be used for outreach purposes. All of the exit and follow-up information can be used to create an infographic. There are many samples available on the Internet. Try googling “student satisfaction infographic” Infographics can easily be displayed in a rack card size / postcard size.
In the Sample Forms section of this module, we have included Connections Adult Learning Centres’ Monthly Client Survey Action Plan Form to track action required due to survey responses.
When?
When will you do the follow-up? An evaluation plan is not just a single event. It should involve a number of inputs that take place over an extended length of time. Your plan may be for three months, a year or be part of a three or five-year strategic plan. When you do the various activities in your plan will depend on why you are doing them, who is doing them, with whom and what they are doing. You may decide to survey community partners once a year, carry out random interviews with active learners at a different time in the year, and complete exit and post-exit interviews with learners monthly.
To meet MTCU requirements, you must complete a learner follow-up interview at exit and at 3, 6 and 12 months after learners exit your program. Some service providers attend to these interviews all at once on a monthly or weekly basis, as they become due, while others start the process on the day, 3, 6 and 12 months after the learners exit.
So, how do programs remember when it is time for these follow-up interviews? While some set up reminders in electronic calendar software or track them on spreadsheets, most programs use the reports and reminder systems generated by the Employment Ontario Information System-Case Management System (e.g., print the “Pending Reviews”page from EIOS-CaMS and use that as a tangible reminder of who needs to be contacted). Some programs also find it helpful to pick a specific date in the month and complete the follow-ups around the 15th of the month. .
