The year is almost over but it’s not too late to create the most effective evaluation plan your program has ever had. In fact, now is the perfect time to use evaluation to reflect back on the past year and plan for the coming year. All you need is one program metric and a willing team.
Let’s do it.
Step 1: Select one metric
You don’t need to already have calculated it, but you should be able to calculate it with a fair level of accuracy. Fair — not perfect. This metric should represent an outcome that is very important in your program. It should be important enough that if it’s not heading in the right direction, everyone will be concerned.
To be effective, the metric must start with a number or a percent. Trust me, it has to. And it will have a time period associated with it. For the purpose of this exercise, we’ll assume the time period is a calendar year.
It should also have communication power outside of your organization. When you tell community members about your metric, they should be able to understand why the metric is important to your organization’s mission and how it demonstrates your impact.
Here are some examples:
- % of participants who report that this training helped them do their job better
- % of 3rd grade students who pass the state standardized test in reading
- % of shelter guests who moved into permanent housing within one year of entering shelter
Step 2: Articulate why it’s important
This is the most important step. You’ll want to articulate why it’s important and you’ll want to make sure that everyone who engages with this metric agrees on why it’s important. One metric can mean very different things. You won’t be able to have a productive conversation about how to interpret or improve the metric if you’re not clear on how you’re using it. Let’s use % of 3rd grade students who pass the state standardized test in reading as an example.
Some teachers, administrators, and parents are going to be worried that that really tests how good students are at taking standardized tests, and not how good they are at reading. We’ll all have to agree that we’re not measuring enjoyment of reading, just ability to demonstrate reading skill for a test. Ideally, we can all agree that this measure, while it doesn’t measure some other important things like joy of learning, is a reasonable proxy for basic literacy. More importantly, we’ll agree that if students are not passing the standardized test, they are probably not enjoying or engaging with reading and that the problem will get worse, not better over time.
Step 3: Indicate polarity
Simply, polarity means should this metric increase, decrease, or aim to be in a specific range? For the reading example, higher is better and 100% is best. Some metrics might have an ideal range, such as # of students enrolled where the ideal value is enough to keep our program financially healthy and few enough that we can serve them well with our current staff.
This is another important exercise to do with your team. You might be surprised that different people have different interpretations. For example, one member might think that enrolling more students is better, another might be concerned about being able to place them after graduation. You might realize here that you need to articulate the measure differently and go back and clarify.
Step 4: Ask “is this good enough?”
If your measure is already good enough, you’re done! Examples would be that 100% of the students passed the test or 100% of the shelter guests moved into permanent housing. If so, make sure that’s included in your annual report and make a plan to monitor this going forward.
But, mostly likely, you chose a metric you’re concerned about, so you’re going to decide it’s not good enough. Don’t panic! If you don’t address it, you can’t improve it. So take a deep breath and let’s start planning.
Step 5: Ask why
Here’s where we’re going to really analyze the problem. Not just describe it again, but try to figure out the root cause of the problem. For example, if there are children who are struggling to read at grade level, do they all have the same struggles or are there different groups who could be helped with different interventions? What are the barriers that you could help remove for them? What are the strengths and assets that they bring? A useful exercise for getting to the root cause of the problem is 5 Why’s (simply ask yourselves why a problem is happening, when you generate an answer, ask why that happens, repeat 5 times.) Here, let’s be careful to avoid blaming the participants. You can’t make them better, you can only make your program better, so let’s focus on the participants’ strengths and assets.
Step 6: Gather best practices
Your program works on hard problems. But the good news, is you’re probably not the only one that has worked on or is working on these issues. So, let’s do some research. Who else works on this? Can we call them? Are there other programs that have had success tackling a similar problem? Reach out to them. They will probably be thrilled to tell you what has worked for them. Are there other organizations in your community that can help? After-school programs? Tutoring programs? The housing authority in your city? What creative partnerships can you create? Have your team work on this for a few days, doing some reading, looking at other programs’ websites and evaluation reports, scheduling calls with potential partners, and have them come back with some proposals. They don’t have to be realistic or feasible at this point. Now’s the time to think big.
Step 7: Make a plan
Now let’s reconvene the team and pick one action to implement. First, everyone will explain their ideas and insights. Next, we’ll prioritize. One way to do this is to sort the ideas based on impact and feasibility. I like to use poster paper. Divide a sheet into 4 quadrants with one side labeled “Feasibility” and the other labeled “Impact”. Mark one side high and the other low so that you have 4 quadrants:
- High feasibility/high impact
- High feasibility/low impact
- Low feasibility/high impact
- Low feasibility/low impact
From the high feasibility/high impact quadrant, agree on one – max two – strategies to implement next year. Assign specific action steps and target dates. Plan specific times to update each other and hold each other accountable to the plan. And emphasize a culture of collaboration and collective problem-solving as you implement the new actions.
For items that were marked low feasibility/high impact, keep working. What could you do that would make those options more feasible? Consider cultivating them for longer term planning.
Step 8: Monitor
Commit now to revisiting this metric at the end of next year, or sooner if possible (for example if your measure is one you can look at more frequently than annually, then commit to doing so). Use steps 4 through 7 to guide your conversation every time you look at the metric. Make sure to celebrate when it starts to improve. Congratulations! You’ve made an effective and useful evaluation plan, engaged your team, and improved your program! Make sure to document and share your success so that you can be a model for other programs or organizations facing similar issues.