Evaluation FAQs

Managing your evaluation plans internally can be complicated. Trying to design the evaluation plan that best represents your work without disrupting the work is crucial and challenging. There are a few questions that clients have asked me over and over. Here, I address some of those frequently asked questions.

Why is my funder asking for a logic model?
It’s definitely not so that you can spend hours fitting lots of tiny text into boxes, just to stash your logic model away and never look at it again. The funder is asking for a logic model because it helps them understand what your program is meant to do. It’s a way of communicating, using a somewhat standardized template, what the program is and what result it is going to have. It helps everyone understand how they’ll know if it’s working or if it needs adjustments.

Are there any other benefits to having a logic model?
Yes! Think of the logic model as a guiding document for your program. Refer to it when you’re making decisions: will this change help us achieve the program’s intended outcome? Think of it when you’re articulating your program’s success or when you need to make a change. Having the entire organization understand the logic model helps everyone stay on the same team, prevents turf battles, and buffers against random ideas. Most importantly, a strong logic model helps you understand what not to do. There are a lot of great ideas out there, but just a handful that your organization is really the best to execute. Your logic model helps you understand what’s outside of your organization’s scope.

How do I create a logic model for an existing program?
What makes it hard to build a logic model for a program that is already running is the desire to incorporate all of the work that you’re doing in that logic model. The logic model should document what your participants do in the program and how they are better off. It’s not the place to document all of the work that you and your staff do to create a successful program.

I recommend starting at the outsides of the logic model template. What is the problem that your program is meant to solve? How will your participants be better off after they have participated in your program? Next fill in the activities. What do the participants do in the program that will lead to those outcomes?

And that’s the basic logic model.

Like good writing, it will take a lot of editing to communicate your program’s design clearly and succinctly. Think of that editing process as an opportunity to remove anything from the program design that is not really serving your purpose and to align resources towards your program’s best possible outcomes.

I am already distributing a survey in my program. Can I use this?

Great start! The answer is yes! Here’s how to adapt it:

Document which element on your logic model each question measures. For example, a question that asks “After this training, how did your confidence with applying the material change?” would map to the outcome “Participants are confident in applying the material?”

Next, if there are questions that map to your logic model and are phrased to collect open-ended responses, update them to collect quantitative data. For example, update “Tell us about how your confidence with the material changed.” to “Please rate your change in confidence with the material: 1 Much less confident, 2. A little less confidence, 3. A little more confident, 4. Much more confident.” This will make it easier to compare values from one time period or cohort to another and will allow you to generate statements like “85% of our participants reported that they were more confident with the material after the training.”

Third, consider dropping questions that do not map to the logic model and aren’t generating data that you use. A shorter survey will get a better completion rate than a long one and is the best use of your and your participants time.

That’s it! Your existing survey will now be a powerful performance measurement tool!

We collect a lot of rich qualitative data. Can we use this?
Yes! Quantitative data (numbers) is really quick and efficient for telling us what is happening – are scores improving over time? Do the participants enjoy the program?” But qualitative data (long-form responses to surveys or interviews) tells us why and how. Here are a couple of ways to use that that most effectively.

1) Before you design a survey. Reading the qualitative data you’ve already collected will help you design a better quantitative data collection tool. For example, it might surface topics that you should make sure to ask about.

2) After you’ve conducted a quantitative survey. Consider conducting some roundtables with people who are representative of your survey population to understand the survey data. Prompts like “this is what we are seeing in the data. Why do you think this is happening?” Can provide important insights into how the program works or could improve.

3) To give more context and texture to survey responses. Open-ended questions like, “What was the best part of this program for you?” surface fantastic illustrations that can be inspiring to staff and donors to read. They can provide insight into what your program is doing well and what to pay attention to. For example, one project I’ve been working on has a survey that measures the number of unmet household needs. The quantitative data tells me that the group who are getting the intervention have slightly fewer unmet household needs. The qualitative data from interviews tells me that because of the intervention, parents are considerably less stressed because the don’t need to make hard choices between buying groceries and paying their utility bills and that prior to the program, they often spent time buying their groceries a little bit a time whereas now they can buy all their groceries in one trip. This kind of texture and detail really helps us understand how the program is making an impact for the participants as well as suggesting additional improvements, like adding a grocery card as a program benefit.

We’re going to adapt a survey that has been validated. How much can we adapt it? Will it still be validated?
So, the reality is that that survey wasn’t specifically validated for your population anyway. It’s been validated (which means that it’s been tested to ensure that it measures what it sets out to measure and that it measures consistently) but it was not tested with your participants or in your region. So you’re already adapting it. Making changes to language to make it apply to your participants (for example, changing “work” to “work or activities”, or changing “teachers” to “adults at the program” is going to make the survey easier for your participants to understand, avoid making them feel that they survey doesn’t apply to them, and provide you with better data. So, generally, as long as the gist of the question stays the same, it’s still better to adapt a validated instrument than to make one up from scratch and you can still refer to it as a validated instrument.

We distributed a survey and not everyone completed it. Can we still use the results?
Yes, something is better than nothing. Just ask yourself whether the completing the survey is random or likely correlated with something that would affect the answers. For example, participants who stay in your program until the very end probably like the program and found value in it so surveys collected at the very end of the program are probably skewed positive. You might want to take that into account when you analyze the data. For example, you might want to also calculate how many of the participants leave the program early and look into whether that might be an issue.

If the data collection was truly random – for example the Monday participants completed the survey and Tuesday participants did not, then I’d consider the results to be most likely representative of all the participants.