Well, these are unprecedented times.
Last week, I held a Facebook Live to talk about some of the ways that programs might be adjusting their data-collection and use right now as we grapple with the Coronavirus pandemic. Programs are being cancelled and adjusted and new programs are being developed. Here are some of my thoughts on the role of evaluators at a time like this.
For Ongoing Programs
Collect some data now
Many programs are not meeting any more. Before you lose track of your participants, try to collect whatever end-of-program feedback you had planned. Data collection will be incomplete or imperfect but this might be your best chance to collect any outcome data for this year. If you are planning to move to a distance or online model, collecting data at the transition point will help you assess the success of the transition later.
Take your collection online
If your post surveys were designed to be distributed on paper, now is the time to make the transition to online survey tools. Platforms such as Google forms and Survey Monkey will allow you to collect that data electronically. You can distribute links by email or social media.
Communicate with your stakeholders
Your funders, board members, and allies know that everything is in flux and you won’t be able to complete your programs or evaluations as planned. So, just reach out and let them know about changes as they’re happening. As an evaluator, focus on recording changes that are being made and adjusting data collection accordingly.
For New Programs
Remember that disaster relief does not need a logic model
If you’re distributing food to kids who won’t eat lunch at school or providing other essential services there are no outcomes to measure and your program does not need a logic model. Your role as an evaluator in this type of program is to ensure that the resource is available to the people who need it most and is distributed in an equitable manner. Now is the time for rapid-response evaluation and it does not need to be perfect. Focus on utilization — how many people have we reached relative to the need? Are there any barriers built into our distribution model that are keeping people from accessing the resource we need?
During a period like this you’re looking at measures that you can measure and report on a daily basis. How many people accessed the resource? Are we reaching all areas of our geographic area? Are we reaching all language groups? You can’t distribute a survey at this time — there is going to be a lot of estimating and that is OK. Think of yourself as a critical friend and project historian — you’re asking questions to make the program more effective and you’re logging changes as they’re made and documenting the rationale. Every day you’re reporting outcomes on key metrics such as reach and equity measures and helping staff to make informed adjustments.
Focus on equity
A very important part of your role as the evaluator is to help keep equity measures front and center. Help your colleagues think about the people with the most barriers to getting your help and on problem-solving to help reach those people. Disaggregate your data as much as you possibly can. Use this reporting to make sure your services are reaching marginalized and disadvantaged populations. This might mean trying to get feedback from individuals who are not accessing the service about want barriers are keeping them away. Consider using phone surveys or distributing surveys by email, social media or QR code.
Maintain or create a data dictionary
New programs and adaptations, new measures. Keep track of new measures with a data dictionary (I’ve written on how to use one here). This will help you keep track of new measures that might be created during this time and determine whether they are still useful going forward, or if they replace older measures that are no longer as important.
I’ve started a Facebook group where mission-based leaders can talk about the changes we are making to adjust right now. I’d love for you to join it here: https://www.facebook.com/groups/DataInformedPrograms/
And if I can help, please schedule a time to talk. 30- or 60- minute consultations are free. You can book using this link: https://app.hubspot.com/meetings/pieta