Better survey results: Writing great questions

Surveys are a go-to method in program evaluation, used both to collect feedback from individuals who have participated in our programs and also to understand the needs and opinions of people in the communities we serve. Particularly because we can often distribute them electronically and collect responses easily, we employ a lot of them. But like all methods, surveys should be employed with care. Not all surveys generate useful results.

I’ve written elsewhere about asking the right people to complete your survey, and about avoiding ethical issues in implementing a survey. I’ve even written about how to avoid conducting a survey altogether. Today, I’m going to talk about writing questions that give you the information you need.

The way to write good survey questions is to start with the answer you need and work backward to formulate a question. This will help you avoid asking unnecessary questions or creating answer choices that are not easy to analyze. You might find that you have to take one question and break it up into several to make sure that it’s really clear. For example, if you want to understand whether your students have internet access at home that they can use to do their homework, you might ask whether they have internet access, whether it is dial-up or broadband, and whether they have their own computer at home in three separate questions. Answer choices like “yes, we have internet access but my sister always hogs the computer” won’t be available.

{{cta(‘9181152a-00a9-45de-9dfd-3e3552d26951’)}}

Think through how you’ll analyze the responses as you’re writing questions. How will you present responses from a question that lets people pick as many choices as they want? Or how will you present a ranking question? Realizing how hard it is to present or use the responses to some questions might help you streamline the way you ask them.

The most common problem that I see with survey questions is the “double-barrelled” question. That’s one where a single question asks about two concepts. For example, if I ask “how convenient was the time and location of this session?” I won’t know whether respondents are rating the time, the location, or both. As a result, I won’t know which element to change if I’m getting negative feedback. Keeping to one concept per question will also help you provide answer choices that are both comprehensive and mutually exclusive (if it’s a pick-one question), so that the respondent knows exactly how to answer. 

It’s very important to ask questions using words that your respondents will understand. If you need to define a term, do so right in the question or in the introductory text above the question so that the respondent doesn’t have to search for a definition. If a respondent isn’t sure what you’re asking, they might guess, or select an answer at random which will make your results meaningless. A good way to tell if your terminology is clear is to have some people who represent the group who will be completing the survey review it for you in advance. 

These reviewers can also help you to understand if you have all the appropriate answer choices listed. You want to make sure that your list is exhaustive. It’s tempting to think of a handful of answers, and then add a space for “other” at the bottom of the question, but you need to take more time than that. When you list answer choices, you’re doing so from your perspective that’s usually not that of the participants in your program. You may leave out huge categories of relevant information, and then depend on your respondents to explain these to you in a tiny little box. A second reason is that it’s extremely time consuming to code those answers. Since each respondent will have written their answer in their own way, there is no automated way to aggregate those answers and it can take hours and hours. If your survey was implemented in multiple languages you’ll also have to have all those “other” responses translated back to the language you read.  

The advantage of a survey is that your respondents can reply in private, and that’s also the survey’s greatest risk. You can’t see your respondents, so you don’t know if they’re confused or frustrated as they’re answering your questions. If they make mistakes because they couldn’t understand your question or answer at random because they are frustrated you won’t be able to tell that that is what happened. You run the risk of missing information, or –worse — producing bad information. Taking time to write great questions and to have your survey reviewed for you will help ensure that you generate useful and powerful data.

 {{cta(‘9181152a-00a9-45de-9dfd-3e3552d26951’)}}