Your Evaluation Questions Answered
How to use evaluation to win campaigns.
That’s how Paul Knepprath of the American Heart Association’s Tobacco Endgame Center for Organizing and Engagement characterized the presentation that TCEC gave on the Tobacco Endgame Center’s March Affinity call.
It was a great opportunity to talk with more than 50 tobacco control projects on what evaluation could do for them. Our slides reiterated that the primary purpose of evaluation is to inform and support project decisions and actions.
Evaluation can guide project strategies by measuring the scope of the problem, weighing policy options, identifying potential opposition, and signaling whether it’s the right time to move ahead. With external audiences, evaluation results can be used to generate buzz, counter misconceptions, build support, and leverage power over elected officials.
The series of data visualizations we showed generated a lot of viewer interest. Not only did the charts and tree map illustrate innovative ways to share results, a discussion of their findings revealed the next steps a project might take based on what the data indicated.
Participants were given a whirlwind tour of TCEC’s website and were asked to identify their biggest evaluation questions in breakout groups. We thought seeing their questions and our responses would be helpful to others, so we’ll be sharing them in a series of articles in the next few issues of the TCEC newsletter.
You may have evaluation questions of your own that you’d like answered. Contact our friendly evaluation associates at tcecta@phmail.ucdavis.edu for a speedy response.
Questions about Data Collection
What’s a good enough sample size for surveys?
It depends on how large the population of interest is and the levels of confidence and rigor expected by the users/audience of the data. For guidance on sampling, see:
What is the minimum number of responses needed to validate a community survey?
A sample size calculator can give you an idea of what is typically considered sufficient, but ultimately it depends on what will be convincing to policymakers, critics or the users of the data. Sometimes it makes sense to ask them. Your sampling strategy can make a difference too! Aim for the most representative sample you can get.
How do we address survey burnout—surveying the same population too much?
It’s hard to answer this without knowing more details but try targeting your samples to specific populations that are the best source of information about the issue. E.g., for issues about smoke-free housing, survey tenants who live in multi-family housing. For general public opinion in your jurisdiction, try collecting data in different ways and different locations (e.g., mailed postcard surveys, via Facebook or crowdsourcing sites to people who live in your city, convenience surveys at grocery stores, post offices, school parking lots, etc.) Collect data in the voting districts of those policy makers you who are undecided rather than across the entire jurisdiction-or county.
How do you increase response rates?
Besides providing incentives for completing the survey, you’ll have more success if:
- The topic is about an issue that will affect them or that they care about.
- The instrument has clear instructions, is well-designed and easy to navigate through.
- The first few questions hook their interest right away (so NOT demographic questions!).
- You keep it brief. Only ask those questions you absolutely need to know vs. information that might be just interesting to know.
See our Tips & Tools resource on Increasing Response Rates.
What method(s) of outreach to policymakers is most effective?
A key informant interview is an opportunity to learn about decision-makers, NOT to educate them about tobacco issues. When you call their office to request an interview, tell them you are with a public health program and want to learn more about their concerns and priorities on public health and the community. Then they’ll be much more likely to give you 20 minutes of their time.
If that doesn’t work, then:
- Leverage coalition connections. Who do your coalition members know who could get you an introduction?
- Find ways to establish ongoing relationships with them and their staff. Become a resource they can turn to for reliable information on a range of health and environmental issues.
- Go to city council and board of supervisor meetings. Stay the entire meeting and introduce yourself during a break or afterwards.
Train and engage youth to interview staff in a professional manner to learn about policymaker schedules, best way to approach them, how they like to receive information on issues of community concern.
How much time is enough to do a key informant interview? We usually spend more time talking to them about tobacco education and only have a little bit of time left to ask them questions for the evaluation.
A KII activity is the time to do more listening than talking—to seek information rather than deliver it. Get to know their priorities and concerns first and then attempt to frame your “asks” and issues in ways that align with that. The goal is to learn about what they know of an issue, the reasons for their views, and find potential points of connection. This will help you develop your talking points, educational materials, policy options, and timing of next steps.
Beyond that, the amount of time needed to explore their views depends on a number of factors—whether you already have a relationship with the informant, how much time the informant will give you, the number and depth of the questions you want to ask, whether the informant answers you directly or tends to go off topic or into too much detail.
For more, visit the Evaluation Guide.
For more evaluation questions answered, stay tuned for the next edition of the Tobacco Control Evaluation Center newsletter!