A few months ago, I was invited to take part in the American Heart Association’s Tobacco Endgame Center for Organizing and Engagement’s Affinity call to share ideas about how evaluation can inform and support project decisions and actions.  Or as Paul Knepprath, the Tobacco Endgame Center’s project director, put it: How to use evaluation to win campaigns. 

The presentation discussed how evaluation activities can guide project strategies by measuring the scope of the problem, weighing policy options, identifying potential opposition, and signaling whether it’s the right time to move ahead. I explained that evaluation results can be used with external audiences to generate buzz, counter misconceptions, build support, and leverage power over elected officials. I had participants look at a series of data visualizations and discuss what next steps a project might take based on what the data indicated. Then I gave a whirlwind tour of TCEC’s website and services so participants could see how the Tobacco Control Evaluation Center can help answer their evaluation needs. Toward the end of the call, I answered the remaining evaluation questions that people had.  

Although the presentation was not recorded, we thought it would be beneficial for you to see the questions participants had and our responses. So, we’re sharing them in a series of articles in the TCEC newsletter. This installment is Part II. You can read Part I here.

If you have evaluation questions of your own that you’d like answered, contact our friendly evaluation associates at tcecta@phmail.ucdavis.edu for a speedy response.

Questions about Data Collection

What is the most effective method to collect information from business owners?

For many retail objectives, it might not be advantageous to collect data from store owners. It’s probably sufficient to document conditions with store observations and then present the data to the elected officials who make jurisdiction-level policy. One instance when it would make sense to interview or survey business owners is if you wanted to find out which policy provisions would be most appealing or feasible to them OR if you were a promoting healthy store incentive program.  In that case, you’ll have a greater chance of engaging them if you:

• Identify industry organizations that could be important gatekeepers
• Frame the topic in wording that aligns with their concerns
• Keep their schedules in mind.  They may only give you 10-15 minutes of time to complete the survey or interview

Make it clear that you are not there from Environmental Health or other compliance-checking entity and that their participation is voluntary and will not affect their status or “grade” in any way.

What tips do you have about conducting electronic surveys using Facebook or other social media platforms to get responses?

Use boosted content or ads with a direct link to the survey to encourage participation. Tailor the post’s format, framing of the issue, the wording of the headline, etc. differently for each specific audience being targeted. Use keywords and demographic characteristics such as zip code, age, and personal interests to focus on your desired audience. The narrower your parameters, the more often users in that target demographic will see your boosted content. To avoid getting flagged by or excluded by algorithms, don’t use the terms “smoking” or “tobacco”.

Utilize multiple social media platforms to reach audiences. If you want to reach public officials or people who follow trends, use Twitter. If you’re asking for the opinions of younger audiences, they’re more likely to be using Instagram.

See Designing Online and Mobile Surveys and Attack of the Bots!

What is the best way to avoid robot attacks on surveys and zoom registrations?

• Recruit participants using offline methods such as postcards or door hangers with QR codes instead of through social media
• Require secondary authentication using cookies, an IP address, or an access code
• Use a CAPTCHA
• Use filters that require entering a valid zip code in order to continue with the survey. Don’t provide the city until the filter is cleared so bot engineers can’t research relevant zip codes.
• Look for multiple registrations with the same IP address

See our article Attack of the Bots! 

What is a good way to streamline consumer product testing of a variety of materials?

It is not clear what this question is trying to get at exactly, but it can be time-saving to test different materials with the same audience at one sitting — if time allows. (So you don’t have to spend time on recruiting more focus group participants.)  However, each material may have a different purpose, message, format, etc. so, each set of questions will need to be unique to the material.  It doesn’t make sense to skip on getting the feedback you need just in the interest of time, because it’s more important to ensure the material is having the effect you want.  Otherwise, you may spend a lot of time and resources on something that doesn’t work.

For guidance on materials testing see this section of the Evaluation Guide.

Also see the consumer testing guide from the Tobacco Education Clearinghouse of California.

What is the best method/tool that you can use to record evaluation data?

For quantitative data, make use of your project’s SurveyAnalytics account whenever possible so that data is recorded electronically on a mobile device or computer. This removes the need to input data and reduces errors.  SurveyAnalytics will also perform basic analysis functions as well!  The platform also allows for data collection in Spanish and other languages.  Qualitative data requires other modes of capturing data such as Zoom recordings and transcriptions, photos or video footage, etc.

See Using Mobile Technology. 

How should we include information that is not captured by the data collection instrument? For example, an observation form asked what the signage looks like at apartment complexes and there are questions that follow, but the complex didn’t have signs so the questions are not applicable.

This issue points to the need for always pilot testing data collection instruments.  The form should have included a “Does not apply/None” option. Hopefully, there is a final comment box at the end of the section or instrument where data collectors can cite any unusual circumstances like that. The other option is to describe the issue in your field notes.

See Pilot Testing. 

Also see Observation Methods and Writing Field Notes.

For the final installment of Evaluation questions answered, stay tuned for the next edition of the Tobacco Control Evaluation Center newsletter!