by Robin Kipke

Once upon a time (it seems so long ago in the wake of current events) during the month of FER-bruary, the Tobacco Control Evaluation Center (TCEC) held a series of four webinars that explained what goes into writing a good final evaluation report.  It all comes down to telling an efficient story that has just the right amount of detail.

For this reason, final evaluation reports and Goldilocks have a lot in common.  To end up with the desired result, it’s important to achieve just the right balance—include too much detail and it’s hard to follow the story and know what is important; include too little detail and readers are left with unanswered questions about what you did.  The trick is to include just enough relevant detail to explain the desired outcome, purpose, context, results and implications of project approaches, tactics and activities.

Let’s break it down into some of the key sections of the report and show you what we mean:

Aim & Outcome

Start off the report by stating up front what the project was aiming to do and what happened in the end.  State the objective and whether or not it was achieved (and to what degree).  Describe the extent to which it has been implemented (if that is part of the objective).

In order to limit youth access to tobacco products, the Unicorn Program set the following objective:
By June 30, 2020, at least two cities in the Coastal Mountain Region will adopt and implement a tobacco retail licensing policy that earmarks a portion of the license fee for enforcement activities. This is a primary objective addressing Communities of Excellence indicator number 3.2.1.

The objective was only partially met by the end of the contract.  While TRL policies which earmark license fees for enforcement have been adopted in two coastal mountain jurisdictions--Isabel Flats as of September 2019 and Jeramiah Junction in January 2020--the law in the latter jurisdiction will not go into effect until January 1, 2021.  The policy in Isabel Flats went into effect March 1, 2020 and already 66 licenses have been issued.

Background

Provide relevant context about the environment in which the work took place.  Identify the specific jurisdictions targeted and describe relevant characteristics about the population, tobacco norms, political and economic climate that the project had to contend with.  Briefly describe past tobacco control efforts in the area, and whether any work had been done on similar objectives before.  In 1-2 sentences, summarize why this objective was selected and who was involved in the selection.

Here’s an example of documenting previous tobacco control efforts in the targeted areas:

In the past 10 years, the Maple County Tobacco Prevention Program has successfully worked with the Board of Supervisors, Pinecone City, and the City of North Elm to pass Tobacco Retail Licensing ordinances in those jurisdictions. Regular monitoring and enforcement led to reductions in the illegal sale rate to minors from 28% to 0.5% on average in the unincorporated areas, from 30.5% to less than 2% in Pinecone City, and from 19% in North Elm to 5%. 

We hoped these impressive reduction rates would motivate the four remaining cities to adopt TRL ordinances.  In 2013, project staff contacted leaders in Pistache and Liquid Amber, but to little avail.  Pistache was in the middle of a scandal in the sheriff’s office.  A few members of the Liquid Amber’s city council were interested but others were staunchly focused on attracting new businesses to the area. No previous outreach has been made in Pecan City or Pine Grove.

Evaluation Methods and Design

 In a table, identify the purpose, timeframe, sample size and composition, and analysis type for all of the key evaluation activities in the objective. For example:

Accompany this with a narrative that describes how each activity supported the objective, such as:

To inform the public education campaign messaging, focus groups were conducted with high school tobacco/vape users to understand youth access, product preferences and motivations for uptake.  The findings were used to create PSAs and fact sheets to educate parents about how youth obtain products illegally, popular products and quit resources.

Implementation and Results

Start the section with a timeline that helps readers understand how the intervention and evaluation activities fit together to move the objective forward.  Then concisely describe the WAHU of each key activity—Why was the project conducting the activity, what Action took place, what Happened as a result, and how was the outcome or learning Used by the project to inform its next steps?  Notice how this example describes what the activity consisted of, who it targeted, and what difference it made to the project.

To continue laying the framework of our outreach to healthcare facilities, collaboration occurred via meetings, phone calls, and emails to share effective resources and health benefits of smoke-free polices. We used Prezi, handouts, statistics, copies of policies in other counties, health ranking statistics etc. to share with individuals at Gold Miner Hospital, Aleut Tribal Health Consortium, Good Health Hospital and Alluvial Community Health Clinic, as well as local dentists.

In Year 3, Ask, Advise, Refer packets were sent to 30 different healthcare facilities including dental offices, pediatricians, behavioral health and residential treatment centers. These packets also contained information on the benefits of a smoke-free environment and the dangers of second-hand smoke.

Educating the community and building relationships with local healthcare facilities helped to move this objective forward. The coalition was invited to attend and present at 18 meetings of organizations such as the Health Leadership Network, Planned Parenthood, and Big Tree Hospital. This allowed us to recruit advocates at many of the target facilities who also spoke at the regional health conference, creating wider interest in the policy across the region and leading to a majority of attending organizations eventually adopting the policy.

Evaluation activity result narratives can be paired with data visualizations, when appropriate, that make it easy for readers to discern the key finding.  The section should also identify how the project tailored its activities or used culturally competent practices appropriate for segments of the community you attempted to work with.  Here’s an example:

We created low-literacy data collection instruments by consulting with members of a local church that served this population.  Because people of Low SES can view organized society and authority with distrust, we created an informal environment by starting out with 10 minutes of small talk. We did not use a written agenda or a PowerPoint presentation.

Conclusions and Recommendations

In addition to stating whether the objective was met, in the conclusion section you’ll need to assess the efficacy of key activities in moving the objective forward.  Attempt to explain why you think activities turned out to the way they did—whether they were really helpful or did not have the anticipated effect.  Based on these conclusions, make concrete recommendations for future work that point to where efforts should be replicated or improved upon.

The Don’t Start Project’s objective to get at least one jurisdiction in Whooping Crane County to adopt and implement a tobacco retail policy was not met. In the end, we could not convince policymakers that a licensing ordinance would actually reduce illegal sales rates to minors and do so without harming the profit margins of local businesses. This is despite having data that shows high public support. To date, only one of the six jurisdictions in the county (Spikey Pines) has adopted a tobacco retail licensing policy.

Policymakers did not find our Youth Tobacco Purchase Survey convincing.  While 2012 YTPS data indicated that illegal sales to underage youth were a problem, the sample size was small.  Despite focus group data from high school students which indicate that youth find ways to buy tobacco/vape products from local stores, the results from our 2017 YATPS done with an even smaller sample did not confirm this. We suspect this was the result of a telephone tree action among retailers more than it was due to their actual compliance with state law.  As a result it was difficult to convince policymakers that the need for a policy outweighed potential effects on tobacco sellers.  Conducting annual YATPS activities in may be necessary, employing strategies to be less conspicuous to tobacco retailers.  Providing data on a comparable jurisdiction that has adopted a tobacco retail licensing policy as well as documenting more public support with larger sample sizes may also be necessary to convince policy makers that a TRL policy is feasible and needed.

Appendix

Each data collection instrument used on the objective should be attached in the appendix.  You can also include documents that supplement the report with additional detail so if a reader wants to delve deeper into a particular intervention activity, data collection methodology or more comprehensive analysis of results, for example, s/he can do so without all that detail interrupting the flow of the report story.

Abstract

The abstract should include the briefest summary of the key aspects of the report—stating the aim and outcome of the objective and activities and/or findings that proved pivotal in achieving the end result.  For more specifics, see the Tell Your Story guidance.

 

The TCEC website has a whole bunch of resources on writing evaluation reports as well as ways to document project progress, activity summaries, progress reports and final or brief reports.  See our page on Reporting Results. 

• To look at the Tell Your Story guidelines for writing evaluation reports, see this document, which includes instructions for each section, sample report, and scoring rubric.
• To watch the series of FER-bruary videos on writing final reports, see the FER-bruary series 1-4 in 2020 in our Webinar Archive.  Each week covers a different section of the report.