We all have our workplans, and in an ideal world, we would complete all activities by the assigned due date without incident. However, we are all painfully aware that we do not live in a perfect world, and oftentimes things do not go according to plan. What do we do then? In this article, we explore some real-life situations in which things did not go according to plan and how some evaluators have handled it.

Communication breakdowns

Sometimes challenges arise when an external evaluator and project staff do not effectively communicate about the deliverable. For example, the staff makes great progress in getting an activity completed, and the data are provided to the evaluator to write the report, but the projects discover it does not meet the requirement outlined in their work plan. The good news here is that these issues can be avoided by having clear communication, roles, and expectations laid out between the project staff and the external evaluator.

The Working with Evaluators section of our website has a suite of resources to help establish clear expectations of the role and a regular communication system. We also invite you to learn about and discuss ways to foster an effective relationship with your evaluator during the quarterly Tobacco Evaluator Alliance (TEA) meetings, where you can learn from other CTCP-funded evaluators and project staff.

Underestimating the time needed

One evaluator shared how projects have waited too long to begin data collection, waiting until right before the deadline, and how this makes completing the work more challenging. Not only does it create stress on the staff and evaluator as they try to complete data collection and reporting writing in a hurry, but it also limits the utility of the data they are gathering.

Yes, data collection is a requirement, but more importantly, data collected is to be used by your project and for your project. Strategic planning of data collection can ensure that your project has data that can be used in real-time to inform and guide implementation activities.

This evaluator decided to work with the project to develop a timeline that builds in more time for analysis and reporting. They offered an earlier deadline by which data collection needed to be completed. In order to write a summary to submit in the January progress report, data collection needed to be completed by mid-November (and to account for holiday closures).

Another evaluator discussed challenges before data collection begins:

One thing that comes to mind is that it took us MUCH longer than expected to get incentives through the internal approval & purchase process. We tried to conduct public opinion poll surveys without incentives because we were coming up on the data collection deadline, but it was so difficult (definitely noticed a difference pre and post Covid – people were mostly not interested in stopping to answer questions). So, we decided to buy a list of registered voters and email the survey to them – out of around 13K registered voters, we received around 450 responses. When we spoke with the mayor about the survey results, she seemed impressed with the number of respondents so it ended up working out.

Needing changes to the workplan

Another common challenge we hear about is when a policy does not get adopted as expected.  How do you evaluate the post-adoption policy environment without policy adoption? The short answer is you cannot.  Since there are post-policy adoption activities in your workplan, you’ll need to work with your PC to figure out what would be appropriate substitutes or actions to take instead. We discuss some common solutions in our Evaluation Guide, and you can also schedule a consultation with us to brainstorm other options.

Key takeaways:

Ultimately, we cannot control many factors that contribute to our work. But we do have ways in which we can address them:

• Communicate with your PC: It is important that we communicate these unforeseen issues with our PC; we are all partners in this work and they will often have suggestions to help mitigate the impact on your workplan. It may also be necessary to make adjustments to your workplan or budget so that you can adapt to unexpected changes.
• Reach out to Statewide TA providers: The statewide projects are there to be a resource to local projects. They can help you brainstorm ideas to make the best of whatever situation arises.
• Ask other projects: Chances are, if your project is experiencing a challenge, others are too.  Be sure to reach out through the Project Director Association, Tobacco Evaluator Alliance, or simply reach out to a neighboring project via email or Infohub. 
• Include it in the report:  Whatever barriers you are experiencing, it is helpful to make sure it is reported on. Not only does it help funders and stakeholders to better understand the project’s journey, but it also serves as institutional memory for future efforts. We are all able to learn from your experiences this way.

And as always, the TCEC team is here to help you with all things evaluation! Email us at tobaccoeval@ucdavis.edu or explore our resources at https://tobaccoeval.ucdavis.edu.