By Sue Haun

During May and June, the Tobacco Control Evaluation Center staff provided 7 regional trainings for 117 participants from projects across the state. The purpose of the training was to help Local Lead Agencies (LLAs) expand their evaluation teams and make the most of everyone on the team.

By July 1, 2018, LLAs are required to have in place a full-time Project Director (PD), a 10% FTE Internal Evaluation Project Manager (EPM), and a 10% FTE External Evaluator (EE).  This in-person training was an opportunity to explain the differences in the evaluation-related roles between a PD, EPM, EE, and Internal Evaluator (IE). 

The training also provided participants with ways to grapple with how to divvy up tasks among team members, and how to use a GANTT project planning method to coordinate intervention and evaluation activities, and engage every member of the team. 

Data Collection & Analysis in Action! 

The training kicked off by asking participants to use sticky notes to identify their role and what they wanted to take away from the session.  Participants then used these sticky notes to create a bar chart of how many PDs, EEs, IEs and OPS (Other Project Staff) were in attendance.  This was data visualization in action— AWESOME!

Communication & Accountability

The second part of the training focused on making sure there are mechanisms within the team that support teamwork.  For many LLAs, having a team that consists of more than a Project Director, part-time staff, and a limited contract with an External Evaluator is really new. 

To help facilitate communication and coordinating mechanisms among these expanding teams, the first part of the training explained the differences between the PD, EPM, EE, IE, and other stakeholders’ roles.  Functional team dynamics were introduced.  These included knowing the different roles of each person on the team; identifying obstacles to the team’s functioning; establishing formal (as well as informal) communication mechanisms and meetings that incorporate the External Evaluator; and agreeing how team members will work together was also discussed.  Participants then got to “rate” their team on elements of communication and accountability, to identify where the team is strong and areas that may need to attention.  One participant commented: “The first thing I am going to do is establish a regular meeting that includes my external evaluator.”

Divvying Up Tasks

The next area of focus was identifying the skills related to carrying out evaluation activities using a sample evaluation plan.  The purpose of this exercise was to help participants allocate various evaluation activities, and think outside the box in utilizing members on the team.  In addition to brainstorming skills required for various evaluation activities, participants were also provided a checklist which could be used to identify skills for each member of the team.

Evaluation Roles

After lunch, the training delved in to the differences in the roles of the PD, EPM, EE, IE, tobacco coalition members and staff in a rousing game of Jeopardy!  After introducing the categories, one by one participants were asked to select a category and respond in traditional Jeopardy style – “What is . . . ?”  Everyone else were then asked to flash green, red or yellow cards – unlike traditional Jeopardy - to indicate whether they agree, disagree or were not sure about the participants’ response.  The answer was then revealed and points were awarded to all who got the answer correct.  The winners were given a box of tea as the prize.  (Why tea? More on that below!) 

For some participants, this activity was the highlight of the training.  One person commented, “I really enjoyed how interactive the training was (loved the Jeopardy game!). 

Another person noted, “[I] had all my questions answered, plus received so much additional content that will benefit my team and I.”

Planning, Collaboration & Monitoring

Where WiFi was available, training participants got to view a hilarious short video of the goal-oriented Minions screwing in a lightbulb.  Then different types of project planning software (e.g., Gantt Project, Smartsheet, etc.) and file sharing software (e.g., Dropbox, Amazon S3, etc.) were introduced. 

To experience a simple, project planning tool, participants then broke in to teams and developed a one-year Gantt Chart project timeline, sequencing activities so that the evaluation activities inform the intervention activities.  One EE participant commented, “it was useful to learn how the implementation tasks tie to the evaluation activities without bottlenecking the activities within each project year.  It was also helpful to learn the roles of the internal agency, since I’m more familiar with the EE role.” 

Participants then got to “rate” their team on planning, collaboration and monitoring mechanisms to identify where the team is strong and areas that may need attention.  As a result, one person intended to “Implement a program planning tool and do [these] exercises with evaluation staff.”

Resources for Evaluators

The final part of the training introduced various resources, including TEA – The Evaluators Alliance.  TEA is a new group initiated by TCEC for the purpose of exchanging information among evaluators and would be similar to the PDA for the project directors.  TEA is still under development, and more information will be available in the coming months. 

For those interested, other resources were also discussed.  These included:

• The American Evaluation Association (AEA)
• Claremont Graduate University Evaluation Professional Development Workshops
• The Evaluators’ Institute
• Of course, TCEC!

Wrap up

At the end of the training, participants shared what they learned as a result of the training.  Comments from participants included: “Thank you!  I feel much more comfortable about roles, responsibilities and how I might structure my team moving forward.”

You can find the PowerPoint and other resources related to this training on the TCEC website at http://tobaccoeval.ucdavis.edu/evaluation-planning/EvaluationResourcesrevised.html.

Call us at (530) 752-9951 or email us at tobaccoeval@ucdavis.edu with your questions any time.