FAQS on FERs
Every quarter, evaluators have an opportunity to connect with other evaluators in the Tobacco Evaluator Alliance. At the last meeting, some requested additional help on FER writing, so we got together with Yaneth from the Hispanic Coordinating Center to quickly offer a FER Q/A session. These are the main takeaways from that session and an invitation to join us for the next one.
Where should I start?
Review the guidelines, sample reports, and scoring rubric in Tell Your Story (TYS): Writing Useful Evaluation Reports, 2017. Be sure that you’re using the correct version written in 2017.
Capture the main components on this pre-writing worksheet. Note that this is not something you will turn in. This is just a way to help you organize what you’d like to include in the FER itself.
Review the other webinars, data visualization templates, and other resources.
Have your workplan handy. You’ll find a lot of information from your workplan and progress reports in OTIS. The PC reviews/feedback from the 6-month progress reports provide nice summative info that can help too.
Start early. It takes time to put it all together and to allow time to get feedback from your team.
How many FERs and BERs do I have to write?
It depends. Most projects generally write 1 FER, but some are required to write more.
FERs are required for primary objectives. For non-primary objectives, a Brief Evaluation Report (BER) is submitted. In OTIS, they might all be called FERs, but this is because that is the choice in the drop-down menu in OTIS. The activity text should have been corrected to state BER for non-primary objectives, but sometimes it is not corrected during the review process.
Check out this article on the differences between BERs and FERs.
Who writes the FER/BER?
Can you talk about how staff can best support the FER writing? Do you have any suggestions for how project staff and the evaluator can work together to complete the FER? For instance, which sections should project staff take the lead on, and which sections should the evaluator take the lead on? What are some tips for reporting program progress and evaluation results in ways that are integrated or built on each other, so they don't seem so separate?
In general, project staff can probably write the Aim & Outcome section, Background section, and the two required data visualizations. These sections can generally be answered with information in your OTIS workplan e.g., rationale for choosing the objective, previous work done, activity dates and details, etc.
Writing FERs should be done collaboratively. An evaluator should not be writing the report by themselves. All members of the team should be part of the FER, not necessarily that everyone writes the FER, but the team should discuss their reflections, learnings, and important takeaways. Remember to also allow for plenty of time for back and forth with the project staff and evaluator.
Integrating activities is something that should be done throughout the project, not just at the end when writing it up. Build in plenty of time to reflect throughout the whole project timeline. For example, create a timeline for how intervention and evaluation activities build upon one another.
Another important reminder is for project staff to document and reflect along the way. These are important ways for staff to contribute to the FER. Here are some other tools for documenting organization progress and facilitating reflection discussion throughout the whole project.
Who is the audience for the FER/BER?
Is it CTCP? The public? Decision-makers at the state level? Our client (the grantee)?
CTCP is one of the main audiences for the FER. They are our funder, and one of the requirements for receiving our funding is our periodic reporting and end-of-contract reporting for accountability and monitoring. Researchers are a potential audience, including us at TCEC where we write summary reports on trends, patterns, or strategies. Decisionmakers may also be a potential audience; however, it will likely need to be written in a different, shorter format. Some may want the full report, but most would likely want something like an executive summary or presentation instead.
Another main audience is your own project and other projects that may be working with the same topics or communities. Be sure to provide extra details on what you would do differently and why. Think of it like you’re writing for your future self. Include details about what you would want to know in the future to be successful, not to make the same mistakes, to replicate the positive experiences, etc.
One important use for FERs is to help transition, so the work done can continue and memory isn’t lost (especially if the project will not pursue additional funding). It is helpful to also write the FER by keeping in mind the things that would be helpful to your future self or future colleagues working with a similar topic or community.
FERs and documenting as you go can also be helpful when there is staff turnover. It is important to write things down periodically. One specific strategy that was helpful for me (as the evaluator) was to do a casual interview/mini-focus group with the staff team before someone left to get some of the narrative/context on what went well, what didn't, and any key learnings/takeaways.
Here are some other tools for documenting organizational progress and facilitating reflection discussion throughout the whole project.
You can also find resources and forms on our instrument database. Scroll past the search options and go to “Project Planning and Monitoring” under the “Instrument Topics” section at the bottom. Select “Organizational History” then click the “Find Instruments” button at the bottom of the page. Depending on your screen, you may need to scroll up to see the forms.
Remember that All FERs are public documents. This is important to note, because you can request FERs from the Rover Library. For example, if you wanted to read through previously submitted FERs on a particular topic or indicator. Also, people are generally willing to share their reports if you ask them. It’s also important to remember that FERs are public documents so that you do not include any personal information like full names or contact information. If the FER refers to individuals, try to use positions rather than real names.
The FER/BER is due on the last day of the grant cycle. Since we will likely be continuing project work for several more months, it would be useful to understand the best way to prioritize staff time.
It can be helpful to choose a date when no new work will be done, for example 30 or 60 days before the end of the contract. This way, you’re able to wrap up most of the work and focus
Write what you’ve done so far, then have a placeholder for the work that will be completed in the coming months. You can even put an actual box to show where you plan on including a chart or graph.
The Aim & Outcome, Background, and Evaluation Methods & Design sections can be written early, because a lot of this information is already in your work plan. The blank data collection instruments can also be compiled for the Appendix ahead of time.
How do you suggest organizing a FER/BER when there are three, four, or five target jurisdictions, each with a different timeline and different policy processes that occurred?
Be sure to report on all of them. Do not just include when an objective is successful. At the very least, write about the number of jurisdictions listed in your objective.
One way to organize a FER or BER with multiple jurisdictions is to discuss them chronologically. Some communities start in one jurisdiction and then move onto another jurisdiction, so it could make sense to discuss them in time order.
Another way is first to discuss the jurisdictions in which the objective was met and then discuss the jurisdictions in which the objective was not met.
In the Results section, some have organized the results by year with subheadings by jurisdiction. Others had sections for each jurisdiction and went through the results by year.
Whatever order or strategy you use, be clear with the labeling so the reader doesn’t get confused.
I remember a data vis template was provided last round. Are we required to use that, or can we come up with something else?
We created templates to make it easy for you, but you can use another format. You don’t have to use the exact format. You can change it, but be sure to have the same basic information. You may want to add more fields. For example, in the table of evaluation activities, I would add how the information was used or disseminated. I would also change the colors so it matches my organization's branding and maybe put it in landscape format to fit everything on one page.
If we have subcontractors, do we include their work as well in answering these questions?
Yes, include the relevant work that was done for the objective.
What other resources should we review before starting?
We listed resources above, and we also created a separate page where we copied resources related to FERs here https://tobaccoeval.ucdavis.edu/final-evaluation-reports. It's the same or similar content as before, the resources are just on a dedicated FER page. As you'll see, there are tons of resources on this topic, and this is because the task of writing FERs requires a proportionally intense amount of time and effort to complete. It is important to appropriately staff, train, and budget for that time and effort.
What is the difference between the worksheet and the TYS guidelines?
The worksheet is an optional tool for your team to use before and during the FER writing process. The TYS guidelines are much more detailed and therefore longer than the worksheet. The guidelines are what we are contractually obligated to follow, while the worksheet is meant as an optional tool. The worksheet is not turned in and is instead used as an internal document. We recommend that both be used in your team's FER writing process.
Participants shared that the worksheet can be especially helpful for LLAs with an external evaluator to facilitate discussions about the FER so that you don't need to rely on memory or sifting through emails. There are additional resources we can share to help you not rely on memory is to be sure to document organizational progress and incorporate opportunities for reflection throughout the timeline. You'll also want to check out the resources on writing summary reports which are all saved to your project's OTIS record. This way, if there is staff turnover or if you want to revisit the details of an evaluation activity, you can review the summary report you uploaded during previous progress reports.
Where does it say that FERs are required for primary objectives and BERs are required for non-primary objectives?
This is stated here, here, and here. You don't have to take our word for it; it's also listed in The Policy Section of the Administrative and Policy Manual on OTIS. A similar question is discussed above.
For Statewide Projects such as TA Providers and Coordinating Centers without policy objectives, a BER is required for each objective. This should be stated in the RFA.