Evaluation report writing can be an intimidating task, especially when it comes to Final Evaluation Reports (FERs). Our website has many resources that can help with FER writing and hopefully ease some of that anxiety. However, after hearing from evaluators in the field, we wanted to answer some of the questions about how FERs are scored. So grab your favorite beverage, sit back, and read on as we share the process with you.

Who scores the FERs?

From time immemorial (or at least since CTCP has funded us to do so) TCEC has had the opportunity to read and score FERs. It would be impossible for just one person to score all the reports, so a team of reviewers at TCEC works together to read and score the reports. Just like the report authors, the team of TCEC scorers relies on the Tell Your Story Guidelines scoring rubric. The scoring process is a large undertaking as there are several dozens of funded programs, some with multiple FERs to submit.

Calibration

As the saying goes, “Many hands make for light work,” but to make sure that all hands are pulling in the same direction, we go through a score calibration process. A sample of reports is randomly selected to be scored by all reviewers to ensure that we are all interpreting the rubric and assigning points in the same way. Typically at this point of process, scores tend to vary, and we know this because after scoring each report, the scoring team meets to discuss each section and why points were given or omitted. This process of double, triple, or quadruple (depending on the number of reviewers) scoring is repeated as many times as it takes to get all scorers calibrated within a 2-point deviation of all scoring team members. This is a standard benchmark for inter-rater reliability.

Then, to ensure reports are scored evenly across the board, the scoring team meets regularly to discuss any reports that may be challenging or unique. Additionally, the team lead monitors the scores to look for any drift in scores throughout the scoring process.

If scoring drift is detected, the team meets to go over scores, and an additional round of score calibration may be necessary. You guessed it, we go back and randomly select additional reports for double scoring. Our team is made up of dedicated evaluators who want to see the field succeed and thrive, so we work to provide helpful feedback in the reports, which can help an evaluator improve in their future report writing.

Hello OTIS, my old friend

Once all reports are complete, it is usually just the team lead that inputs the scores into OTIS. We know there are several buttons for each project to press to successfully upload their FERs. Someone at TCEC is also clicking several buttons to send useful feedback.

This all may sound like a long process, and it is. It takes our team several months to complete the scoring because we like to take our time to give feedback and assign appropriate scores to each report. We appreciate everyone’s patience as our small team carefully looks through each report to provide tips for how to improve next time all while consistently applying the rubric.

What to do with the scores and feedback

One question we receive most of the time is about the impact of the score. We even wrote a newsletter about it the last time around. We want to reiterate that the scores are not a reflection of the work or effort an agency put toward an objective. Whether an objective was achieved or not does not impact the score. The score simply shows how closely the report adhered to the rubric we referenced above. The main thing to focus on is the feedback about how to improve next time.  Although your FERs may not be due for a while, the feedback could be useful for your summary reports and progress reports as well.

It is a goal of all TCEC team members that FER scores reach “exemplary” ratings, so this is why we continue to put out resources on FER writing, from webinars to newsletter articles, monthly Q & A calls and FER Clinics. We are invested in your success and will have your back every step of the way!