The limitations section in an evaluation report can feel a bit like a confessional.  When writing these, one can feel like they are admitting all the ways they got it wrong; revealing the skeletons in their evaluation closet.  It can be uncomfortable and maybe even a little disheartening.  So why do we do it?

We can start with the fact that it is a reporting requirement.  An excerpt from Tell Your Story: Guidelines for Preparing Useful Evaluation Reports asserts, “Lastly, state potential design limitations. For example, due to limited resources, the public opinion survey sample size was too small to be fully representative.”

Ok fine, you have to.  But there are a lot of other compelling reasons to be thorough when including your limitations in your report:

• It is honest: We don’t want to unintentionally mislead the reader.
• It helps other evaluations, and our future selves, to better replicate efforts: This is easier when we have a clear picture of what was actually done in the first place (not just a vague and general description).
• It allows documenting of both successful strategies and tactics to overcome barriers and challenges: Whether something worked or not, specific descriptions help us better understand strategies used.
• It speaks to gaps, needs, and real-life circumstances: If a strategy is not a good fit for a community, we can demonstrate that here.
• It makes your data and story more USEFUL: Transparency in reporting allows the reader to understand decisions that were made which can help them avoid misrepresenting the information or worse, replicating strategies with undue bias embedded.

One can argue that the Recommendation section of an evaluation report is the most powerful component to come out of your entire evaluation effort. The Centers for Disease Control and Prevention’s (CDC) Framework for Program Evaluation in Public Health discusses why the Recommendation section of an evaluation report is critical to an entire evaluation:

Recommendations are actions for consideration resulting from the evaluation. Forming recommendations is a distinct element of program evaluation that requires information beyond what is necessary to form judgments regarding program performance.

Simply stated, this section elevates the utility of your evaluation from a program ‘grading’ tool to something that can help guide future programs forge a quicker path to success.  This CDC document calls for a “complete and fair assessment,” explaining that “[t]he evaluation should be complete and fair in its examination and recording of strengths and weaknesses of the program so that strengths can be enhanced and problem areas addressed."

Heather Krause’s We All Count is a project that continually develops tools, case studies, practices, and systems to improve equity in data science. In a recent post titled, Motivation Touchstones: Using Restrictions to Your Advantage, Krause speaks to how reporting limitations or restrictions in our evaluations can be an equity issue:

Being open about the time, money, capacity, and regulatory restrictions that you’re working under explains to your stakeholders and your team members why you are going about your project in the way you are--Your team, your funders, and your stakeholders will be confused when they can sense a compromise but they don’t know why you made it. Defining your restrictions beforehand sets them up as reasonable and realistic considerations that you used to make efficient and transparent decisions, rather than flaws in your final data product.

When transparent in our evaluation reporting, we have the opportunity to explain our motivations behind our decision-making, decreasing the risk of introducing undue bias into our data projects.

Yes, listing limitations in the Recommendations section can feel like one more (maybe unpleasant) aspect to report, but it may be the one you and others will come back to for answers when moving forward.  If we can take the time to provide a complete presentation of our evaluation limitations, we will enrich the reader’s understanding of the context around your project’s evaluation successes and challenges.