Multiple factors can weaken a study’s design and, as a result, lower confidence that the program caused observed changes. Such factors weaken the design by casting doubt on the source of observed changes.
Here are some important factors to consider:
- Very Similar Treatment and Comparison Groups One essential step that contributes to the strength of the research design is constructing the intervention and comparison groups so they are the same or very similar before the program starts. This similarity or equivalence of the two groups (known as baseline equivalence) provides the confidence that changes in outcomes are due to the intervention, rather than to other factors. Groups should be very similar or equivalent on all relevant characteristics. For example, for a program that aims to reduce depression and depressive symptoms, groups should have very similar depression levels before the program starts. Other important characteristics include race/ethnicity, socioeconomic status, geographic location, clinic type, and type of prior mental health diagnosis.
- Attrition (Loss of Study Participants) In almost every study, some participants drop out. When participants drop out of both groups, it is known as “overall attrition.” When one group loses more participants than the other, it is known as “differential attrition.” Both kinds of attrition may undermine the similarity or equivalence between the two groups, making it hard to trust that observed changes are due to the program.
- Missing Data. Missing data can also undermine the credibility of an evaluation. If data are missing at random and affect each group in the same way, an evaluator can make a stronger case that the results are credible. If the missing data differ between groups, or if these missing data are associated with a particular sample characteristic (for example, homelessness), then the findings are not as trustworthy.