I’m an educational researcher. I am invested in the improvement of education for children throughout the world. While some educational research is observational in nature, some research conducted in education seeks to make causal claims about the efficacy of interventions. Causal research is important to continue conducting in order to discern interventions that could lead to improvements in the lives of the students. However, cause-and-effect is harder to isolate in social contexts, such as education, and therefore it is more difficult to design and carry out studies that address the threats to validity that it might encounter. In education there are many well-intentioned individuals with big ideas, and money is thrown into improvements and interventions without reference to the actual impact of the intervention. By conducting experimental and quasi-experimental research in education, we can isolate effects and see whether the intervention was worth the cost that it took. Sharing the results of the studies gives us persuasive power beyond anecdotal evidence of efficacy. Educational policies and practices that are backed up by experiments with defensible design choices will be able to help us move forward into new territory by helping us discard notions that are not research-based, and accept those that are.
These are lofty goals. It is not simply enough to conduct an experiment (which includes random assignment and random selection) or quasi-experiment (which does not include random assignment). Both types of studies are susceptible to “threats of validity” — logical holes in the experiment’s design, which can undermine the findings or, if properly addressed, can strengthen a study’s causal claims. While I will not attempt to detail every threat to validity in this essay, I will address a few common ones that come up. The first threat to validity for quasi-experimental designs is selection bias. When individuals select themselves into a treatment or control group, there are unobserved personal characteristics that went into that decision, which can confound the effects of the treatment. Another threat to validity includes “pretesting effects”– a research subject who remembers taking a pretest (and its contents) can have an effect on their posttest scores. Low statistical power is another threat to validity: if you have very few research subjects, can you truly find whether the effects of the treatment were significant? Yet another threat is attrition; when you lose test subjects over the course of a study, you are losing valuable data from which to draw conclusions. It isn’t difficult to see that the design of a study has many considerations which must be addressed in order to be logically sound and stand up to threats of validity which might be leveled at it by other researchers.
One challenge in educational research is the actual implementation of interventions that are successful. We work with people in a complex ecosystem, some of whom are resistant to change. Making the findings of a study easy to understand and relevant to the group that would be implementing changes requires a great deal of time and effort if they are to carry the changes forward. Incentives for overturning the status quo are often lacking in education due to cultural conditioning; a proactive school leader might not be around to help make the changes needed in practice that are found to be effective. Sometimes there is cronyism involved in curriculum design and selection which can get in the way of switching to other methods that might have improved student benefits. But despite these challenges, it is still worth it to try new things and be open to educational research as a means of providing us with evidence for making small changes over the years.