Part 2 – Existing Research on Selection and Possible Solutions
from MasonARC’s 3 Part Series: The Fundamental Questions of Arts Education and Causal Outcomes.
This post is by: Alenamie Alegrado and Dr. Adam Winsler
This is part two of a three-part series that explores the question: Do the arts make kids smart, or do already smart kids choose the arts? In part one of the series, we explained the problem of “selection effects” and how they influence those who do and do not participate in the arts. This leaves researchers with the challenge of comparing two groups of kids (arts and non-arts) who are often already very different.
The arts are valuable for their own sake (National Endowment for the Arts [NEA], 2012; Winner, Goldstein, & Vincent-Lancrin, 2013). Yet simultaneously, arts educators and researchers have proclaimed the benefits of high-quality arts engagement describing how arts training supports children’s cognitive, social, emotional, and behavioral development, which ultimately leads to improvement school performance (Eisner, 1998; Menzer, 2015; Winner et al., 2013). When we see that arts kids do better in school compared to non-arts kids, it seems like there is something about the arts that gives that gives these kids an edge, but is it fair to simply compare arts-involved kids to non-arts kids?
There are many things that make children different from each other besides the fact that some take dance classes and other don’t. A child’s decision and ability to select into arts experiences depend on more than just interest in the arts. Researchers call this the problem of selection. For example, it’s likely not fair to simply compare a child who received violin lessons for 7 years to a child who did not forgetting about other factors. It’s only fair if the two kids have equal backgrounds or equal opportunity to participate in the arts – otherwise, we could be comparing apples with access to oranges without opportunity. And what is worse, the ways in which the families are different (i.e., income, opportunities) are strongly related to academic performance in children. If we can’t simply compare arts kids to non-arts kids, what can we do?
The best way to definitively conclude that early arts exposure causes enhanced academic, social, motivational, or behavioral outcomes for children is to randomly assign students to long-term arts exposure groups and measure outcomes afterwards. You can imagine that this kind of study would be difficult to organize and execute with accuracy. It might look like this – 100 kids would be randomly assigned to get 6 years of ballet training for 4 hours a week (and they have to stick with it all 6 years!) and another group of 100 kids is not allowed to have any dance training over the same time period. It’s simply not realistic. However, a few experimental studies have impressively randomly assigned children to arts conditions and found that children in arts groups outperformed children in non-arts groups on a variety of outcomes (Greenfader & Brouillette, 2017; Goldsten & Lerner, 2017; Holochwost et al., 2017; Moreno et al., 2011; Schellenberg, 2004; Lobo & Winsler, 2006). Although these studies were able to remove the problem of selection by randomly assigning children to arts conditions, the possibility of additional (unmeasured) arts engagement outside of the study remains. We should also question whether experimental studies included enough diversity in their sample for us to apply the results to children of all backgrounds. Finally, a challenge for experimental studies is to extend their treatment/exposure group long enough to answer questions about long-term arts exposure.
Studies that use random assignment can be quite costly and require careful attention to ethics to ensure researchers aren’t denying some participants access to a potentially beneficial treatment. One limitation of true experimental designs is that researchers cannot always control the groups they want to study, and intervention programs typically are restricted to short –term after-school programs or programs during early childhood. One can’t randomly assign students to take arts elective courses in school for years even though an important question that researchers have is what is the effect of taking orchestra or drama in school for years? This is why comparing naturally occurring groups, like orchestra students, drama students, and non-arts students in public schools, is more commonly done with correlational and quasi-experimental designs.
Although a few randomized control trials for short-term musical or drama training programs have been conducted (and they have typically found positive causal effects), most of the published research that has been conducted is correlational or quasi-experimental in nature. This means researchers compare different groups of naturally self-selected children to see which group does better on a particular outcome.
Like a correlational study, researchers who use quasi-experimental design compare naturally occurring groups but they can also take into account selection effects in a way that experimental and correlational research do not. Quasi-experimental studies acknowledge the pre-existing differences in child background and can appropriately compare children in group A to similar children in group B. Essentially, this design can identify the apples and oranges in group A and compare them to apples and oranges in group B, respectively. By doing this we can be more confident that the results found are due to actual group differences in exposure to the arts and not background differences.
So our question remains – does arts engagement help kids do better socially, emotionally, and academically, or are children who already have better social, emotional, and academic skills the ones who end up in arts activities? It is similar to the, “What comes first? – the chicken or the egg” – question. It is clear that one can’t just note that arts-exposed kids do better on average than non-exposed kids on some measure and then conclude that the arts caused the better outcomes. There is a fairly good chance that the arts-exposed kids were already better off on that measure than others even before the arts exposure began. Thus, researchers need to know more about all selection factors, or ways that children who are and are not exposed to the arts are different before the arts exposure. This is where long-term longitudinal studies are helpful where earlier skills and early experiences can be measured and known about before exposure to said middle school arts classes. Once these differences are identified, researchers can statistically control for their effects when making comparisons on later outcomes for those who are and are not exposed to the arts. This is what we accomplished in our recent large-scale, longitudinal study and the results of that study will be included in part 3 of this 3-part blog series.