Research is a big thing right now. Of course it is — people seek certainty when things are uncertain, as they surely are amidst the COVID-19 pandemic. Research brings certainty, right? Yes, when it is appropriately designed and the findings are statistically significant. Only randomized control trials replicated over time can give us some degree of certainty, but even these rare studies involve human design and interpretation, which, of course, introduces error. And in research, error equals uncertainty.
This makes the demand for research during a pandemic worth discussing. In this blog, I suggest some ways we might think about, participate in, and interpret research in the age of COVID-19. I’ll also share how we are thinking about research at ACA, and what you might look out for in the coming months.
Research and COVID-19
Research today seems to fall into one of two broad categories: research about COVID-19 (and its effects), and research about normal things in the context of COVID-19. Some examples of research about COVID-19 would include studies about the virus itself and studies about outbreaks and why they happen. Research about the normal things in the context of COVID-19 includes studies about childcare centers and camps that are operating and what they are doing to prevent the spread of the virus.
In both categories we think about research in the same way we always do: rigor and generalizability. Whether the study is about the virus or about camp in the context of the virus, we need to first think about how the study was conducted, with whom, by whom, when, and where. Research on the virus is critical, and there’s lots of it; however, some of this research is specific to a place or a population. This limits the extent to which we can generalize beyond that specific place or group of people. The moral of the story here is to think critically about the research you consume, whether about the virus or anything else.
Research that might help us think about camp and communicable disease is a little trickier. First, it is difficult to design and implement rigorous research at camp. ACA’s Impact Study is certainly an example of a rigorous study, but it took years and considerable resources to make it that way. We don’t have that kind of time with COVID-19.
But that’s not to say we can’t do meaningful research; in fact, we have to. Without the time or resources to conduct large-scale experimental research, we use more exploratory methods. These generally come in two flavors. First are descriptive studies that help us understand how camps are responding to COVID-19, typically through qualitative narrative. This kind of study cannot tell us why something did or did not happen (really only randomized control trials can do this), but it can give us a far richer sense of specific people and situations. The second flavor is what I’ll call monitoring: tracking data points, ideally a lot of them, from which to identify patterns. Depending on the amount of data, and the extent to which those data represent a larger population, we might even be able to use these patterns to help us predict to other times or populations.
Camp on Camps and COVID-19
Let’s switch gears and look at some examples at camp. I’m going to use a logic model to map out several key studies related to camp and COVID-19 — so first a quick review. Logic models are visual representations of a program, including the things that go into the program (inputs), what happens with those inputs (activities), who the program serves (outputs), what they gain from the experience (outcomes), and the result of those gains (impact). For a crash course in logic models, check out our previous Research 360 blogs (here, and here, and here). In these posts we’ve emphasized logic models as a way to consider how and what we can evaluate at camp. In short, we can evaluate inputs, activities, outputs, outcomes, and impact — which means the logic model is also a useful tool for ensuring our research on camps and COVID-19 together will give us a complete picture.
Inputs are ingredients that make camp happen (like our facilities, equipment, and operations).
Our COVID-19 research is focused on inputs in two ways. First, we will evaluate camp budget and operations as a part of our regularly scheduled Camp Industry Reports. Typically, this study includes questions related to revenue and expenses, fees, scholarships, and annual spending patterns. This year we’ll include questions specific to COVID-19 to help us understand the impact of the virus on vital camp inputs, which will include the range of camp operations in summer 2020.
The second way we are looking at camp inputs is by assessing how camps implemented the recommended COVID-19 preventive practices, such as those outlined in the Camp Operations Guide Summer 2020. We will look at specific inputs, likely personal protective equipment (PPE), supplies, and how camps that operated trained their staff to implement the recommended practices. Look for this research in early Fall 2020.
Activities are what happens at camp — in a typical year, this likely includes various programs or different activities that are part of a single program.
Evaluating implementation of the Camp Operations Guide is also a way to understand activities. We’ll look at inputs, such as masks and hand sanitizer, and we'll also look at how staff implemented these practices throughout the summer. As we well know, there is often a big difference between policies, how camp staff are trained to implement policies, and what actually happens at camp. And this is most often OK! Except perhaps in the case of communicable disease, especially one that can be prevented by diligent practices. So, a big focus of the study I mentioned above will look specifically at how camps that operated implemented the guide.
A second way we are looking at camp activities as they relate to COVID-19 is by an exploratory study of camps and the various ways they are responding to the pandemic. The ACA research team sent out a request to participate in this study in early June, and from those responses selected 30 camps that indicated different COVID-19 responses: canceling or hibernation mode for Summer 2020, pivoting to virtual programming, and programming as usual. Our first round of data collection revealed numerous variations within these general response categories — like camps that offered family camp or family-resort experiences, and camps that partnered with community organizations to provide food for critical service providers. You can read more about this study in late Fall 2020.
Outputs are generally numbers — in our case, they are numbers of campers served through our programs. Some may include staff as outputs, especially when staff participate in some kind of leadership development program.
One example of ongoing research related to outputs is the work of Professor Emily Oster from Brown University and a team of researchers who are tracking COVID-19 cases in camps. This team is asking camp operators to voluntarily share information about campers and staff on site, and then to report instances of COVID-19 positive cases. You can be a part of their tracking sample beginning with a baseline form here. More information can be found here.
In a logic model, outcomes are typically represented in terms of short-term outcomes and medium- or long-term outcomes. Short-term outcomes are things that happen immediately and are measurable right away, like a new skill. Medium- and long-term outcomes are the ways the short-term outcomes transfer to another setting. At camp, this typically means outcomes that transfer to home or school. One way camp professionals measure outcomes is by using the Youth Outcomes Battery (look out for the newly designed online version coming soon!).
There are two general buckets of outcomes-related research right now. First are the many innovative ways camps that pivoted to virtual programming are evaluating their programs. Evaluating new programs is not easy, but virtual platforms provide opportunities to observe campers in action and collect data that are not as easy to document in traditional face-to-face programs. Camps that are measuring outcomes of virtual experiences are using what they learn to train staff, improve their new programs, and to promote their programs to parents and funders.
We are also exploring camper outcomes through research with campers, parents, and caregivers. In late Spring 2020, we interviewed parents-camper “dyads” (meaning we interviewed them together, as a pair) as a part of our ongoing Impact Study. While most participants said it was too early to know whether or not they would attend camp, they expressed trust in their camp’s decisions and protocols related to COVID-19 and grief over the expected loss of camp and its benefits.
A Fall 2020 study of parents and caregivers will explore these themes and take a more focused look at parents’ perceptions of camper outcomes. Unlike the previous study, which involved parents recruited from ACA-accredited camps, this upcoming study will work with a third-party research firm to recruit parents from the general population — including parents who have never sent a child to camp. Recruiting both camp and noncamp parents will allow us to measure outcomes from new and diverse perspectives, related to COVID-19 and to summertime experiences in general. Look for the results of this study in late Fall 2020.
Impact, in a logic model, represents the end result of the short- and medium- or long-term outcomes. One way to think about impact is how the world will look when your desired outcomes are achieved. Impact is generally big and broad and difficult to measure. A camp logic model might state their impact is a safe and healthy community, justice and equity for all, or environmental sustainability.
While it is too soon to guess what the impact of COVID-19 might be, we are certainly thinking about the impact on camps. Preliminary projections based on our tracking of camps suggest a near 80-percent loss in campers served, camp operations, jobs, and overall economic impact. We estimate more than 16 million children who would have attended camp will not be able to do so in Summer 2020.
These numbers are grim. But they are also an opportunity — sometimes impact is best measured in the absence of a thing, such as, in this case, in the absence of summer camp. We expect the parent study to give us important insight into the outcomes of camp that parents value most, and what happens when those outcomes cannot be achieved. The media is already recognizing this, in pieces like this and this and this, laying the groundwork for a powerful opportunity to position camps as critical experiences for young people.
Research Participants Needed
While research in the time of COVID-19 is challenging, it is also important to our ability to learn about and from this unprecedented time. The following map summarizes what we shared in this blog. As you can see, Fall 2020 will be a busy time for research — please consider participating if you can, and use your critical research lens to interpret and apply what we learn to your unique situation.
Laurie Browne, PhD, is the director of research at ACA. She specializes in ACA's Youth Outcomes Battery and supporting camps in their research and evaluation efforts. Prior to joining ACA, Laurie was an assistant professor in the Department of Recreation, Hospitality, and Parks Management at California State University-Chico. Laurie received her PhD from the University of Utah, where she studied youth development and research methods.
Thanks to our research partner, Redwoods.
Additional thanks goes to our research supporter, Chaco.