The Unexpected Upside of Evaluation: How Your Program May Improve, Even Before Results Are In

March 26, 2018
Dave Brown, LCSW

Perhaps like all camp directors, I don’t need data to believe in the power of camp.  As a camper, I experienced it. As a director, I see it every summer.  But as a social worker, I also believe that it’s critical to show evidence beyond personal experience and anecdotes. 

We want to have real data to demonstrate the power of camp.  Not just to have “proof” for current camp families, or a marketing piece for prospective families, but to move toward demonstrating that camp is a vital, educational experience, that should be accessible to all children. 

So in 2015, our camp partnered with Challenge Success, an education research organization affiliated with the Graduate School of Education at Stanford University, to conduct a multi-year evaluation of our program. To start, Challenge Success guided us through the development of a logic model to clearly identify the short-term and long-term benefits, or outcomes, we believe our campers gain by coming through our program.  For us, these short-term outcomes include social and emotional learning (SEL) skills - empathy and perspective taking, initiative and perseverance, self-awareness, problem solving, and collaboration skills. Ultimately, we believe these lead to our long-term outcome, campers who grow up to be successful in school, in their jobs, and in their relationships.  After two summers of data collection, final results of our study will be ready later this year. 

So then why are we writing about this experience before the results are in?  Because another one of social work’s tenants has unexpectedly come to light:  the power of the process.  Regardless of the final results of the study, our program has already improved significantly through the very process of developing and executing the evaluation.

Our logic model has been key to that process.  Existing research says that if children have opportunities to practice SEL skills, they are developing core competencies that are essential for long-term success in school, jobs, and relationships.  Understanding this has allowed us to stop worrying so much about measuring how much individual campers change over the course of their stay at camp, and instead focus on measuring camper opportunities to practice and develop SEL skills provided by our program.

By shifting our focus in this way, we immediately saw opportunities to adjust our counselor training curriculum and became more intentional about creating effective learning environments for our campers.  Regardless of the activity (archery, arts and crafts, sailing) we started putting more emphasis on being intentional about several small (but important) things:

Basic Needs — Research shows that kids learn best when they feel a strong sense of belonging and physical and emotional well-being.  We’ve always placed high importance on a sense of inclusion and belonging, but are now training counselors to focus on more nuanced aspects of a “safe” environment such as feeling welcomed and receiving clear instructions. We examined our bedtimes to make sure campers were getting the recommended hours of sleep, and even simple counselor responsibilities like offering water and bathroom breaks in between activities improved once we became more intentional about meeting campers’ needs.

Engagement — Research shows that “voice and choice” are critical to establishing an engaging learning environment.  Our campers have always had the opportunity to choose their own activities, and we’ve always wanted counselors to actively engage with all of their campers.  But, we began to examine the extent to which campers have opportunities for making choices within individual activities, and how counselors encourage kids to voice suggestions and feedback before and during those activity periods.  Now, as we develop new activities for camp, “voice and choice” have become guiding tenants for how they are structured and run. 

Reflection — Research shows that reflection supports the development of problem-solving skills and internalizes and strengthens new learning.  We began to provide counselors with training on how to build reflection into their activities.  Counselors began prompting kids to identify what they learned, how it might relate to other parts of their lives, or even just how the activity period could have been better for them and their group.

Our counselors seem to have benefitted as well.  Having a clear logic and structure behind our training, a better understanding of their role to intentionally develop campers’ social and emotional skills, and an easy way to measure how well these strategies are being implemented has also helped them to be more engaged and energized.  Our goals, and our methods of achieving them, are more clearly defined, which I think makes the job feel more rewarding for the staff.

For our Leadership Team, the process of evaluating activities and other camp experiences according to specific criteria gave us much more powerful tools for assessing quality, and a more direct focus on the elements of camp that support learning and the development of social and emotional skills.

Engaging in evaluation doesn’t have to be big and expensive, but it does require resources, “buy in” from staff and families, and a willingness to honestly reflect on and improve your programs. If you are already confident in your outcomes, and not worried about having evidence to demonstrate them, it can be easy to discount the value.

But consider how the process of undertaking an evaluation might improve your program and promote positive camper outcomes, regardless of the data that emerges.  Simply by being more intentional about assessing your program, and using existing research to sharpen your focus onto the things that matter most, you may be surprised by just how easy it is to make meaningful changes.  Changes that may ultimately strengthen the claim that camp is a vital, educational experience, that should be accessible to all children.

Dave Brown is a licensed clinical social worker who works year-round as a director for Mountain Camp, an independent, resident camp near Lake Tahoe, California.  He practiced social work with children and families in non-profits and public schools before joining Mountain Camp full time in 2010.  He lives with his wife and two young sons in the Bay Area.

Photo courtesy of Mountain Camp

Thanks to our research partner, Redwoods.

Redwoods

Additional thanks goes to our research supporter, Chaco

Add comment