Refining Survey Instruments: Amplifying the Youth Voice

January 6, 2020
Katie Wheatley

Helping Kids Change Their Stories — A Taste of Camp Measurement North of the Border is a short blog series that explores Tim Hortons Foundation Camps’ summer and school programs and their approach to impact measurement and camp research. The series outlines Tim Hortons Foundation Camps’ research journey from a practitioner perspective, while highlighting challenges and early learnings in their research process.

Through our ongoing efforts to inform and continually improve, we’re building a multi-faceted approach to collecting camper impact data and feedback. In addition to more formal measurement practices outlined in the previous post, we’re working to incorporate the voice and perspectives of our campers in both our measurement strategy and overall organizational priorities. When we made our programmatic shift in 2017, we recognized that despite being a youth serving organization, we were run by adults and lacked the youth voice in our decision making processes. From programs to policies and procedures, the voice of our most crucial stakeholder was buried. Our outcomes framework and accompanying measurement instruments were created by internal and external professionals — and although this ensures the credibility of our approach, we soon realized the opportunity to uncover the youth voice through involving campers in our program design and the instruments used to measure them.

One way we began to magnify the youth voice was through the creation of our first-ever Youth Advisory Council. Formed in January 2018, the Tim Hortons Foundation Camps Youth Advisory Council is composed of 16 members from across Canada and the United States; ranging in age from 16–21. Members on the council are a combination of current campers, recent graduates and campers turned staff. The council’s mandate is to amplify the youth voice across the organization to influence and inform programmatic and strategic decisions. In April 2019, our Program Innovation Team facilitated a workshop with the council to dive in and dissect our current survey instruments to increase camper comprehension and survey completion.

The 90-minute workshop began by explaining our outcomes framework and the goals of the impact study. We asked for their help in addressing challenges related to question comprehension and survey length to ensure we are asking the right questions in the right way. The council was divided into four small groups to explore existing survey questions. When in their groups, the council was tasked to read the question, summarize what they thought the question was asking, and re-write the question in their own words. The council was not provided with the specific indicator each question measures to avoid influencing their feedback. Of the 75 questions on the original survey, 50 were translated into more youth-friendly language. These edits were implemented in the 2019 summer survey instrument and we saw an 85 percent completion rate, coupled with more response consistency suggesting an overall increase in comprehension.

Our work with the council did not stop there. In October 2019, we shared early and emerging themes from the 2019 summer data. We asked for their reactions to the data — particularly in outcome areas where our campers were indicating need for support. For example, it was surprising for staff to learn that only 60 percent of campers surveyed reported growth in their ability to make positive connections; however, members of the Youth Advisory Council were not surprised. Members suggested this is an area of need because of the stark contrast between the community and connection experienced at camp compared to elements of loneliness and isolation upon returning home. The Council indicated that generally campers do not have control over the adults in their lives or the communities in which they live making it challenging to feel empowered to make positive connections outside of their camp experience. This dialogue has reinforced the need to focus on how we can better support our youth between summers and upon completion of the program. Sharing the results with the Youth Advisory Council provided unique insight into understanding the why behind survey results and how to make effective programmatic and organizational changes to better meet the needs of youth in our programs. 

Katie Wheatley, director of program innovation with Tim Hortons Foundation Camps, specializes in data-driven curriculum design for youth who thrive in alternative learning environments. With a background in special education, Katie combines her passion for education and experience in camping to lead the innovation, measurement and development of Tim Hortons Foundation Camps programs in Canada and the US. For more information about Tim Hortons Foundation Camps, please visit timhortons.com/foundation.

Photo courtesy of Katie Wheatley

Thanks to our research partner, Redwoods.

Redwoods

Additional thanks goes to our research supporter, Chaco.