Program quality (PQ) is essential to youth programs, and research shows that high-quality programs are directly linked to more positive outcomes.

PQ refers to the environment that is created at camp through staff practices. When that environment is safe, supportive, interactive, and engaging, participants experience greater PQ. Examples of quality staff practices include modifying activities to suit campers’ needs, prompting learning reflection, and supporting teamwork and social-emotional skill development.

Conducting PQ assessments can provide invaluable data to strengthen programs and to help more youth benefit from the positive outcomes camp programs are designed to foster. At Tim Hortons Foundation Camps (Tims Camps), we have dedicated significant time and resources toward our PQ initiative for the past three years, and plan to continue to do so. We understand the direct link between PQ and positive youth outcomes, as well as the value of data as a tool for continuous improvement. We highlight herein the value of PQ assessments and a continuous improvement cycle, share our insights, and provide tangible action steps to support your own PQ journey. We’ll focus on continuous improvement and the first two phases of the improvement cycle: prepare and assess. Part two in this series will detail the final two phases: plan and improve.

Tims Camps

Before we dive into Tim Camps’ approach to PQ, it is important to share some context about our operations and who we are. Our mission is to help youth from underserved communities achieve their full potential. We operate seven camp locations across Canada and the United States and serve youth through three different modes of program delivery: summer, school, and partnerships. We highly value research and measurement, and we approach measurement holistically, measuring not only the outcomes and impact of our programs, but also how we deliver programs using a PQ lens. We prioritize continuous improvement and strive to always be better by using data to inform changes. This belief initiated our PQ journey.

Continuous Improvement

The continuous improvement cycle we adhere to for PQ assessments consists of the following four phases: 

1. Prepare: set your team up for success with PQ assessments through education, training, and planning.
2. Assess: measure PQ through observations and scoring.
3. Plan: use data collected during assessments to plan for future program improvements.
4. Improve: implement action plans to directly and positively impact program quality.

The timing of each phase can vary based on the camp or organizational needs and capacity. For example, it may make sense that the cycle takes place on an annual basis. At Tims Camps, we engage in an organization-wide annual cycle, and each of our camp locations participates in smaller, more frequently occurring cycles. This way, camps can make timely use of data for quicker, real-time improvements.

Prepare

At Tims Camps, we engage staff in various roles to use one research-validated tool and two modified tools rooted in the same research-backed approach. We learned early on that using the full research-validated tool required capacity that seasonal camp staff were unable to provide. This led us to create modified versions of that tool to meet seasonal staff needs. This approach helped overcome barriers regarding the time commitment required for education and training on the research-validated tool and allowed us to better meet our organization’s needs during peak operations.

Educating and training our full-time team on the PQ approach and assessment tools and coaching seasonal staff took a considerable amount of time (approximately 30 hours in the spring of our first year of implementation). Once baseline knowledge has been established, less time will be required each year. Although, given regular staff turnover, it’s important to consider new managers each season and allow time to prepare for any improvements or adaptations that have been made to the assessment approach. Additionally, for a well-rounded understanding of PQ, extra time is required with seasonal staff to train them in what high-quality programs look like in action, how they can deliver them, and their role on the assessment team.

Prepare: Tips and Tricks

The first consideration in the prepare phase is who should be involved. You will need to assemble a team for your PQ initiative that makes sense. Consider your operational capacities for staff and decide:

• Who has the capacity to learn about and execute PQ assessments during peak operations?
• Who is able to put insights from the data into action after assessments? 

The second consideration is what assessment tools make sense for you and how much data should be collected. Some tips are to: 

• Find a research-grounded tool that works for your purposes. If you’re using this work to apply for government and other grants, the research tool may be best.
• Make adaptations to that tool and approach, as appropriate. This will make your PQ framework responsive to your own organizational needs and capacity and increase staff buy-in and engagement.

Lastly, training your team is key to the cycle’s prepare phase:

• Spend time with your full-time staff year-round to help them feel prepared.
• For seasonal staff, weave PQ components throughout your already scheduled training: 
  • Begin with a training session specific to PQ to provide foundational understanding (what PQ is and why it’s important).
  • Make PQ a recurring and experiential topic throughout the rest of your training schedule (e.g., have staff use a modified tool to assess your facilitation during training; have leadership staff intentionally model great and not-so-great facilitation skills to promote conversation about quality practices; give each staff a chance to try and facilitate a small game or activity during training and debrief with a PQ lens).

For many camps, staff capacity is a barrier to the education and training required for PQ assessments. Often, seasonal staff are delivering programs in the spring, preparing camp property, or have start dates close to campers arriving. Realistically, there’s not enough time to engage them in the same training process as full-time staff; however, their input and feedback related to PQ is critical. Making minor modifications to tools and trainings can help you collect more data and engage your entire staff team in the process — but make sure that any modified tools are still rooted in research and align with your overall framework. Including your frontline staff in assessments brings diversity to the data collected, and it comes from the staff who most directly impact the quality of programs.

Assess

Once the team is prepared, the next step is to conduct assessments and gather data. At Tims Camps, we engage leadership staff at each camp location and a centralized support team in assessments using the full research-validated tool. Additionally, we engage at least four staff members with different roles at each camp location to complete assessments using modified tools. We also try to creatively engage our youth in assessments, as appropriate. We use an approach that requires debriefing at each location after every assessment to review the data and identify improvement action steps that can be immediately implemented. On a larger scale, we collect assessment data from all locations over the entire assessment period and compile that to create organization-wide and site-specific PQ scores.

Assess: Tips and Tricks

We recommend determining a framework ahead of your assessment period that addresses:

• Who will be using which tool and how often. Remember, a plan should include a variety of people with different roles at camp to provide a holistic view of PQ at your camp.
• A contingency plan for assessments. What happens if assessments are missed or the assessor is unavailable?
• How scores will be recorded and submitted. Do you prefer paper and pencil or an online data management system?
• How learnings from assessments will be implemented. PQ assessments are wonderful to support staff learning and program development in real time.

Next Steps

Part two of this look at continuous improvement and program quality will dive into the final two phases of the continuous improvement cycle (plan and improve) in more detail. For more information on each of these phases and ACA’s guidance for how to engage in this work, check out the ACA Camp Program Quality webpage. The workbook titled Continuous Improvement: Improving the Youth Experience provides clear and actionable steps to support you in assessing PQ at your camp. 

Stay tuned to learn more about how to review, understand, and make improvements based on PQ data. 

Photos courtesy of Tim Horton Camp Kentahten, Campbellsville, KY.

Lacey Maglinger is the senior manager, Program Quality & Development at Tim Hortons Foundation Camps. She has worked with the organization since 2012 in various roles. Currently, her work includes conducting PQ assessments, supporting camp teams in collecting PQ data through assessments, and using PQ data to improve programming and create resources for camp teams.

Victoria Povilaitis, PhD, is the director, Program Innovation at Tim Hortons Foundation Camps. In this role she guides program quality, outcome, and impact measurement and uses these data to inform program development and facilitation.