By Wolfgang Ilg
Group travel is among the most attractive activities of youth work. In Germany, it is very common for groups of ten to thirty teenagers to participate in two-week youth travel camps led by volunteers or professional youth workers during summer vacation. Youth exchange programs are additional attractions of youth travel; for example, the meeting of groups of young people from different countries at a youth hostel. But, these types of youth programs require careful evaluation. Educational challenges, responsibilities, possibilities, risks, and the amount of money exceed those in other fields of youth work by far. The ability to carry out an appropriate evaluation offers many advantages. Internally, it supports the development of a clearer and stronger profile. Externally, it facilitates a self-confident depiction of pedagogic chances within the working field. Additionally, the establishment of an "evaluation-culture" can contribute to the quality of discussion within a whole organization, which offers supervisors the chance of a goaloriented development of their work.
But, appropriate evaluation instruments were not available for a long time. Many teen travel practitioners worked with "self-made" questionnaires, which often had to be evaluated through tallies. On the other hand, current academic research only shows intricate studies of a few selected programs, which are not always transferable to a wider context. Consequently, a cooperation of academics and practitioners initiated the development of standardized and widely applicable questionnaires for the evaluation of youth camps. The studies resulted in a standardized evaluation-procedure that can now be used by anyone who leads youth travel groups. The CD-ROM, Jugend und Europa (Youth and Europe), (sponsored by the Federal Agency for Civic Education in Germany) contains all questionnaires, tables for the comparison of the results, and an easy-to-use computer program called "GrafStat" for statistical analyses. To avoid fears of external control, the procedure can be used by the organizations independently. Afterward, they can voluntarily e-mail their data to the project team, where all data is anonymously collated for a continuous evaluation.
Methodology of the Standardized Evaluation
The CD contains a series of questionnaires, which can be applied to different program concepts. The language within the questionnaires is easy to understand for young people ages thirteen and up. The questionnaires consist of four pages. They are designed as protected Microsoft Word documents that can be complemented with further items. These can either be chosen from a series of scientifically tested items ("item pool") or supplemented with individual questions.
A questionnaire for members of staff (referring to the team's goals) has to be filled in by the whole team either before or during the first days of the program. The questionnaire for participants should be completed by the day before the last day of the camp program. The completion of the form takes about twenty minutes. Alternatively, "GrafStat" also offers the opportunity for a "screen interview," where participants answer their questions directly on a computer.
The answers from all questionnaires, both staff and participants, are entered into a template, which is provided by the software, and the rest is automatically done by "GrafStat." The user receives data, graphics, and tables with statistics such as mean values for all aspects of the youth camp. The evaluation provides information such as socio-demographic data (e.g., age and type of school); degree of satisfaction (e.g., journey, food, rules); and answers to central statements such as, "We (the campers) had enough opportunities for an active participation within the youth camp."
Advantages of the Evaluation Procedure
Because of the high demand for the CD, a second edition had to be published briefly after the CD first came out. What are the reasons that led to this increased interest among operators of youth travel and international exchange programs?
- Reliable results can only be achieved through a scientifically developed questionnaire. The wording of questions as well as the design of the questionnaires (a predominantly sevenfold ranking scale) had been refined in a long process. Test runs with thousands of participants were carried out and finally confirmed an efficient procedure. The questionnaires contain only questions that have proven to make sense and which have shown stable results over a longer period of time (assured through a follow-up questionnaire in the baseline survey). Thus, a reliable evaluation instrument could be provided.
- The benefit of a questionnaire is strongly linked with important prerequisites: a) participants have to understand the questions and b) they have to be prepared to complete the form. Therefore an additional item within the above mentioned baseline survey was dedicated to the question, of how participants assessed the completion of the questionnaire. Answer: Only 15 percent said that the completion of the questionnaire irritated them. And, 62 percent thought the questionnaires were "okay"; 23 percent said that filling in the questionnaire was fun (N=1415). Therefore, young peoples' disposition to feedback should not be underestimated.
- The interpretation of data is particularly interesting, if further comparable data is available. Data that originates from the baseline survey in 2005 can be used for such comparison. Every organization can compare its own data with the 2005 results. A comparative table is added to the CD Jugend und Europa. The table allows the comparison between one's own results and the results of other camps. This supports independent benchmarking, while the control over the interpretation stays with the camp organizer. The continuous data flow from camp organizers to the project team provides a data-basis for updating the comparison tables.
- An evaluation of data clarifies and helps us appreciate the individuality of different types of programs rather than just assuming that all are equal. According to the principle that a good youth camp/ exchange program is a program that achieves its own goals, the goals of the team will be compared to the statements of the participants in the end. It is particularly interesting to compare different camps with each other as this will show clearly if, and to which degree, the goals of the teams can be found within the participants' feedbacks. As an example, the graph on page 20 depicts this kind of coherence for the subject "political stimulation": each dot represents a different camp within the baseline survey. On the x-axis it shows the intensity of the average employees' goal, "The participants should receive impulses for social/ political themes." An average participants' feedback with regards to the question, "During this camp I dealt with a number of social/ political topics" can be found on the y-axis. The ellipse shaped cloud of dots depicts a high coherence between employees' goals and actual achievements (correlation between goals and achievements r = 0.76): low goals among the members of staff correlate with low results among participants; high goals correlate with high results. Every team decides individually if political discussions are desired. The message is clear; those who want to stimulate political discussions normally succeed in doing so. A similar scientifically recorded influence of predefined pedagogic goals on the effects described by participants is very rare. The evaluation of youth camps/international exchange programs can open supervisors' eyes: it can contribute to a general recognition of the meaning, which one's own pedagogic work can have with regards to young peoples' experiences during their stay.
The described evaluation system is a free service that is designed to support teams leading youth travel groups. It has become the standard procedure for evaluating youth group travel in Germany, France, and Poland.
The approved procedure is currently being adapted and further developed for several projects in different contexts. The development of new versions in different languages is one of these processes (there is currently only a first test version of the questionnaires available in English). Another step is the development of shorter and simpler questionnaires for children between eight and twelve years of age. The further development of the evaluation tool might also become a starting point for cross-Atlantic exchange on academic research concerning youth travel — there is quite a lot to discover on both continents!
Ilg, W. (2008). Evaluation von Freizeiten und Jugendreisen. Einführung und Ergebnisse zum bundesweiten Standard-Verfahren. Hannover (Germany): aej. This book is only available in German.
Dubiski, J.; Ilg, W. (eds.) (2008). Evaluation Internationaler Jugendbegegnungen. Ein Verfahren zur Auswertung von Begegnungen. Berlin (Germany); Paris (France); Warsaw (Poland). This book is available in German, French, or Polish.
Note: The CD, Jugend und Europa, is enclosed in the books above. It contains all material in German, French, and Polish, as well as a first pilot version of the questionnaires in English.
For further information, visit www.freizeitenevaluation.de .
Wolfgang Ilg, graduate theologian and psychologist, is a research associate at the University of Tuebingen/Germany and works as evaluator and academic author. He has extensive experience both as a full-time and a voluntary member of staff for youth camps. Contact the author at email@example.com .
Originally published in the 2009 July/August issue of Camping Magazine.