What Does Camp Do for Kids? Appendix F

INTRODUCTION

   This key is designed to guide coders through the processes of rating the methodologies of the population of studies, and for extracting pertinent data from those studies.

   To limit any potential bias, coders are kept blind to the hypotheses of the meta-analysis as well as those of the individual studies being analyzed. Blinding is accomplished by supplying coders with the sections of the studies that deal with the methodology and data collection of the studies. Given the varied writing styles of the different authors it is impossible to achieve 100% blinding. Because of the blinding goal and the impossibility of achieving a 100% blinding of the coders, coders are asked to impose self-blinding. Self-blinding implies ignoring information about hypothesis and outcomes while in the process of extracting data. In support of the blinding goal, only that information that was used in the primary coding has been supplied. Coders should make every effort to complete all requests for information to the best of their ability, utilizing the coding key.

   Once the verification coding is completed, a discussion meeting with verification coders and the primary coder will determine the level of agreement between all coders. This meeting will generate any needed revision of the coding sheet and key. The need for revision will be based on a level of 90% agreement on all aspects of the coding. If coding does not attain that level then those aspects that are not agreed upon will be re-coded. For disagreements above the 90% level, the majority opinion will be used as the final coding measure. 

Training for Verification Coders - Thursday, 18 February 1999.
   Training will consist of 4 phases:
   . Coders will be given the coding key to review the content, prior to the first training
      meeting.
   . A training meeting will be convened to review the content of the coding key and to
      answer any questions about the coding process.
   . A practice coding session will give the verification coders the opportunity to individually
      code a study. These codings will then be compared to the primary coding of the 
      same study.
   . Based on the practice coding verification, the coding process will be amended as 
      required.

Verification Coding - Saturday, 20 Feb. to Saturday, 27 Feb. 1999.
   Following the training, coders will code a random sample of 6 studies drawn from the total population of studies (N=20) identified and gathered for the meta-analysis. The coding key will be used to code the sample of studies. It will not be possible to code data in the order that categories appear on the coding sheet. Coders are encouraged to use diligence in extracting information.

Verification Consultation with the Primary Coder - 1 or 2 March 1999.
   A discussion of the coding results will be held between the primary and verification coders. The percentage level of agreement will be determined. In the event of a less than 90% agreement level among coders, adjustments will be made to coding protocols and the relevant aspects of all studies will be re-coded by the primary coder. If the level of coding agreement is less than 75%, the primary and verification coding processes will be repeated.

Coding Materials
   Verification coders will be supplied with 7 studies that represent a randomly chosen sample from the population of studies identified as meeting the delimitations of the meta-analysis. One study will be used for a practice coding during training. The remaining 6 studies are to be coded without discussion between the verification coders, or with the primary coder. Each study in the sample will contain only that part of the study that is relevant to the coding verification. A coding key and coding sheets will also be supplied.

Coding Objectives
   Primary: To rate the methodology of each study, so that a mean rater score for the studies can be established. Fulfilled through the use of the following coding key and a coding sheet. This score will then be used in weighting each study as one aspect of the meta-analysis.
Secondary: To confirm extraction of pertinent data for each study, as outlined later in this document. This data will then be used to explore any correlations in the analysis.

Coding Environment
   Coders are encouraged to allot adequate time to code each study. This will prevent interruption. Coders are encouraged to start over on any study coding that is interrupted. Any attempt to meet a deadline or to code within a specific period of time should also be avoided. The environment should allow for adequate concentration so that a standard of thoroughness can be maintained.

Coding Procedure

Order of Procedures
   The following steps were taken in the coding of data from the studies:
   1) Pilot code twenty of sixty identified studies.
   2) Revise coding sheet.
   3) Expert review and revision of coding.
   4) Revise Coding and Develop coding key.
   5) Re-code the method components of the population.
   6) Coding by Verification Panel.
   7) Verification meeting.
   8) Revise code key.
   9) Expert review of code sheet and code key, necessary revisions and re-coding.
 10) Extract statistical data and determine Effect Size.
 11) Analysis of data.

Coding and Blinding Procedures
   Coders were exposed only to the relevant information on methodology and data that were available from each study. Only after the final rating of methodology, were all aspects of each study in the population included in the analysis. It was inevitable that 100% blindness to hypotheses and conclusions could not be achieved. Coders were advised to ignore information on these aspects of a study if they were exposed to it during their coding of the studies.

Interpretation of Studies in Order to Extract Data
   The key components of an interpretation judgement on study quality should be:

   1) Is there documentation to determine that there were controls in place for the given issue? An interpretation based on inference is not considered valid.

   2) Is the control measure documented for the instrument of interest? The literature examined should specifically state the use of a procedure for an instrument. It is not valid to infer that because the use of a measure is evident for one instrument that that step was taken for all instruments.

CODING SHEET

Circle the appropriate answer or fill in the required information.
I) Research Design:              Design:________________________________
Experimental                    Quasi-experimental                    Pre-experimental
[All 6]                      [Non- Random or No Control Group]         [Missing 2 or more]

1) Statistical Equivalence      2) At Least 2 groups      3) Manipulates Independent Variable (Treatment)

4) Measures Dependent Variable       5) Inferential Statistics      6) Control Extraneous Influences
    (Outcome)

Longitudinal: # of years _______

II) Internal Validity: (based on Wolfe, 1984; McMillan, & Schumacher, 1997)
degree of experimenter blindness:
1            2            3                4                   5               6              7             8              9
| NO No/Recognized Risky |      documentation to determine:
                     Experimenter Influence,    Assistant Influence,    Researchers Characteristics
                     Observer Blinding,    Subject Blinding,    Need to Know Basis

randomization:
1           2            3                4                   5                6             7              8             9
                                              |                    |                 |              |               |              |
| NO No/Recognized Risky |               non-random,                  |                   cluster,             |            prop. stratified
                                                                                   systematic,                    non-prop.stratified                             simple,

sample size:  
1           2            3                4                   5                6             7             8             9
| NO No/Recognized Risky | documentation to determine:
                                               m in size  >  experiment 30/survey 100 ,
Instrument Reliability,   Population validity,    Ecological validity,    Hawthorne Effect,   No Maturation/History

controls for recording of errors or cheating:
1            2            3              4                  5                   6             7             8             9
| NO No/Recognized Risky | documentation to determine:
           training,   practice,    pretrial,   adequate resources,   controls for cheating,   inter-rater reliability
                                                                                                    IRR score _______

publication bias: (source, review process, type)
1            2            3              4                  5                   6             7              8            9
                                             |                   |                    |              |               |             |
| NO No/Recognized Risky|            unpublished,       credible source       journal article           thesis,       dissertation       juried journal
                
                                              Mean Coder Score: ___________________________

III) Subject Age Range:        9 to 16        within        85%,      ________________
    
        out of range                                      9 to 16     9 to 16      reported range   

        Human Subjects Protocols:   Compliance        Non-Compliance

IV) Construct Measured:
       Self-:    Concept        Confidence        Esteem        Global Self-Worth

       Instrument used:__________________________________________________
       Instrument Rating:_________________________________________________

       When Measured:                Before              After        Follow-up
                                    1st day               Last Day
                                          1st week          Last Week        1x During

       Type of dependent variable:    self-reporting observed

V) Length of Stay
               in weeks: 1      2      3      4      5      6      7      8      9      10

     Pre-requisite number of years (if any):__________ # of Sessions_______
     Percentage of returning campers:_____________ # of Camps________

VI) Camp Type: ____________________________________________________
__________________________________________________________________

      Resident    Program Goals/Philosophy:_________________________________
      Day            ______________________________________________________
                        ______________________________________________________

                    Coed             Boys             Girls

VII)Demographics: Cultural Distinction______ Socio-Economic _________

VIII) Source:    juried journal        dissertation        thesis
                                   journal             unpublished          "home"

   Funding: self        sponsored        disinterested 3rd party        not evident

   Research Sponsored by:_____________________________________________

IX) Other outcomes studied:___________________________________________

X) Statistical Information:
N__________________    n-    Experimental_____________      n-Control________________
M_______ F________ treatment: __________________________

                             Calculated             Data to Calculate             Reported
                           pre | post change                                   change pre | post

   Std. Dev. (S)     ___|___   ____             yes    no       ____  ___|___
   Mean (X-bar)    ___|___   ____             yes    no       ____  ___|___
   F                       ___|___   ____             yes    no       ____  ___|___
   Probability (p)   ___|___   ____             yes    no       ____  ___|___    
   Type II (beta)   ___|___   ____             yes    no       ____  ___|___
   t                        ___|___   ____             yes    no       ____  ___|___
   df                      ___|___   ____              yes    no     ____   ___|___ 
   SS                    ___|___    ____              yes    no      ____  ___|___
   Z                      ___|___   ____              yes    no      ____   ___|___
   Significance (  ) ___|___   ____              yes    no       ____  ___|___
   Hedge's g         ___|___   ____              yes    no       ____  ___|___
   Pearson r         ___|___   ____               yes   no       ____  ___|___
Fisher's Zr           ___|___   ____               yes   no       ____  ___|___

formula used:
calculations: 

CODING KEY

REVIEW OF THIS KEY SHOULD BE DONE USING A CODING SHEET

I) Research Design
     The type of experimental design is determined by evaluating the following factors:

   1) Statistical Equivalence: generally achieved through random sampling and equivalent
       group size.
   2) At Least 2 groups were compared, one being a control group.
   3) Manipulation of an Independent Variable(s): a Treatment. Measurement of some
       outcome/dependent variable before and after a camp session establishes the camp
       session as a treatment. Because camp is a treatment, history and maturation are not
       relevant threats to internal validity while a resident camp is in session. Extended day
       camp exposure and post-post evaluations are threatened by maturation and history
       issues.
   4) Measurement of a Dependent Variable(s): an Outcome.
   5) Inferential Statistics: used in the analysis of data.
   6) Control of Extraneous Influences: such as isolation in a camp setting.

Action Required: Check these conditions off on the coding sheet as you identify the characteristics in the study.

   A True Experimental design contains all 6 components. A Quasi-experimental design is either non-random or has no control group. A Pre-experimental design is missing 2 or more aspects. Summary reminders are included under the design types on the coding sheet.

Other notations:
Design: indicate the type of characteristics that describe the study,
             eg: Pre-test/Post-test, Pre/Post/Post....

   OR     the type of design if the study is NOT experimental, if known.
             eg: Descriptive

Longitudinal: # of years _______: indicate the number of years over which the
                      study took place.

   For the purposes of this study a period of less than 12 months is considered to be a time series study, but not a longitudinal study.

II) Internal Validity:
   This rating scale uses a 9 point Likert scale to determine the "quality" of the study. The study needs to be interpreted to establish the individual categories listed below.

Action Required: As characteristics of the study are identified, cross them off of the check list that is below each category on the coding sheet.
degree of experimenter blindness:

There is documentation to determine:

Experimenter Influence: The researcher avoided actions that might influence the subjects.

Assistant Influence: Assistants or observers avoided actions that might influence the subjects.

Researchers Characteristics: The researcher's identity/dress/manner did not appear to be an influence on the subjects.

Observer Blinding: Observers were blind to the research question and hypothesis.

Subject Blinding: Subjects were blind to the research question, and hypothesis.

Need to Know Basis: Information about the study and its purpose was shared with assistants, observers, counselors and campers only on a need to know basis.

Action Required: To determine a final score cross off the listed components as they are identified, sum and add 3, circle the appropriate score. If no control measures have been checked for the issue refer to the "For Inadequate Validity Measures" scale below.

randomization:
There is documentation to determine:

Non-random: selection of subjects was not random.

Systematic: subjects are selected as every nth individual from the population. Use of a list is preferred.

Cluster: groups are drawn from the population and subjects are randomly selected from the groups.

Non-proportional stratified: population is divided in sub-groups based on an identified characteristic. The same number of subjects is randomly selected from each group.

Proportional stratified: population is divided in sub-groups based on an identified characteristic. A percentage of subjects are randomly chosen from each sub-group based on that sub-groups percentage representation in the population.

Simple: subjects are chosen from the population in such a way that each member of the population has an equal chance of being selected.

Action Required: To calculate a final score, identify method used and circle the corresponding score. If no conditions are met, refer to the "For Inadequate Validity Measures" scale below.

sample size:
There is documentation to determine:

Minimum sizes: Experiment & control groups have at least 30 subjects, survey instruments
            should have at least 100 subjects.

Instrument Reliability: Instrument reliability should be reported in the description of the
            instrument. Reliability information is coded elsewhere.

Population validity: The subject characteristics can be generalized to the population.

Ecological validity: The research results can be generalized to similar conditions that are
            "normal" in the population. Excluding the Hawthorne effect.

Hawthorne Effect: The potential for subjects to act differently because of the research was
            minimized.

No Maturation or History:
            Maturation is the change in a subject that can occur over time, as part of natural
            development processes. Studies which take follow-up measurements or are
            conducted over several months are susceptible.

            History refers to incidents occurring during the research that may effect the results.
            Periods of long treatment are susceptible to this influence.

   A resident camp session is interpreted as being a continuous period of treatment that is uniquely shaped by the experience of the time period. Therefore, there is not considered to be a threat of maturation or history in studies where an instrument measures change over some portion of a resident camp session. Post-post measurement designs and extended day camp treatments may be subject to these threats to validity.

Action Required: To determine a final score, cross off the listed components as they are identified, sum and add 3, circle the appropriate score. If no control measures have been checked for the issue refer to the "For Inadequate Validity Measures" scale below.

controls for recording of errors or cheating:
There is documentation to determine:

Training: was given to anyone assisting in the research, or the researcher referred to the
              use of the instruments manual.

Practice: with the research protocols took place, prior to recording the data used.

Pretrial: was used with assistants prior to the instruments administration.
             This is NOT a pilot test.

Adequate resources: were available for the data recording process, including characteristics
            such as: adequate time, an unobtrusive location, and/or an environment that
            produced minimal anxiety in the subjects.

Controls for cheating: were in place and described.
            eg: the instrument was used in an environment that was supervised by the
            experimenter or an assistant. The home environment would not meet this criteria.

Inter-rater reliability: was measured and reported.
            Record this measure in the space provided next to this condition on the coding
            sheet.

Action Required: To determine a final score cross off the listed components as they are identified, sum and add 3, circle the appropriate score. If no control measures have been checked for the issue refer to the "For Inadequate Validity Measures" scale below.

publication bias:
There is documentation to determine:

Unpublished: the work is unpublished, and does not fit any of the other publication
             categories.

Credible source: the work is unpublished, and can be located using a credible source,
             eg: ERIC.

Journal article: the work is reported in a journal article and was not subject to peer review.

Thesis: the work is a masters thesis, located through Dissertation Abstracts Internationa
            (DAI).

Dissertation: the work is a dissertation and was located through DAI.

Juried journal: the work is published in a journal that is subject to peer review, also known
            as a refereed journal.

Action Required: To calculate a final score, identify method used and circle the corresponding score. If no conditions are met, refer to the "For Inadequate Validity Measures" scale below.

For Inadequate Validity Measures: apply one of the following ratings.
1: the measure does not apply , was overlooked, or can not be inferred.

   It was assumed that studies published as dissertations and in juried journals could be considered to be minimally credible for all aspects of validity. Given the nature of both of these types of publications, the use of referees provides minimal assurance that the study is not completely worthless. Therefore, these types of studies were not rated with a score below a 2.

2: the control measure was not addressed, and the failure to control was recognized.

3: the methodology for this measure appears adequate or can not be identified from the documentation but there is reason to infer that control steps may have been taken.

Before a study is rated inadequate, and after all other coding, the section that presents the study will be examined to determine if the validity issue is addressed outside of the methodology and data sections. Re-coding will make amendments to the study's mean rater score.

III) Subject Age Range:
   Circle the appropriate category AND write the actual range in the "reported range" field.

   9 to 16: Subjects range in age from 9 to 16 years old.

   Within 9 to 16: Subject age range falls within, but does not completely cover the 9 to 16
            age range.

   85% 9 to 16: 85% of subjects are within the age range, and not more than 15% fall
            outside the age range. Please, Calculate the percentage.

   Outside of range: The subjects studies do not fall into any of the 3 categories above.

   Reported range: Write the range that the study reports in this field.

Notations: Use "years . months" to indicate data that is more exact than just year.

Human Subjects:
   Compliance: Was the permission of parents sought? Were the subjects given the
   opportunity to withdraw at any time without giving reason?

   Non-Compliance: There is blatant evidence that human subjects protocols were violated.

IV) Construct Measured:
   Circle the name of the construct as designated by the study's author. Take this information from the tables and figures provided and not the name of the instrument used. Write in any construct name that is not on the coding list,
eg: self-perception or self-image.

Instrument used: Write the name of the instrument used to establish the self construct
   score, including author name and year of publication, if available. Write the names of
   instruments that do not measure a "self" construct in section IX.

Instrument rating: Write test-retest reliability scores, validity information and alpha
   coefficients that reflect the instrument validity. Note the type of score.

When Measured: circle the appropriate indicator.
   Before and After refer to the 2 weeks before and the week after camp.

   Follow-up is any measurements taken after an "After" measure.

   1st Day reflects either the day of subject arrival or the first full day of the
   camp program. Also use when the term "beginning of camp" is used as the only
   description for the pre-test time.

   Last Day reflects the last full day of camp or the departure day. Also use when the term
   "end of camp" is used as the only description for the post-test time.

   1st Week reflects a measurement taken during the first week, after the 1st day, as
   defined above.

   Last Week reflects a measurement taken during the last week, before the last day, as
  defined above.

   1x During reflects a measurement taken during the camp session. The measurement was
   not taken during any of the periods defined above. Note which week of the session the
   measure was taken.

Notations: for multiple measurements or measurements that do not fit the above time definitions, locate the approximate definition and write the pertinent information within the circle that you draw around that definition.

   Eg: If measurements were taken 1, 3 and 6 months after the study. Then you would circle
   "Follow-up" and write 1 mo., 3 mo. and 6 mo. within the circle.

Type of dependent variable:

   Code the data based on the instrument that measures the construct of interest. Do not assume that a step referred to in the use of one instrument was used in the administration of the instrument that measured the self construct.

   Self-reporting indicates that the subject supplied the information by filling out a form, or
            using some other means.

   Observed indicates that an observer supplied the information about the subject.

V) Length of Stay:

   Circle the number of weeks applicable, for the purpose of this study a week can be considered any period from 5 to 8 days. The scale goes from 1 week to 10 weeks. Studies that focus on a period of time shorter than 1 week are not eligible for inclusion. In the case where the length of the study is shorter than the length of time spent in the camp, report the length of the study, the shorter time period.

Fill in any information that is available for:

   Pre-requisite number of years at the camp, if any were required to participate in the study.

   The Percentage of returning campers, if the information is available.

   The Number of Sessions studied, eg: 4, one week sessions.

   The Number of Camps where the study took place.

VI) Camp Type:
   Write the type of the camp down, eg: Sports, Science, Hearing Impaired, Type of Culture such as jewish, or type of agency such as YMCA.

   Circle Resident if the campers regularly spend the night at the camp or participating in a camp activity.

   Circle Day if the campers return home most nights, or leave the supervision of the camp most nights.

   Write the description of the camping program goals or philosophy, or both, if stated.

   Circle the appropriate indicator of the gender of the camp population:

   Coed camps serve boys and girls.

   Boys for an all boys camp.             Girls for an all girls camp.

VII) Demographics:
Cultural Distinction: Write in the following codes to describe the ethnicity of the subjects:

            AA = African American
            H = Hispanic
            C = Caucasians
            J = Jewish

   Others - write out the name of the ethnicity if it is not listed above.

   If not indicated, leave blank .

Socio-Economic: Write in the following codes to describe the socio-economic status of the subjects.

            L = Low economic or disadvantaged economic status.
            MC = Middle Class
            UMC = Upper Middle Class

   Leave Blank if not indicated.

VIII) Source:
Circle the type of research that you are reading, if you can determine it from the information provided.

Funding: Fill in the source of funding for the study, if the information is available.

IX) Other Outcomes Studied:
   Write down the other outcomes studied in the research. Include the name of the instruments used and a description of what those instruments were designed to measure. If instrument names are not available, write a brief description of the issue that was being studied.

X) Statistical Information:
N is the number of subjects included in the post-test or final set of measurements reported for the sample. The sample size.

M the number of males in the sample.

F the number of females in the sample.

n-Experimental: record the number of subjects in the group that receives the treatment, if a control group was used.

n-Control: record the number of subjects in the control group, if one was used.

Treatment: Describe the independent variable/treatment. In other words, what were the subjects exposed to in order to cause the change in the dependent variable being measured. This may be as simple as writing the word "camp", if going to camp is the only discernable treatment.

X) Statistical Information: continued
   Write the information for the various statistics listed below. Use the appropriate columns for pre-test and post-test data. Use the appropriate columns for data that is reported in the study, or data that you calculated from data available in the study.

   Circle yes or no as the appropriate response to the question of whether or not there is sufficient raw data to calculate the measure.

formula used:
Calculate the following: Hedge's g, Pearson r, & Fisher's Z transformation. Write the equation(s) used, from the equation list, in the formula used field on the coding sheet. Make note of any calculations in the calculations field.

Coding Key Conversion Formulas
(Adobe Acrobat Reader is needed to view this file)

Tags: