|
INTRODUCTION
This key is designed to guide coders
through the processes of rating the methodologies
of the population of studies, and for extracting
pertinent data from those studies.
To limit any potential bias, coders
are kept blind to the hypotheses of the meta-analysis
as well as those of the individual studies being
analyzed. Blinding is accomplished by supplying
coders with the sections of the studies that deal
with the methodology and data collection of the
studies. Given the varied writing styles of the
different authors it is impossible to achieve
100% blinding. Because of the blinding goal and
the impossibility of achieving a 100% blinding
of the coders, coders are asked to impose self-blinding.
Self-blinding implies ignoring information about
hypothesis and outcomes while in the process of
extracting data. In support of the blinding goal,
only that information that was used in the primary
coding has been supplied. Coders should make every
effort to complete all requests for information
to the best of their ability, utilizing the coding
key.
Once the verification coding is
completed, a discussion meeting with verification
coders and the primary coder will determine the
level of agreement between all coders. This meeting
will generate any needed revision of the coding
sheet and key. The need for revision will be based
on a level of 90% agreement on all aspects of
the coding. If coding does not attain that level
then those aspects that are not agreed upon will
be re-coded. For disagreements above the 90% level,
the majority opinion will be used as the final
coding measure.
Training for Verification Coders - Thursday,
18 February 1999.
Training will consist of
4 phases:
. Coders will be given the coding
key to review the content, prior to the first
training
meeting.
. A training meeting will be convened
to review the content of the coding key and to
answer any questions
about the coding process.
. A practice coding session will
give the verification coders the opportunity to
individually
code a study. These
codings will then be compared to the primary coding
of the
same study.
. Based on the practice coding verification,
the coding process will be amended as
required.
Verification Coding - Saturday, 20 Feb. to Saturday,
27 Feb. 1999.
Following the training, coders will
code a random sample of 6 studies drawn from the
total population of studies (N=20) identified
and gathered for the meta-analysis. The coding
key will be used to code the sample of studies.
It will not be possible to code data in the order
that categories appear on the coding sheet. Coders
are encouraged to use diligence in extracting
information.
Verification Consultation with the Primary
Coder - 1 or 2 March 1999.
A discussion of the coding results
will be held between the primary and verification
coders. The percentage level of agreement will
be determined. In the event of a less than 90%
agreement level among coders, adjustments will
be made to coding protocols and the relevant aspects
of all studies will be re-coded by the primary
coder. If the level of coding agreement is less
than 75%, the primary and verification coding
processes will be repeated.
Coding Materials
Verification coders will be supplied
with 7 studies that represent a randomly chosen
sample from the population of studies identified
as meeting the delimitations of the meta-analysis.
One study will be used for a practice coding during
training. The remaining 6 studies are to be coded
without discussion between the verification coders,
or with the primary coder. Each study in the sample
will contain only that part of the study that
is relevant to the coding verification. A coding
key and coding sheets will also be supplied.
Coding Objectives
Primary: To rate the methodology
of each study, so that a mean rater score for
the studies can be established. Fulfilled through
the use of the following coding key and a coding
sheet. This score will then be used in weighting
each study as one aspect of the meta-analysis.
Secondary: To confirm extraction of pertinent
data for each study, as outlined later in this
document. This data will then be used to explore
any correlations in the analysis.
Coding Environment
Coders are encouraged to allot adequate
time to code each study. This will prevent interruption.
Coders are encouraged to start over on any study
coding that is interrupted. Any attempt to meet
a deadline or to code within a specific period
of time should also be avoided. The environment
should allow for adequate concentration so that
a standard of thoroughness can be maintained.
Coding Procedure
Order of Procedures
The following steps were taken in
the coding of data from the studies:
1) Pilot code twenty of sixty identified
studies.
2) Revise coding sheet.
3) Expert review and revision of
coding.
4) Revise Coding and Develop coding
key.
5) Re-code the method components
of the population.
6) Coding by Verification Panel.
7) Verification meeting.
8) Revise code key.
9) Expert review of code sheet and
code key, necessary revisions and re-coding.
10) Extract statistical data and determine
Effect Size.
11) Analysis of data.
Coding and Blinding Procedures
Coders were exposed only to the relevant
information on methodology and data that were
available from each study. Only after the final
rating of methodology, were all aspects of each
study in the population included in the analysis.
It was inevitable that 100% blindness to hypotheses
and conclusions could not be achieved. Coders
were advised to ignore information on these aspects
of a study if they were exposed to it during their
coding of the studies.
Interpretation of Studies in Order to Extract
Data
The key components of an interpretation
judgement on study quality should be:
1) Is there documentation to determine
that there were controls in place for the given
issue? An interpretation based on inference is
not considered valid.
2) Is the control measure documented
for the instrument of interest? The literature
examined should specifically state the use of
a procedure for an instrument. It is not valid
to infer that because the use of a measure is
evident for one instrument that that step was
taken for all instruments.
CODING SHEET
Circle the appropriate answer or fill in the
required information.
I) Research Design:
Design:________________________________
Experimental
Quasi-experimental
Pre-experimental
[All 6]
[Non- Random or No Control Group] [Missing
2 or more]
1) Statistical Equivalence
2) At Least 2 groups
3) Manipulates Independent Variable (Treatment)
4) Measures Dependent Variable
5) Inferential Statistics
6) Control Extraneous Influences
(Outcome)
Longitudinal: # of years _______
II) Internal Validity: (based on
Wolfe, 1984; McMillan, & Schumacher, 1997)
degree of experimenter blindness:
1
2
3
4
5
6
7
8
9
| NO No/Recognized Risky |
documentation to determine:
Experimenter Influence, Assistant
Influence, Researchers Characteristics
Observer Blinding, Subject Blinding,
Need to Know Basis
randomization:
1
2
3
4
5
6
7
8
9
|
|
|
|
|
|
| NO No/Recognized Risky |
non-random,
|
cluster,
|
prop. stratified
systematic,
non-prop.stratified
simple,
sample size:
1
2
3
4
5
6
7
8
9
| NO No/Recognized Risky | documentation to determine:
m in size > experiment 30/survey
100 ,
Instrument Reliability, Population
validity, Ecological validity,
Hawthorne Effect, No Maturation/History
controls for recording of errors or cheating:
1
2
3
4
5
6
7
8
9
| NO No/Recognized Risky | documentation
to determine:
training, practice, pretrial, adequate
resources, controls for cheating, inter-rater
reliability
IRR score _______
publication bias: (source, review process,
type)
1
2
3
4
5
6
7
8
9
|
|
|
|
|
|
| NO No/Recognized Risky|
unpublished,
credible source
journal article
thesis,
dissertation
juried journal
Mean Coder Score: ___________________________
III) Subject Age Range:
9 to 16
within
85%, ________________
out
of range
9 to 16 9 to 16
reported range
Human
Subjects Protocols: Compliance
Non-Compliance
IV) Construct Measured:
Self-:
Concept
Confidence
Esteem
Global Self-Worth
Instrument
used:__________________________________________________
Instrument
Rating:_________________________________________________
When Measured:
Before
After
Follow-up
1st day
Last Day
1st week
Last Week
1x During
Type of
dependent variable: self-reporting
observed
V) Length of Stay
in weeks: 1 2
3 4
5 6
7 8
9 10
Pre-requisite number
of years (if any):__________ # of Sessions_______
Percentage of returning
campers:_____________ # of Camps________
VI) Camp Type: ____________________________________________________
__________________________________________________________________
Resident
Program Goals/Philosophy:_________________________________
Day
______________________________________________________
______________________________________________________
Coed
Boys
Girls
VII)Demographics: Cultural Distinction______
Socio-Economic _________
VIII) Source: juried
journal
dissertation
thesis
journal
unpublished
"home"
Funding: self
sponsored
disinterested 3rd party
not evident
Research Sponsored by:_____________________________________________
IX) Other outcomes studied:___________________________________________
X) Statistical Information:
N__________________ n-
Experimental_____________
n-Control________________
M_______ F________ treatment: __________________________
Calculated
Data to Calculate
Reported
pre | post change
change pre | post
Std. Dev. (S)
___|___ ____
yes no
____ ___|___
Mean (X-bar) ___|___
____ yes
no ____
___|___
F
___|___ ____
yes no
____ ___|___
Probability (p)
___|___ ____
yes no
____ ___|___
Type II (beta)
___|___ ____
yes no
____ ___|___
t
___|___ ____
yes no
____ ___|___
df
___|___ ____
yes no
____ ___|___
SS
___|___ ____
yes no
____ ___|___
Z
___|___ ____
yes no
____ ___|___
Significance ( ) ___|___
____
yes no
____ ___|___
Hedge's g
___|___ ____
yes no
____ ___|___
Pearson r
___|___ ____
yes no
____ ___|___
Fisher's Zr
___|___ ____
yes no
____ ___|___
formula used:
calculations:
CODING KEY
REVIEW OF THIS KEY SHOULD
BE DONE USING A CODING SHEET
I) Research Design
The type of experimental
design is determined by evaluating the following
factors:
1) Statistical Equivalence: generally
achieved through random sampling and equivalent
group size.
2) At Least 2 groups were
compared, one being a control group.
3) Manipulation of an Independent
Variable(s): a Treatment. Measurement of some
outcome/dependent
variable before and after a camp session establishes
the camp
session as
a treatment. Because camp is a treatment, history
and maturation are not
relevant
threats to internal validity while a resident
camp is in session. Extended day
camp exposure
and post-post evaluations are threatened by maturation
and history
issues.
4) Measurement of a Dependent
Variable(s): an Outcome.
5) Inferential Statistics:
used in the analysis of data.
6) Control of Extraneous Influences:
such as isolation in a camp setting.
Action Required: Check these conditions
off on the coding sheet as you identify the characteristics
in the study.
A True Experimental design
contains all 6 components. A Quasi-experimental
design is either non-random or has no control
group. A Pre-experimental design is missing
2 or more aspects. Summary reminders are included
under the design types on the coding sheet.
Other notations:
Design: indicate the type of characteristics
that describe the study,
eg: Pre-test/Post-test, Pre/Post/Post....
OR
the type of design if the study is NOT
experimental, if known.
eg: Descriptive
Longitudinal: # of years _______: indicate
the number of years over which the
study took place.
For the purposes of this study a
period of less than 12 months is considered to
be a time series study, but not a longitudinal
study.
II) Internal Validity:
This rating scale uses a 9 point
Likert scale to determine the "quality"
of the study. The study needs to be interpreted
to establish the individual categories listed
below.
Action Required: As characteristics of
the study are identified, cross them off of the
check list that is below each category on the
coding sheet.
degree of experimenter blindness:
There is documentation to determine:
Experimenter Influence: The researcher
avoided actions that might influence the subjects.
Assistant Influence: Assistants or observers
avoided actions that might influence the subjects.
Researchers Characteristics: The researcher's
identity/dress/manner did not appear to be an
influence on the subjects.
Observer Blinding: Observers were blind
to the research question and hypothesis.
Subject Blinding: Subjects were blind
to the research question, and hypothesis.
Need to Know Basis: Information about
the study and its purpose was shared with assistants,
observers, counselors and campers only on a need
to know basis.
Action Required: To determine a final
score cross off the listed components as they
are identified, sum and add 3, circle the appropriate
score. If no control measures have been checked
for the issue refer to the "For Inadequate
Validity Measures" scale below.
randomization:
There is documentation to determine:
Non-random: selection of subjects was
not random.
Systematic: subjects are selected as every
nth individual from the population. Use of a list
is preferred.
Cluster: groups are drawn from the population
and subjects are randomly selected from the groups.
Non-proportional stratified: population
is divided in sub-groups based on an identified
characteristic. The same number of subjects is
randomly selected from each group.
Proportional stratified: population is
divided in sub-groups based on an identified characteristic.
A percentage of subjects are randomly chosen from
each sub-group based on that sub-groups percentage
representation in the population.
Simple: subjects are chosen from the population
in such a way that each member of the population
has an equal chance of being selected.
Action Required: To calculate a final
score, identify method used and circle the corresponding
score. If no conditions are met, refer to the
"For Inadequate Validity Measures" scale
below.
sample size:
There is documentation to determine:
Minimum sizes: Experiment & control
groups have at least 30 subjects, survey instruments
should have at least 100 subjects.
Instrument Reliability: Instrument reliability
should be reported in the description of the
instrument. Reliability information is coded elsewhere.
Population validity: The subject characteristics
can be generalized to the population.
Ecological validity: The research results
can be generalized to similar conditions that
are
"normal" in the population. Excluding
the Hawthorne effect.
Hawthorne Effect: The potential for subjects
to act differently because of the research was
minimized.
No Maturation or History:
Maturation is the change in a subject that can
occur over time, as part of natural
development processes. Studies which take follow-up
measurements or are
conducted over several months are susceptible.
History refers to incidents occurring during the
research that may effect the results.
Periods of long treatment are susceptible to this
influence.
A resident camp session is interpreted
as being a continuous period of treatment that
is uniquely shaped by the experience of the time
period. Therefore, there is not considered to
be a threat of maturation or history in studies
where an instrument measures change over some
portion of a resident camp session. Post-post
measurement designs and extended day camp treatments
may be subject to these threats to validity.
Action Required: To determine a final
score, cross off the listed components as they
are identified, sum and add 3, circle the appropriate
score. If no control measures have been checked
for the issue refer to the "For Inadequate
Validity Measures" scale below.
controls for recording of errors or cheating:
There is documentation to determine:
Training: was given to anyone assisting
in the research, or the researcher referred to
the
use of the instruments manual.
Practice: with the research protocols
took place, prior to recording the data used.
Pretrial: was used with assistants prior
to the instruments administration.
This is NOT a pilot test.
Adequate resources: were available for
the data recording process, including characteristics
such as: adequate time, an unobtrusive location,
and/or an environment that
produced minimal anxiety in the subjects.
Controls for cheating: were in place and
described.
eg: the instrument was used in an environment
that was supervised by the
experimenter or an assistant. The home environment
would not meet this criteria.
Inter-rater reliability: was measured
and reported.
Record this measure in the space provided
next to this condition on the coding
sheet.
Action Required: To determine a final
score cross off the listed components as they
are identified, sum and add 3, circle the appropriate
score. If no control measures have been checked
for the issue refer to the "For Inadequate
Validity Measures" scale below.
publication bias:
There is documentation to determine:
Unpublished: the work is unpublished,
and does not fit any of the other publication
categories.
Credible source: the work is unpublished,
and can be located using a credible source,
eg: ERIC.
Journal article: the work is reported
in a journal article and was not subject to peer
review.
Thesis: the work is a masters thesis,
located through Dissertation Abstracts Internationa
(DAI).
Dissertation: the work is a dissertation
and was located through DAI.
Juried journal: the work is published
in a journal that is subject to peer review, also
known
as a refereed journal.
Action Required: To calculate a final
score, identify method used and circle the corresponding
score. If no conditions are met, refer to the
"For Inadequate Validity Measures" scale
below.
For Inadequate Validity Measures: apply
one of the following ratings.
1: the measure does not apply , was
overlooked, or can not be inferred.
It was assumed that studies published
as dissertations and in juried journals could
be considered to be minimally credible for all
aspects of validity. Given the nature of both
of these types of publications, the use of referees
provides minimal assurance that the study is not
completely worthless. Therefore, these types of
studies were not rated with a score below a 2.
2: the control measure was not addressed,
and the failure to control was recognized.
3: the methodology for this measure appears
adequate or can not be identified
from the documentation but there is reason to
infer that control steps may have been taken.
Before a study is rated inadequate, and after
all other coding, the section that presents the
study will be examined to determine if the validity
issue is addressed outside of the methodology
and data sections. Re-coding will make amendments
to the study's mean rater score.
III) Subject Age Range:
Circle the appropriate category AND
write the actual range in the "reported range"
field.
9 to 16: Subjects range in
age from 9 to 16 years old.
Within 9 to 16: Subject age
range falls within, but does not completely cover
the 9 to 16
age range.
85% 9 to 16: 85% of subjects
are within the age range, and not more than 15%
fall
outside the age range. Please, Calculate the
percentage.
Outside of range: The subjects
studies do not fall into any of the 3 categories
above.
Reported range: Write the
range that the study reports in this field.
Notations: Use "years . months"
to indicate data that is more exact than just
year.
Human Subjects:
Compliance: Was the permission
of parents sought? Were the subjects given the
opportunity to withdraw at any time
without giving reason?
Non-Compliance: There is
blatant evidence that human subjects protocols
were violated.
IV) Construct Measured:
Circle the name of the construct
as designated by the study's author. Take this
information from the tables and figures provided
and not the name of the instrument used. Write
in any construct name that is not on the coding
list,
eg: self-perception or self-image.
Instrument used: Write the name of the
instrument used to establish the self construct
score, including author name and
year of publication, if available. Write the names
of
instruments that do not measure a
"self" construct in section IX.
Instrument rating: Write test-retest reliability
scores, validity information and alpha
coefficients that reflect the instrument
validity. Note the type of score.
When Measured: circle the appropriate
indicator.
Before and After refer
to the 2 weeks before and the week after camp.
Follow-up is any measurements
taken after an "After" measure.
1st Day reflects either the
day of subject arrival or the first full day of
the
camp program. Also use when the term
"beginning of camp" is used as the only
description for the pre-test time.
Last Day reflects the last
full day of camp or the departure day. Also use
when the term
"end of camp" is used as
the only description for the post-test time.
1st Week reflects a measurement
taken during the first week, after the 1st day,
as
defined above.
Last Week reflects a measurement
taken during the last week, before the last day,
as
defined above.
1x During reflects a measurement
taken during the camp session. The measurement
was
not taken during any of the periods
defined above. Note which week of the session
the
measure was taken.
Notations: for multiple measurements or
measurements that do not fit the above time definitions,
locate the approximate definition and write the
pertinent information within the circle that you
draw around that definition.
Eg: If measurements were taken 1,
3 and 6 months after the study. Then you would
circle
"Follow-up" and write 1
mo., 3 mo. and 6 mo. within the circle.
Type of dependent variable:
Code the data based on the instrument
that measures the construct of interest. Do not
assume that a step referred to in the use of one
instrument was used in the administration of the
instrument that measured the self construct.
Self-reporting indicates
that the subject supplied the information by filling
out a form, or
using some other means.
Observed indicates that an
observer supplied the information about the subject.
V) Length of Stay:
Circle the number of weeks applicable,
for the purpose of this study a week can be considered
any period from 5 to 8 days. The scale goes from
1 week to 10 weeks. Studies that focus on a period
of time shorter than 1 week are not eligible for
inclusion. In the case where the length of the
study is shorter than the length of time spent
in the camp, report the length of the study, the
shorter time period.
Fill in any information that is available for:
Pre-requisite number of years
at the camp, if any were required to participate
in the study.
The Percentage of returning campers,
if the information is available.
The Number of Sessions studied,
eg: 4, one week sessions.
The Number of Camps where
the study took place.
VI) Camp Type:
Write the type of the camp
down, eg: Sports, Science, Hearing Impaired, Type
of Culture such as jewish, or type of agency such
as YMCA.
Circle Resident if the campers
regularly spend the night at the camp or participating
in a camp activity.
Circle Day if the campers
return home most nights, or leave the supervision
of the camp most nights.
Write the description of the camping
program goals or philosophy, or both, if
stated.
Circle the appropriate indicator
of the gender of the camp population:
Coed camps serve boys and girls.
Boys for an all boys camp.
Girls for an all girls camp.
VII) Demographics:
Cultural Distinction: Write in the following
codes to describe the ethnicity of the subjects:
AA = African American
H = Hispanic
C = Caucasians
J = Jewish
Others - write out the name of the
ethnicity if it is not listed above.
If not indicated, leave blank .
Socio-Economic: Write in the following
codes to describe the socio-economic status of
the subjects.
L = Low economic or disadvantaged economic status.
MC = Middle Class
UMC = Upper Middle Class
Leave Blank if not indicated.
VIII) Source:
Circle the type of research that you are reading,
if you can determine it from the information provided.
Funding: Fill in the source of funding
for the study, if the information is available.
IX) Other Outcomes Studied:
Write down the other outcomes studied
in the research. Include the name of the instruments
used and a description of what those instruments
were designed to measure. If instrument names
are not available, write a brief description of
the issue that was being studied.
X) Statistical Information:
N is the number of subjects included in
the post-test or final set of measurements reported
for the sample. The sample size.
M the number of males in the sample.
F the number of females in the sample.
n-Experimental: record the number of subjects
in the group that receives the treatment, if a
control group was used.
n-Control: record the number of subjects
in the control group, if one was used.
Treatment: Describe the independent variable/treatment.
In other words, what were the subjects exposed
to in order to cause the change in the dependent
variable being measured. This may be as simple
as writing the word "camp", if going
to camp is the only discernable treatment.
X) Statistical Information: continued
Write the information for the various
statistics listed below. Use the appropriate columns
for pre-test and post-test data. Use the appropriate
columns for data that is reported in the
study, or data that you calculated from
data available in the study.
Circle yes or no as
the appropriate response to the question of whether
or not there is sufficient raw data to calculate
the measure.
formula used:
Calculate the following: Hedge's g, Pearson
r, & Fisher's Z transformation.
Write the equation(s) used, from the equation
list, in the formula used field on the
coding sheet. Make note of any calculations
in the calculations field.
Coding Key Conversion Formulas
(Adobe
Acrobat Reader is needed to view this file)
|
