What Does Camp Do for Kids? Appendix C

Purpose

  • increase statistical power

  • resolve uncertainty

  • improve estimate of effect size

  • answer questions not posed at the start of individual trials.

Study Design

  • Protocol

    • criteria for inclusion

    • methods to be used

  • Literature Search

    • consult electronic databases

    • reviews

    • textbooks

    • experts in the field

    • review references in trials found

  • List of Trials Analyzed and Log of Rejected Trials

    • generate list

    • enumerate and reason for exclusion

  • Treatment Assignment

    • record method of control in primary study

    • pooling of randomized vs. non-randomized data

  • Ranges of Characteristics and Treatments

    • present range of subject characteristics in primary studies

    • present range of treatment methods in primary studies

Combinability
A major issue in pooling data is whether the results of separate trials can be meaningfully combined. (451)

  • Criteria detailed

  • Measurement of homogeneity

Control and Measurement of Potential Bias

  • Selection Bias (blinding technique for accepting and rejecting)

    • look at methods only

    • code method and result separately

  • Data-Extraction Bias

    • proper blinding of extraction process

    • multiple coders or statistical validation of primary coder

  • Source of Support Stated

Statistical Analysis

  • Statistical Method

    • any recognized technique

    • no simple addition of successes across trials

  • Statistical Errors

    • awareness of Type I

    • awareness of Type II

  • Confidence Intervals

    • of difference between success rate of treatments

  • Subgroup Analysis

    • purpose of meta-analysis

Sensitivity Analysis

    • depends on statistical test chosen

    • data combined to give different conclusions

  • Quality Assessment

    • methodologic rigor

    • scientific quality

    • randomization process

    • measurement of subject compliance

    • blinding of subjects and observers

    • statistical analysis

    • handling of withdrawals in primary studies

  • Varying Methods Sensitivity Analysis

    • to establish result spectrum

  • Publication Bias

    • calculate number of negative studies needed to refute findings

    • published vs. Unpublished

Application of Results

  • Caveats

    • how should results be used/focus of observation and recommendation

    • opportunities for research

    • research method refinements that are needed

  • Economic Impact

    • of adopting new methods

Remaining Problems

  • appropriateness of meta-analysis

    • efficacy established through definition and design

    • role exists when definitive randomized trial impractical or impossible

    • gives quantitative estimates of weight of available evidence

  • meta-analysis question stated and relevant

  • Limitations

    • location of primary studies

    • statistical analysis

    • publication bias vs. study quality

  • Sufficient Information for Reader Conclusion About Validity

  • Scientific Rigor of Meta-Analysis

    • clear question

    • clear method to be used

    • intraobserver and interobserver variability estimates

    • identify and minimize bias

    • establish validity of pooling

Sources:
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A.,Chalmers, T. C. (1987, February 14). Meta-analyses of randomized controlled trials. The New England Journal of Medicine, (316), 8, 450-455.

Cooper & Hedges, 1994.

Tags: