Effect Size (ES)

Effect size (ES) is a name given to a family of indices that measure the magnitude of a treatment effect. Unlike significance tests, these indices are independent of sample size. ES measures are the common currency of meta-analysis studies that summarize the findings from a specific area of research. See, for example, the influential meta-analysis of psychological, educational, and behavioral treatments by Lipsey and Wilson (1993).

There is a wide array of formulas used to measure ES. For the occasional reader of meta-analysis studies, like myself, this diversity can be confusing. One of my objectives in putting together this set of lecture notes was to organize and summarize the various measures of ES.

In general, ES can be measured in two ways:

a) as the standardized difference between two means, or

b) as the correlation between the independent variable classification and the individual scores on the dependent variable. This correlation is called the "effect size correlation" (Rosnow & Rosenthal, 1996).

These notes begin with the presentation of the basic ES measures for studies with two independent groups. The issues involved when assessing ES for two dependent groups are then described.

1. Standardized difference between two groups.

Cohen's d

The interpretation of Cohen's d

Hedges's g

Glass's delta

2. Correlation measures of effect size

The ES correlation, r_Yl

The relationship between d, r, and r�

3. Computational Examples

The following data come from Wilson, Becker, and Tinker (1995). In that study participants were randomly assigned to either EMDR treatment or delayed EMDR treatment. Treatment group assignment is called TREATGRP in the analysis below. The dependent measure is the Global Severity Index (GSI) of the Symptom Check List-90R. This index is called GLOBAL4 in the analysis below. The analysis looks at the the GSI scores immediately post treatment for those assigned to the EMDR treatment group and at the second pretreatment testing for those assigned to the delayed treatment condition. The output from the SPSS MANOVA and CORR(elation) procedures are shown below.

 
Cell Means and Standard Deviations

Variable .. GLOBAL4 GLOBAL INDEX:SLC-90R POST-TEST
FACTOR CODE Mean Std. Dev. N 95 percent Conf. Interval
TREATGRP TREATMEN .589 .645 40 .383 .795
TREATGRP DELAYED 1.004 .628 40 .803 1.205
For entire sample .797 .666 80 .648 .945
* * * * * * * * * * * * * A n a l y s i s o f V a r i a n c e -- Design 1 * * * * * * * * * * * *
Tests of Significance for GLOBAL4 using UNIQUE sums of squares
Source of Variation SS DF MS F Sig of F
WITHIN CELLS 31.60 78 .41
TREATGRP 3.44 1 3.44 8.49 .005
(Model) 3.44 1 3.44 8.49 .005
(Total) 35.04 79 .44
- - Correlation Coefficients - -
GLOBAL4
TREATGRP .3134
( 80)
P= .005

Look back over the formulas for computing the various ES estimates. This SPSS output has the following relevant information: cell means, standard deviations, and ns, the overall N, and MSwithin. Let's use that information to compute ES estimates.

d = M1 - M2 / �[( s1� + s 2�)/ 2]

= 1.004 - 0.589 / �[(0.628� + 0.645�) / 2]

= 0.415 / �[(0.3944 + 0.4160) / 2]

= 0.415 / �(0.8144 / 2)

= 0.415 / �0.4052

= 0.415 / 0.6366

= .65

= 1.004 - 0.589 / �0.41

= 0.415 / 0.6408

= .65

D = M1 - M2 / s

= 1.004 - 0.589 / 0.628

= 0.415 / 0.628

= .66

r

= �[F(1,_) / (F(1,_) + df error)]

= �[8.49 / (8.49 + 78)]

= �[8.49 / 86.490]

= �0.0982

= .31

The next computational is from the same study. This example uses Wolpe's Subjective Units of Disturbance Scale (SUDS) as the dependent measure. It is a single item, 11-point scale ( 0 = neutral; 10 = the highest level of disturbance imaginable) that measures the level of distress produced by thinking about a trauma. SUDS scores are measured immediately post treatment for those assigned to the EMDR treatment group and at the second pretreatment testing for those assigned to the delayed treatment condition. The SPSS output from the T-TEST and CORR(elation) procedures is shown below.

 t-tests for Independent Samples of TREATGRP TREATMENT GROUP
Number
Variable of Cases Mean SD SE of Mean
-----------------------------------------------------------------------
SUDS4 POST-TEST SUDS
TREATMENT GROUP 40 2.7250 2.592 .410
DELAYED TRMT GROUP 40 7.5000 2.038 .322
-----------------------------------------------------------------------
Mean Difference = -4.7750
Levene's Test for Equality of Variances: F= 1.216 P= .274
t-test for Equality of Means 95%
Variances t-value df 2-Tail Sig SE of Diff CI for Diff
-------------------------------------------------------------------------------
Unequal -9.16 73.89 .000 .521 (-5.814, -3.736)
-------------------------------------------------------------------------------
- - Correlation Coefficients - -
SUDS4
TREATGRP .7199
( 80)
P= .000
(Coefficient / (Cases) / 2-tailed Significance)
 

Use the data in the above table to compute each of the following ES statistics:

There is some controversy about how to compute effect sizes when the two groups are dependent, e.g., when you have matched groups or repeated measures. These designs are also called correlated designs. Let's look at a typical repeated measures design.

A Correlated (or Repeated Measures) Design

OC1 OC2

OE1 X OE2

This research design can be analyzed in a number of ways including by gain scores, a 2 x 2 ANOVA with measurement time as a repeated measure, or by an ANCOVA using the pretest scores as the covariate. All three of these analyses make use of the fact that the pretest scores are correlated with the posttest scores, thus making the significance tests more sensitive to any differences that might occur (relative to an analysis that did not make use of the correlation between the pretest and posttest scores).

An effect size analysis compares the mean of the experimental group with the mean of the control group. The experimental group mean will be the posttreatment scores, O_E2. But any of the other three means might be used as the control group mean. You could look at the ES by comparing O_E2 with its own pretreatment score, O_E1, with the pretreatment score of the control group, O_C1, or with the second testing of the untreated control group, O_C2. Wilson, Becker, and Tinker (1995) computed effect size estimates, Cohen's d, by comparing the experimental group's posttest scores (O_E2) with the second testing of the untreated control group (O_C2). We choose O_C2 because measures taken at the same time would be less likely to be subject to history artifacts, and because any regression to the mean from time 1 to time 2 would tend to make that test more conservative.

Suppose that you decide to compute Cohen's d by comparing the experimental group's pretest scores (O_E2) with their own pretest scores (O_E1), how should the pooled standard deviation be computed? There are two possibilities, you might use the original standard deviations for the two means, or you might use the paired t-test value to compute Cohen's d. Because the paired t-test value takes into account the correlation between the two scores the paired t-test will be larger than a between groups t-test. Thus, the ES computed using the paired t-test value will always be larger than the ES computed using a between groups t-test value, or the original standard deviations of the scores. Rosenthal (1991) recommended using the paired t-test value in computing the ES. A set of meta-analysis computer programs by Mullen and Rosenthal (1985) use the paired t-test value in its computations. However, Dunlop, Cortina, Vaslow, & Burke (1996) convincingly argue that the original standard deviations (or the between group t-test value) should be used to compute ES for correlated designs. They argue that if the pooled standard deviation is corrected for the amount of correlation between the measures, then the ES estimate will be an overestimate of the actual ES. As shown in Table 2 of Dunlop et al., the overestimate is dependent upon the magnitude of the correlation between between the two scores. For example, when the correlation between the scores is at least .8, then the ES estimate is more than twice the magnitude of the ES computed using the original standard deviations of the measures.

The same problem occurs if you use a one-degree of freedom F value that is based on a repeated measures to compute an ES value.

In summary, when you have correlated designs you should use the original standard deviations to compute the ES rather than the paired t-test value or the within subject's F value.

Overview

A meta-analysis is a summary of previous research that uses quantitative methods to compare outcomes across a wide range of studies.  Traditional statistics such as t tests or F tests are inappropriate for such comparisons because the values of  those statistics are partially a function of the sample size.   Studies with equivalent differences between  treatment and control conditions can have widely varying t and F statistics if the studies have different sample sizes.  Meta analyses use some estimate of effect size because effect size estimates are not influenced by sample sizes.  Of the effect size estimates that were discussed earlier in this page, the most common estimate found in current meta analyses is Cohen's d.

In this section we look at a meta analysis of treatment efficacy for posttraumatic stress disorder (Van Etten & Taylor, 1998). For those of you interested in the efficacy of other psychological and behavioral treatments I recommend the influential paper by Lipsey and Wilson (1993).

The Research Base

The meta analysis is based on 61 trials from 39 studies of chronic PTSD. Comparisons are made for Drug treatments, Psychological Treatments and Controls.  

Drug treatments include: selective serotonin reuptake inhibitors (SSRI; new antidepressants such as Prozac and Paxil), monoamine oxidase inhibitors (MAOI, antidepressants such as Parnate and Marplan), tricyclic antidepressants (TCA; antidepressants such as Toffranil), benzodiazepines (BDZ; minor tranquilizers such as Valium), and carbamazepine (Carbmz; anticonvulsants such as Tegretol). 

The psychotherapies include: behavioral treatments (primarily different forms of exposure therapies), eye movement desensitization and reprocessing (EMDR), relaxation therapy, hypnosis, and psychodynamic therapy. 

The control conditions include: pill placebo (used in the drug treatment studies), wait list controls, supportive psychotherapy, and no saccades (a control for eye movements in EMDR studies).

Data Reduction Procedures

Effect sizes were computed as Cohen's d where a positive effect size represents improvement and a negative effect size represents a "worsening of symptoms."

Ninety-percent confidence intervals were computed.  Comparisons were made base on those confidence intervals rather than on statistical tests (e.g., t test) of the mean effect size.  If the mean of one group was not included within the 90% confidence interval of the other group then the two groups differed significantly at p < .10.

Comparisons across conditions (e.g., drug treatments vs. psychotherapies) were made by computing a weighted mean for each group were the individual trial means were weighted by the number of cases for the trial.  This procedure gives more weight to trials with larger ns, presumably the means for those studies are more robust.

Treatment Comparisons

Measures of effect size in ANOVA are measures of the degree of association between and effect (e.g., a main effect, an interaction, a linear contrast) and the dependent variable. They can be thought of as the correlation between an effect and the dependent variable. If the value of the measure of association is squared it can be interpreted as the proportion of variance in the dependent variable that is attributable to each effect. Four of the commonly used measures of effect size in AVOVA are: 

• Eta squared, h² 

• partial Eta squared, h_p² 

• omega squared, w²  

• the Intraclass correlation, r_I 

Eta squared and partial Eta squared are estimates of the degree of association for the sample. Omega squared and the intraclass correlation are estimates of the degree of association in the population. SPSS for Windows displays the partial Eta squared when you check the "display effect size" option in GLM. 

See the following SPSS lecture notes for additional information on these ANOVA-based measures of effect size:  Effect size measures in Analysis of Variance

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.

Dunlop, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170-177.

Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

Mullen, H., & Rosenthal, R. (1985). BASIC meta-analysis: Procedures and programs. Hillsdale, NJ: Earlbaum.

Nunnally, J. C. (1978). Psychometric Theory. New York: McGraw-Hill.

Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage.

Rosenthal, R. & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York: McGraw Hill.

Rosenthal, R. & Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400-406.

Rosnow, R. L., & Rosenthal, R. (1996). Computing contrasts, effect sizes, and counternulls on other people's published data: General procedures for research consumers. Pyschological Methods, 1, 331-340.

Van Etten, Michelle E., & Taylor, S. (1998). Comparative efficacy of treatments for posttraumatic stress disorder: A meta-analysis. Clinical Psychology & Psychotherapy, 5, 126-144.

Wilson, S. A., Becker, L. A., & Tinker, R. H. (1995). Eye movement desensitization and reprocessing (EMDR) treatment for psychologically traumatized individuals. Journal of Consulting and Clinical Psychology, 63, 928-937.