Post Hoc Statistical Procedures:

Suppose you have the following quasi-experimental design:

Participants are not randomly assigned to your two groups (treatment vs. control). Two observations are made for each group, one prior to treatment (pretest, O₁) and one after treatment (posttest, O₂).

If there are no pretest differences between the treatment and control groups the analysis is straightforward. You can run a t test or analysis of variance on the posttest scores.

But what if there are pretest differences between the two groups? Then any posttest differences might be attributed to the pretest differences. The statistical procedures of blocking, matching, analysis of covariance, and gain scores are different statistical techniques for controlling for differences in pretest scores.

Blocking looks at groups (blocks) of participants in the experimental and control conditions who have equivalent pretest scores.

Matching is a slightly more precise procedure that looks at pairs of participants in the pretest and control conditions who have equivalent pretest scores.

Analysis of covariance is more precise still. It "holds constant" the pretest scores by estimating the posttest scores that would have been obtained if everyone had the same pretest score. This is a powerful analysis if the assumptions of the analysis of covariance are met.

Gain score analysis analyzes the difference between each persons pretest and posttest score. It will be shown that an analysis of the gain scores is the same as the interaction in a 2 (pretest vs. posttest) by 2 (experimental vs. control) analysis of variance.

These techniques can be used in randomized designs to control for unwanted differences in pretest scores.

No Difference At Pretest

Figure 1. Overlapping distributions If there are no differences between the groups at pretest, then posttest differences may be attributed to the treatment. In this figure the experimental group is indicated by the blue scatterplot; The control group is indicated by the yellow scatterplot. Pretest scores are shown on the x-axis, posttest scores are shown on the y-axis. The lines that drop to each axis from the scatterplot represents the mean pretest and posttest scores for that condition.. In this example there are no pretest differences. The posttest scores are higher for the treatment group than for the control group.

Pretest Scores Differ

Figure 2. Nonoverlapping distributions Where there are differences at the pretest, then the interpretation of differences at the posttest must take into account the pretest differences. Can the pretest differences account for all the differences in the posttest scores? The blocking approach looks only at the pretest scores from the two conditions that overlap each over. The basic procedure is to find the participants whose pretest scores overlap each other and then compute the posttest means for only those overlapping participants. Pretest scores are held constant by deleting those participants whose pretest scores are too high or too low

Treatment Group PRE TEST POST TEST 1* Control 10 13 2* Control 11 13 3* Control 11 8 4 Control 5 6 5 Control 8 9 6 Control 7 9 7 Control 5 7 8 Control 7 9 9* Control 11 8 10 Control 8 9 11* Control 9 13 12* Control 9 9 13* Control 9 12 14* Control 11 8 15* Control 9 6 16 Treatment 15 12 17* Treatment 9 13 18 Treatment 12 11 19 Treatment 14 14 20* Treatment 9 10 21* Treatment 9 11 22 Treatment 12 15 23 Treatment 12 9 24* Treatment 9 11 25* Treatment 9 7 26* Treatment 11 15 27 Treatment 12 9 28* Treatment 11 8 29 Treatment 15 15 30 Treatment 13 10

Means (SD)   Pretest Posttest Control 8.67 (2.02) 9.27(2.40) Treatment 11.47 (2.17) 11.33(2.61) t test value (df = 28) 3.66, p = .001 2.26, p = .032

The t tests indicate that there are significant differences between the treatment and control groups at both the pretest and the posttest.

In order to control for the pretest differences we need to select out those participants that have overlapping pretest scores. The scatterplot indicates that the overlapping pretest scores fall between 9 and 11. So we would select only those participants who have a pretest score of 9, 10, or 11 and run a t test to see if the posttest scores differed for those participants. The asterisks next to the identification numbers in the raw data table above indicate 16 participants who had pretest scores of 9, 10, or 11.

The results of the t test (run on the posttest scores) indicate that there was no difference between the treatment (M = 10.71, SD = 2.75) and control (M = 10.00, SD = 2.74) conditions when the pretest scores were held constant, t(14) = 0.86, p = .405.

The t test was run on a little over half of the original data. This loss of data is a serious problem for the blocking model. It is possible that in some cases there will be no overlap in pretest scores. In that extreme instance it will not be possible to run this type of analysis. There are no agreed upon rules of thumb for how much data loss is acceptable. If the loss were more than 25% I couldn't feel very comfortable about the analysis.

Technical note:

The SPSS syntax commands for this analysis are as follows:

You could substitute the following select if command for the compute and filter commands:

SELECT IF (pretest GE 9 and pretest LE 11).

Additional Blocks

If more data were available then it would be possible to create smaller blocks of similar pretest scores. For example, you might be able to create a block with scores between, say 9 and 10 (block 1), and another block with scores between 11 and 12 (block 2). Blocks could then be used as a factor in an analysis of variance model. The syntax for the analysis of variance would look like this:

The analysis would yield group and block main effects and a group by block interaction. The group main effect would be the test of whether or not there were differences between the treatment and control groups. The block and and block by group interactions would remove the variability due to pretest differences from the error term, making the group main effect test more powerful.