Statistical Analysis of Quasi-Experimental Designs:
I. Apriori Selection Techniques
|
I. Overview II. Matching in Experimental Designs III. Matching in Quasi-Experimental Designs |
I. Overview
Random assignment is used in experimental designs to help assure that different treatment groups are equivalent prior to treatment. With small n's randomization is messy, the groups may not be equivalent on some important characteristic.
In general, matching is used when you want to make sure that members of the various groups are equivalent on one or more characteristics. If you are want to make absolutely sure that the treatment groups are equivalent on some attribute you can use matched random assignment.
When you can't randomly assign to conditions you can still use matching techniques to try to equate groups on important characteristics. This set of notes makes the distinction between normative group matching and normative group equivalence. In normative group matching you select an exact match from normative comparison group for each participant in the treatment group. In normative group equivalence you select a comparison group that has approximately equivalent characteristics to the treatment group.
II. Matching in Experimental Designs:
Matched Random Assignment
In an experimental design, matched random sampling can be used to equate the groups on one or more characteristics. Whitley (in chapter 8) uses an example of matching on IQ.
The Matching Process
|
1. Obtain scores on the variable of interest (e.g., IQ) and rank order participants according to that score. The scores in Table 1 have been ranked according to IQ scores. 2. Take the people with the top two scores (Block 1) and randomly assign them the control and experimental conditions. Take the people with next highest scores (Block 2) and randomly assign them to the control and experimental group. Continue until all participants have been assigned to conditions. Expand as necessary according to the design of your study. For example, in a 2 x 3 factorial design, take the people with the top 6 IQ scores (Block 1) and randomly assign them to each of the six cells in the design. This procedure will assure that each of the treatment and control groups are equivalent on IQ (or whatever characteristic(s) is(are) matched).
|
Note: Tx = Treatment group, Ctl = Control Group. |
Analysis of a Matched Random Assignment Design
If the matching variable is related to the dependent variable, (e.g., IQ is related to almost all studies of memory and learning), then you can incorporate the matching variable as a blocking variable in your analysis of variance. That is, in the 2 x 3 example, the first 6 participants can be entered as IQ block #1, the second 6 participants as IQ block #2. This removes the variance due to IQ from the error term, increasing the power of the study.
The analysis is treated as a repeated measures design where the measures for each block of participants are considered to be repeated measures. For example, in setting up the data for a two-group design (experimental vs. control) the data would look like this:
| IQ Scores |
Performance Scores |
|||
| Block Identifier | iq_ctl | iq_tx | ps_ctl | ps_tx |
| 1 | 132 | 133 | 27 | 32 |
| 2 | 130 | 130 | 19 | 17 |
| 3 | 129 | 128 | 30 | 35 |
| 4 | 125 | 128 | . | 37 |
| etc | ||||
|
Note: tx = Treatment Group; ctl = Control Group |
||||
The analysis would be run as a repeated measures design with group (control vs. experimental) as a within-subjects factor.
If you were interested in analyzing the equivalence of the groups on the IQ score variable you could enter the IQ scores as separate variables. An analysis of variance of the IQ scores with treatment group (Treatment vs. Control) as a within-subjects factor should show no mean differences between the two groups. Entering the IQ data would allow you to find the correlation between IQ and performance scores within each treatment group.
One of the problems with this type of analysis is that if any score is missing then the entire block is set to missing. None of the performance data from Block 4 in Table 2 would be included in the analysis because the performance score is missing for the person in the control group. If you had a 6 cells in your design you would loose the data on all 6 people in a block that had only one missing data point.
I understand that Dr. Klebe has been writing a new data analysis program to take care of this kind of missing data problem.
SPSS Note
The SPSS syntax commands for running the data in Table 2 as a repeated measures analysis of variance are shown in Table 3. The SPSS syntax commands for running the data in Table 2 as a paired t test are shown in Table 4.
Table 3. SPSS Syntax Commands for a 2-Group, Matched Random Assignment Design Run as a Within-Subjects ANOVA
|
Table 4. SPSS Syntax Commands for a 2-Group, Matched Random Assignment Design Run as a Paired t Test.
|
III. Matching in Quasi-Experimental Designs:
Normative Group Matching
Suppose that you have a quasi-experiment where you want to compare an experimental group (e.g., people who have suffered mild head injury) with a sample from a normative population. Suppose that there are several hundred people in the normative population.
One strategy is to randomly select the same number of people from the normative population as you have in your experimental group. If the demographic characteristics of the normative group approximate those of your experimental group, then this process may be appropriate. But, what if the normative group contains equal numbers of males and females ranging in age from 6 to 102, and people in your experimental condition are all males ranging in age from 18 to 35? Then it is unlikely that the demographic characteristics of the people sampled from the normative group will match those of your experimental group. For that reason, simple random selection is rarely appropriate when sampling from a normative population.
The Normative Group Matching Procedure
Determine the relevant characteristics (e.g., age, gender, SES, etc.) of each person in your experimental group. E.g., Exp person #1 is a 27 year-old male. Then randomly select one of the 27 year-old males from the normative population as a match for Exp person #1. Exp person #2 is a 35 year-old male, then randomly select one of the 35 year-old males as a match for Exp person #2. If you have done randomize normative group matching then the matching variable should be used as a blocking factor in the ANOVA.
If you have a limited number of people in the normative group then you can do caliper matching. In caliper matching you select the matching person based a range of scores, for example, you can caliper match within a range of 3 years. Exp person #1 would be randomly selected from males whose age ranged from 26 to 27 years. If you used a five year caliper for age then for exp person #1 you randomly select a males from those whose age ranged from 25 to 29 years old. You would want a narrower age caliper for children and adolescents than for adults.
This procedure becomes very difficult to accomplish when you try to start matching on more than one variable. Think of the problems of finding exact matches when several variables are used, e.g., an exact match for a 27-year old, white female with an IQ score of 103 and 5 children.
Analysis of a Normative Group Matching Design
The analysis is the same as for a matched random assignment design. If the matching variable is related to the dependent variable, then you can incorporate the matching variable as a blocking variable in your analysis of variance.
III. Matching in Quasi-Experimental Designs:
Normative Group Equivalence
Because of the problems in selecting people in a normative group matching design and the potential problems with the data analysis of that design, you may want to make the normative comparison group equivalent on selected demographic characteristics. You might want the same proportion of males and females, and the mean age (and SD) of the normative group should be the same as those in the experimental group. If the ages of the people in the experimental group ranged from 18 to 35, then your normative group might contain an equal number of participants randomly selected from those in the age range from 18 to 35 in the normative population.
Analysis of a Normative Group Equivalence Design
In the case of normative group equivalence there is no special ANOVA procedure as there is in Normative Group Matching. In general, demographic characteristics themselves rarely predict the d.v., so you haven’t lost anything by using the group equivalence method.
A Semantic Caution
The term "matching" implies a one-to-one matching and it implies that you have incorporated that matched variable into your ANOVA design. Please don’t use the term "matching" when you mean mere "equivalence."
-03/12/00