These notes provide some basic principles you need to bear in mind when designing a research project. If you have already taken a research methods course, they will serve as a useful reminder. If you have never taken a course in research methods, they should point you in the right direction.
See also the notes on generating a research idea and on writing a research report.
The goal of research is to answer questions. Of course, the question itself must be an important one. We do not address importance in these notes, but see the notes on generating strong (and weak) research ideas.
Assuming your research question is worth answering, your success depends on meeting two other important criteria, power and control. A third concern, relevant if the first two criteria are satisfied, is generalizability.
Power is the probability that your study will find an effect if it is present. Absence of power means that your research fails, not because you are asking the wrong question, but because you were unable to find what you were looking for.
Control refers to the validity of any conclusion you reach. Control guarantees that your answer is not tainted because of errors in the design of your study.
The results of your research are generalizable if they apply beyond the specific details of your study. Often referred to as external validity, generalizability determines how seriously your research conclusion can be taken in the world at large.
How you evaluate your study depends on the kind of research question you ask. Research generally addresses two kinds of questions, correlational and causal. A correlational question asks if there is a relationship between two or more aspects of behavior. It seeks a description rather than an explanation. A causal question seeks an explanation for behavior. It asks if the behavior is causally affected by some feature of the situation or the person. As we'll see, causal questions are much harder to answer than correlational questions.
2. Correlational and experimental research
Even if you never had a course in research methods, you have presumably learned that correlational data cannot be used to answer causal questions. Unfortunately, many students (and some advanced researchers) fail to understand the full implications of this principle.
Suppose we find that children who share their toys frequently with others also score higher on a test of empathy. What can we conclude from this finding? Here are four possible conclusions:
1. Requiring children to share their toys will enhance their sense of empathy.
2. Increasing a child's empathy for others will lead to their being more willing to share.
3. As children grow older, both empathy and willingness to share will increase.
4. Children who score higher on a test of empathy are more likely than others to share their toys.Probably, only the lastconclusion is justified.
Whenever we observe a correlation between two variables, X and Y, the result can be interpreted in three ways. Either X caused the changes in Y, or Y caused the changes in X, or some third variable caused both X and in Y to change. These are, of course, conclusions 1 through 3 above. In the absence of further information, we cannot choose between them.
If all we want to do is predict behavior, then correlational data are enough. Conclusion 4 is a prediction, and is therefore justified.
If the hypothesis we wish to support is that X causes changes in Y, then we must show that deliberate changes in X are associated with predicted changes in Y, with all other factors controlled. To support a causal hypothesis we must conduct a true experiment, in which the presumed causal variable is deliberately manipulated, while all other variables are controlled.
Causal hypotheses are almost always of more interest than correlational hypotheses. They are necessary if we want to understand or modify a psychological process.
Unfortunately, many variables of interest cannot be manipulated, for practical or for ethical reasons. For example, we cannot manipulate pathological conditions, which makes it hard or impossible to attribute changes in behavior to any pathology. In many cases, then, there is no alternative but to conduct correlational research. In these cases, it is still possible to exercise some control statistically. You must recognize, though, that the conclusions will never be as strong as you would like them to be.
4. Manipulating independent variablesAll research involves the use of two or more variables. A variable is anything that varies during the research. You will be looking for relationships among variables, and in the case of causal hypotheses trying to show that a relationship is a causal one. You need to identify a number of different kinds of variables.
a. Dependent variables. In any research project you will measure one or more aspects of subjects' behavior. The variables that you measure are dependent variables. Presumably it is performance on these variables that you hope to predict or explain.
b. Independent variables. An independent variable is any characteristic of the research setting that you manipulate. You must be able to determine freely for any one subject what value the variable will take on. If the variable is not under your control in this way, it cannot be an independent variable.
Independent variables are the presumed causes of behavior. In a true experiment there must be at least one independent variable. Presumably you are interested in the effect of each independent variable on one or more dependent variables, and you would like to draw causal conclusions.
Note that it is very hard, sometimes impossible, to treat characteristics of a person as independent variables.
c. Classification variables. Consider the age of your subjects. It is not an independent variable, since it is not manipulated. You have no direct control over any one subject's age. On the other hand, although you need to measure it, it is not a typical dependent variable. It does not represent behavior that you want to explain, or even predict.
Age is a classification variable. Such variables are often treated as if they were independent variables (i.e., we think of them as being responsible for something happening), but because they are not manipulated they do not permit you to make unambiguous causal statements. Strictly speaking, they support only correlational conclusions.
Typical classification variables are age, sex, psychopathological diagnosis, socio-economic status, educational level, etc. Many characteristics of a person are classification variables. Whenever you include classifications variable in your research question, you must be wary of drawing illegitimate causal conclusions.
d. Extraneous variables. Dependent, independent, and classification variables are the ones you pay attention to. But every research project includes a potentially infinite number of other variables that are neither manipulated nor measured. These are the extraneous variables.
Extraneous variables come in two forms, confounding and random. Confounding variables are a direct threat to the validity of your research, and must be controlled. We examine confounding variables in Section 5. Random variables do not threaten the validity of your conclusions, but they have an impact on power. We examine power in Section 7.
You should make sure you understand the difference between random and confounding variables. If an extraneous variable is not a confounding variable, it is a random variable. Random variables can be a nuisance, but they are not as serious a threat as confounding variables.
Typical extraneous variables include individual differences among the subjects, changing features of the research setting, and events occuring outside the research setting that might impact subjects' behavior.
e. Constants. Constants are features that might have been variable, but they have been fixed, so they do not vary. Classification variables that are often fixed in this way. For example, your study might use only female subjects. Sex, then, is a constant. Sometimes a classification variable is partially constant. For example, all subjects might be between 18 and 24 months old. Age is fixed within that range. Many features of the research setting are also fixed.
As we'll see, holding variables constant is one way to prevent confounding, and thus enhance control. There are two other principles to keep in mind about constants, and they tend to be in conflict:
1. You can increase power by fixing variables that might otherwise have a large effect on your dependent variables. For example, if your dependent variable changes a great deal with time of day, then holding time of day constant or partially constant will increase power (see Section 7).
2. You can reduce the generalizability of your results by fixing variables that might change the way an independent variable affects dependent variables. For example, if your independent variable has different effects for young children and older children, your results may be misleading if you use only young subjects. If you suspect this could be the case you may need to conduct one or more replications and look for interaction effects (see Section 9).
5. Confounding and controlBy definition, independent variables are manipulated. This implies that for any subject in the experiment you have complete control over the independent variable, and can set its level to be anything you wish. You should think of independent variables as something you do to the experimental conditions, not something you do, at least not directly, to the subjects. For example, you cannot manipulate age by selecting, say, a five year old subject. You cannot do anything to people to change their age.
In designing an experiment, independent variables may be manipulated in one of two ways, between subjects or within subjects. In a between subjects design, separate groups of subjects are used for each possible value of the variable. In a within subjects design, each subject is tested under all different values of the variable.
If you use a between-subjects design it is essential that you use random assignment to allocate subjects to the various conditions. Any non-random assignment will introduce confounding into your experiment. If you use random assignment, all extraneous individual difference variables are random variables.
Within subject designs are almost always more powerful than between subjects designs (Section 7). You are controlling all the individual differences that might be extraneous variables, including intelligence and personality factors. We sometimes talk about using subjects as their own controls.
The drawback to within subjects designs is that you must be careful to avoid confounding caused by the order in which different values of the variable are tested (see Section 5).
In fact, whenever you manipulate an independent variable, you must exercise great caution to make sure that no other variable is systematically varied at the same time. This would lead to confounding, and it destroys the validity of your research. Sometimes it takes great vigilance to recognize that confounding has occurred, and it may require considerable ingenuity to overcome that confounding (see Section 5).
Confounding can arise in either correlational or in experimental research. It is a serious threat to the validity of your research, especially if you are asking a causal research question. To guard against confounding you can exercise a number of techniques that control the confounding variables.
a. Correlational research. In correlational research, a variable is a confounding variable if it is correlated with both variables involved in a correlational. Suppose, for example, you find that autistic children show less empathy than non-autistic children. Suppose further that intelligence differs between autistic and non-autistic children, and is related to empathy as well. Then intelligence is a confounding variable. Thus, you do not know if the differences in empathy were due to autism, to differences in intelligence, or perhaps to other confounding variables.
Confounding in correlational research is almost inevitable. Because nothing is manipulated, there are many variables that can potentially confound a correlation. You can control them one at a time by fixing them, and thus turning them into constants, but can never eliminate all possible confounding this way.
If you are using a classification variable you may be able to effect some control by matching subjects for important confounding variables. For example, you may be able to ensure that for every autistic child you have selected a non-autistic child of equal intelligence. This way you have controlled for intelligence. However, there will always be other potentially confounding variables, more than you can control by matching.
There are also a number of advanced statistical procedures, including path analysis and structural equation modeling, that can sometimes substitute for direct experimental control, but they are beyond the scope of these notes.
Finally, note that if you are concerned only about predicting behavior, the existence of confounding in correlational research may be irrelevant. If your predictions are successful, you may not care why they are successful.
b. Experimental research. In experimental research a variable is a confounding variable if it is systematically related to both an independent variable and a dependent variable. Confounding will invalidate any causal conclusions you try to draw from your experiment.
Suppose, for example, you manipulate the abstractness of words (abstract versus concrete) in a memory experiment. Then you find out that the abstract words occur less frequently in the English language than the concrete words. You do not know if changes in performance were due to abstractness or to word frequency.
Incidental confounding that occurs when you manipulate an independent variable is called a manipulation artifact. The solution is to find some way of manipulating the variable while ruling out the incidental changes, or at least determining if and when they occur. In the example above, you may need to make sure that words used in all conditions are matched for word frequency.
When the independent variable is manipulated within subjects (all subjects are tested with all values of the variable) you need to take special care to avoid confounding due to practice effects, fatigue effects, or other order effects.
A standard control procedure for order effects is counterbalancing. Use separate groups of subjects that receive the treatments in different orders. If you have only two treatments, A and B, you will need two groups of subjects, one tested in the order AB, the other tested in the order BA. With more than two treatments, counterbalancing becomes more complicated. If you do not use counterbalancing, you must present the conditions in a random order, chosen separately for each subject.
6. Measuring dependent variables
The most important concern with your dependent variables is the reliability of your measurements. If reliability is poor, it is impossible to obtain meaningful results. You can conduct pilot studies to assess reliability, but normally the best way to ensure reliability is to use measuring procedures that other investigators have used successfully.
Reliability is notoriously poor for variables that rely on ill-defined human judgments for their measurement. Ratings of children's behavior, for example, should be checked carefully for reliability. It is also wise to avoid open-ended questions unless a reliable coding scheme has been developed. Any other flaw in your measurement procedure that introduces error will reduce reliability. Reliability of your measurements is an important determinant of power in your research (see Section 7).
You may see reference in some textbooks to concerns about the validity of a measuring procedure. Validity of measurements is a complex subject. For most purposes, measurements can be said to be valid if they predict what you expect them to predict, or are changed in ways you expect them to be changed. Thus, validity of measurement in your research is closely tied to support for your hypotheses. If your hypotheses are supported, you can usually take measurement validity to be a given.
7. Factors that determine power
Power is the probability that your study will find an effect when the effect really does exist. A number of factors can affect the power of your research. Some of these factors have been discussed in previous sections.
a. Experimental design. If you use a within-subjects procedure to manipulate an independent variable, power will be greater than it would be if you use a between-subjects design (Section 4). If you use a between-subjects design you can increase power by using matched groups of subjects, matching them on a variable that you know to be highly correlated with your dependent variable. The greater the correlation between matching variable and dependent variable, the greater the increase in power.
b. Reliability. Power is closely related to the reliability of your dependent variable (Section 6). Power is reduced if you use a measurement procedure that has low reliability.
c. Extraneous variables. Random extraneous variables (those that are not confounding) will reduce power if they are correlated with your dependent variable (Section 3d). For example, some reaction times may vary with time of day. If reaction time is your dependent variable and your experiment is run at varying times of day, power may be reduced. You may be able to increase power by holding the variable constant, or partially constant (Section 3e).
d. Number of subjects. The most direct way to increase power is to use more subjects. There are statistical procedures for estimating the number of subjects you will need to find an effect of a given size.
8. Results: Main effects and interaction effects
When an independent variable (or classification variable) is shown to have an effect on a dependent variable, we call it a main effect. In the simplest study, we would have one independent or classification variable and one dependent variable, and we would look for a main effect.
Interesting main effects are rare in studies of human behavior. Whenever one asks, "What effect does X have on Y", the answer is almost always, "It depends". Descriptions of how and on what it depends are descriptions of interactions.
We define an interaction as a change in how an independent variable or a classification variable affects a dependent variable as a function of additional independent or classification variables. Here's a simple example:
How does performing in front of an audience affect the quality of a person's performance? It is known that performance will generally improve if the performer is skilled, but it may deteriorate if the performer is a novice. That is, presence or absence of an audience interacts with skill level to affect performance.
Note that in describing an interaction we always talk about two independent (or classification) variables interacting with each other in their effect on a dependent variable. It is incorrect to say that an independent variables interacts with a dependent variable.
Interactions are used to clarify the psychological processes that determine behavior. For example, the interaction between skill level and audience tells us a lot about how a performer exercises her skill. Thus, if you are interested in exploring the processes that underlie behavior, you should include at least two independent variables in the design of your research. This is usually the best way to test theories of behavior. Use the theory to derive predicted interactions.
"For a research psychologist, interactions are the spice of life".
9. Generalizability: Replication and interactions
There is an important connection between interactions and generalizability. If you have found a main effect for an independent variable, you probably want to know if the results are generalizable. In other words, you want to know what would happen if you changed one or more of the constant features in the experiment.
Suppose you have tested the effects of an educational intervention in a particular class at a particular school. It is necessary to find out if the results will generalize to other classes at other schools. In effect, you are asking if the intervention variable interacts with other variables that might differentiate the tested class from others.
Of course, you could answer the question by repeating the experiment with other clasees at other schools, i.e., by replicating the study with some variation in the constants. If you find the same main effect each time, there is no interaction, and the results will generalize. If there are interactions the results may not generalize. The degree of generalizability depends on the size of the interactions: the bigger the interactions, the poorer is the generalizability.
It should be clear, then, that the only safe way to assess generalizability is to carry out a number of replications. If you are not able to do this, you should at least be aware of the concerns.