Experimental Design

• Definition: Experimental design is a method of investigating cause-and-effect relationships by manipulating one or more independent variables and measuring the effect on a dependent variable.
• Purpose: The purpose of experimental design is to establish causal relationships between variables in a controlled environment.
• True Experimental Design:
  • Definition: A true experimental design is a research method in which the researcher randomly assigns participants to different groups or conditions, and manipulates the independent variable to test the effect on the dependent variable.
  • Characteristics: True experimental designs typically have a control group that does not receive the manipulation, and a treatment group that does receive the manipulation. In addition, true experimental designs uses a random assignment process to eliminate selection bias.

  Types of True Experimental Designs:

  • Pre-test post-test control group design: This design involves giving a pre-test to all participants before the manipulation of the independent variable, and giving a post-test to all participants after the manipulation. Participants are randomly assigned to either the control group or treatment group. The control group does not receive the manipulation of the independent variable, while the treatment group does receive the manipulation. The post-test scores of the control group and treatment group are compared to see if there is a difference. This design is useful for determining causality.
  • Post-test only control group design: This design involves giving a post-test to all participants after the manipulation of the independent variable. Participants are randomly assigned to either the control group or treatment group. The control group does not receive the manipulation of the independent variable, while the treatment group does receive the manipulation. The post-test scores of the control group and treatment group are compared to see if there is a difference. This design is useful when the pre-test would introduce bias.

  Advantages of True Experimental Design:

  • Establishing cause-and-effect relationships: By manipulating the independent variable and measuring the effect on the dependent variable, true experimental designs can establish causal relationships between variables.
  • Control of extraneous variables: In true experimental designs, the researcher can control extraneous variables that may influence the dependent variable by holding them constant across groups or randomly assigning participants to groups.
  • Studying variables that cannot be manipulated: True experimental designs allow the researcher to manipulate variables that cannot be manipulated in real-world settings.

  Disadvantages of True Experimental Design:

  • Practical and ethical limitations: In some situations, true experimental design may not be practical or ethical. For example, it may not be ethical to manipulate a variable that could cause harm to participants.
  • Lack of external validity: True experimental designs are conducted in a controlled environment and may not be generalizable to the real-world.
  • Difficulty in controlling all extraneous variables: Despite efforts to control extraneous variables, it may not be possible to eliminate all sources of extraneous variability.
  • Costs and time-consuming: True experimental designs can be expensive and time-consuming, both for the researcher and for the participants.

Non-Experimental Design:

• Definition: Non-experimental design is a method of investigating relationships between variables without manipulating the independent variable. Instead, it relies on observations or self-report data and does not involve random assignment to conditions or groups.
• Characteristics: Non-experimental designs are used to describe a phenomenon or a specific population, and not to establish a causal relationship between variables. The researcher does not control or manipulate any variables and simply looks at the relationship between the two variables.

Types of Non-Experimental Designs:

• Correlational research: Correlational research is a type of non-experimental research that examines the relationship between two or more variables. Correlational studies can be conducted using a variety of data collection methods such as surveys, observational studies, or experiments that do not manipulate an independent variable. A correlation coefficient is used to measure the strength and direction of the relationship between two variables.
• Survey research: Survey research is a non-experimental design method where participants are asked to answer a series of questions. Surveys can be administered through various methods such as mail, phone, online, or in-person interviews. Surveys are particularly useful when a researcher wants to obtain a large sample size and when the study participants are scattered across a wide geographic area.
• Case study research: Case study research is a non-experimental design in which an in-depth examination of an individual or group is conducted. Case studies can provide a rich and detailed understanding of a particular case or group of cases and can allow for the examination of complex phenomena and contextual factors that may not be apparent in other research methods.
• Naturalistic observation: Naturalistic observation is a non-experimental research method where the researcher observes and records the behavior of participants in their natural environment without manipulation of any variables. Naturalistic observation can be useful in understanding the behavior in real-life situations and can be used in a variety of settings such as homes, schools, workplaces, etc.
• Longitudinal research: Longitudinal research is a non-experimental research design where data is collected from the same individuals at different points in time, over a period of months or years. Longitudinal research allows for the examination of change over time and can provide a more comprehensive understanding of a phenomenon than cross-sectional research.
• Cross-sectional research: Cross-sectional research is a non-experimental research design where data is collected from different individuals at the same point in time. This type of design allows for the comparison of different groups, often based on age or other demographic characteristics and can provide useful information on the prevalence or distribution of a phenomenon in a particular population.

Advantages of Non-Experimental Design:

• Cost and time-efficient: Non-experimental designs are typically less expensive and time-consuming than experimental designs.
• Examine variables that cannot be manipulated: Non-experimental designs allow the researcher to examine variables that cannot be manipulated, such as past events or traits.
• Generating hypotheses: Non-experimental designs can be useful for generating hypotheses for further research.

Disadvantages of Non-Experimental Design:

• Lack of causality: Non-experimental designs cannot establish a cause-and-effect relationship between variables.
• Confounding variables: Non-experimental designs may be subject to confounding variables, which can influence the relationship between the independent and dependent variables.
• Selection bias: Non-experimental designs often rely on a sample of participants that is not randomly selected, which can lead to selection bias. Selection bias occurs when the sample is not representative of the population being studied, and this can lead to inaccurate conclusions.
• Lack of control: In non-experimental designs, the researcher does not have control over the independent variable and cannot control extraneous variables that may influence the dependent variable.
• Limited generalizability: Non-experimental designs may not be generalizable to other populations or settings due to the lack of random assignment and control over extraneous variables.

Quasi-Experimental Design:

• Definition: Quasi-experimental design is an experimental design in which the researcher does not have complete control over the assignment of participants to conditions or groups. In other words, it is an experimental design where the researcher do not use random assignment to assign participants to groups, it use other methods that might have some limitations or biases, but still allows the researcher to infer causality between variables.
• Characteristics: Quasi-experimental designs rely on statistical techniques to control for extraneous variables, and may not have a control group that does not receive the manipulation. They use non-randomized assignment, where the researcher either uses preexisting groups or convenience sampling to form groups.

Types of Quasi-Experimental Designs:

• Nonequivalent Control Group Design: This type of design involves the selection of a control group that is not equivalent to the treatment group in terms of the variable being manipulated. The researcher makes an effort to match the two groups on relevant characteristics, but it is not a true randomization.
• Interrupted Time Series Design: This type of design involves repeated measures of the dependent variable over time, both before and after the manipulation of the independent variable. It allows the researcher to control for temporal trends and to infer causality between the independent and dependent variable.
• Nonrandomized Control Group Pretest-Posttest Design: This design uses non-random assignment of participants to groups, and uses pretest-posttest measures to infer causality between variables.

Advantages of Quasi-Experimental Design:

• Practicality: Quasi-experimental design can be useful when a true experimental design is not practical or ethical.
• Establishing Causality: Quasi-experimental design can still establish a causal relationship between variables, despite not using random assignment.
• Study of variables that cannot be manipulated: Quasi-experimental design can be used to study variables that cannot be manipulated in a true experimental design.

Disadvantages of Quasi-Experimental Design:

• Selection bias: Quasi-experimental design may be subject to selection bias, which can influence the relationship between the independent and dependent variables.
• Control of extraneous variables: The researcher may not have complete control over extraneous variables, which can lead to inaccurate conclusions.
• Difficulty in generalizing: Quasi-experimental designs may not be generalizable to other populations or settings due to the lack of random assignment and control over extraneous variables.

Matching and Control in Quasi-Experimental Design:

• Definition: Matching and control are techniques used in quasi-experimental designs to reduce the threat of selection bias, which occurs when the sample is not representative of the population being studied. They help control for extraneous variables, increasing the internal validity of the study.

Different Techniques for Matching and Control in Quasi-Experimental Designs:

• Propensity Score Matching: Propensity score matching is a statistical technique that helps to balance the treatment and control groups by using a numerical score that predicts the likelihood of being assigned to a certain group. Researchers use statistical models to compute the propensity scores and then match participants in the treatment group to those in the control group with similar scores.
• Covariate Control: Covariate control is a technique that helps to control for extraneous variables by including them as covariates in the statistical analysis. Researchers can include demographic, background, and pre-existing characteristics as covariates in their analyses, to help control for their effects on the dependent variable.
• Block Randomization: Block Randomization is a technique where researchers divide participants into blocks based on some pre-existing characteristics such as age, gender, etc., and then randomly assign participants within each block to the treatment and control groups. This technique helps to control for the effects of these characteristics on the dependent variable.
• Restriction: Restriction is a technique in which researchers limit the population of participants they recruit to study. Researchers can restrict their sample by only recruiting participants who meet certain criteria, and this can help to reduce the effects of extraneous variables on the dependent variable.

How Matching and Control are Used to Reduce the Threat of Selection Bias in Quasi-Experimental Designs:

• By using these techniques to match or control for extraneous variables, researchers can reduce the threat of selection bias in their study. By controlling for extraneous variables, researchers can increase the internal validity of their study and make more accurate causal inferences.
• By using propensity score matching, for example, researchers can match participants in the treatment group to those in the control group who are similar in terms of certain characteristics. This can help to balance the groups and reduce the threat of selection bias.
• Using techniques such as block randomization or restriction, researchers can also reduce the threat of selection bias by limiting the population of participants they recruit or by randomly assigning participants within certain defined groups.

Analysis in Experimental and Quasi-Experimental Design:

• Definition: Analysis in experimental and quasi-experimental designs involves using statistical techniques to examine the relationship between the independent and dependent variables, and to test hypotheses about the causal relationship between the variables.

Different Types of Statistical Analyses Used in Experimental and Quasi-Experimental Designs:

• T-test: A t-test is a statistical test that is used to determine if there is a significant difference between the means of two groups. It can be used in experimental and quasi-experimental designs to compare the means of a treatment group and a control group.
• Analysis of Variance (ANOVA): ANOVA is a statistical test that is used to determine if there is a significant difference between the means of more than two groups. It can be used in experimental and quasi-experimental designs to compare the means of multiple treatment groups and a control group.
• Multiple Regression: Multiple regression is a statistical technique that is used to examine the relationship between multiple independent variables and a single dependent variable. It can be used in experimental and quasi-experimental designs to examine the relationship between several independent variables and a single dependent variable, while controlling for the effects of other variables.
• Propensity Score Analysis: Propensity score analysis is a statistical technique that is used to examine the relationship between an independent variable and a dependent variable, while controlling for the effects of other variables. It is specifically used for the matching in propensity score matching technique in Quasi-experimental designs.
• Covariate Analysis: Covariate Analysis is a technique that helps to control for extraneous variables by including them as covariates in the statistical analysis. Researchers can include demographic, background, and pre-existing characteristics as covariates in their analyses, to help control for their effects on the dependent variable.

How to Correctly Interpret the Results of These Analyses:

• For t-tests and ANOVA, researchers should examine the p-value, which represents the probability that the results are due to chance. A p-value less than .05 is considered statistically significant, indicating that the results are unlikely to be due to chance.
• For multiple regression and propensity score analysis, researchers should examine the coefficients of the independent variables, which represent the relationship between the independent variables and the dependent variable. A coefficient with a positive sign indicates a positive relationship, and a coefficient with a negative sign indicates a negative relationship.
• For all statistical analyses, it is important to examine the effect size, which represents the magnitude of the relationship between the independent and dependent variables. Effect sizes can be reported as Cohen’s d, which indicates the standardized mean difference between the treatment group and the control group, or as r-squared, which indicates the proportion of variance in the dependent variable that is explained by the independent variables.
• For Quasi-experimental designs, special attention should be given to how the groups were formed, how the researcher controlled for extraneous variables, and how the researcher interpreted the results considering the limitations of the design.