| Dr. V.K. Maheshwari
M.A, M.Ed, Ph.D Roorkee, India
|
Rakhi Maheshwari
M.A, B.Ed Noida, India
|
“Good data are always separable, with respect to their scientific importance, from the purposes for which they were obtained.”
Murray Sidman (1960) .Tactics of Scientific Research
It has became widely accepted that the best way to resolve issues on which there are a large number of studies is to carry out a meta-analysis. The 1980s and 1990s witnessed a rapid upsurge of this statistical approach (Anastasi and Urbina, 1997).
Although commonly viewed as a relatively modern advancement, the basic elements of meta-analysis can be traced back to R. A. Fisher, who developed an early quantitative procedure for combining the probabilities from multiple hypothesis tests.
A common problem in many laboratories is that multiple studies are conducted on the same hypothesis and some way of “combining” the studies is sought. Often each study provides weak evidence but there may appear to be some consistency in findings across studies. A way of capturing that consistency is desired.
An equally common problem is that hundreds of studies accumulate in a research area. Here too there are likely to be many weak studies and considerable inconsistency in study outcomes. A precise method for combining the studies would be desirable. An additional problem is that the studies may be different in a variety of ways.
Modern methods of meta-analysis were developed to solve this second kind of problem. The meta-analytic “revolution” began in the late 1970s. The psychotherapy outcome analysis published by Smith and Glass (1977) brought the method to the attention of many psychologists, highlighting its potential advantages. Smith and Glass (1977) examined over 300 therapy outcome studies and concluded that, overall, therapy was quite effective, placing the average treated person better off than 80% of untreated people.
Meta-analysis is a quantitative statistical analysis of several separate but similar experiments or studies in order to test the pooled data for statistical significance
Meta-analysis attempts to apply to a collection of studies the same methodological rigor and statistical precision ordinarily found in primary research.
In a meta-analysis, the collection of studies test the same conceptual hypothesis, but may do so using a wide variety of methods, measures, sample, and settings.
The challenge that meta-analysis answers is to provide a way to combine the seemingly disparate studies to provide a convincing overall test of the hypothesis and to explore its moderators.
Meta-analysis should be viewed as an observational study of the evidence. The steps involved are similar to any other research undertaking:
v Formulation of the problem to be addressed,
v Collection and analysis of the data,
v Reporting of the results.
Researchers should write in advance a detailed research protocol that clearly states the objectives, the hypotheses to be tested, the subgroups of interest, and the proposed methods and criteria for identifying and selecting relevant studies and extracting and analysing information.
Meta-analysis summarizes the results of many quantitative studies that have investigated the same problem. It provides a numerical way of expressing the average result of a group of studies. It delineates specific procedures for finding, describing, classifying, and coding research studies to be included in a meta-analysis review, and for measuring and analysis of findings.
A central characteristic that distinguishes meta-analysis from more traditional approaches is the emphasis placed on making the review as inclusive as possible. This technique was first proposed by Glass (1976) and by the end of the 1980s it had become accepted as a useful method for synthesizing the results of many different studies.
Glass distinguished between Other Forms of Analysis the primary, secondary, and meta-analysis of research.
Primary analysis is the original analysis of data in a research study. The analysis of data from a single study to test the hypotheses originally formulated.
Secondary analysis is re-analysis of data for the purposes of answering the original research question with better statistical techniques, or answering new questions with all data. The re-analysis of data from a single study to test new hypotheses or to apply more appropriate statistical procedures to test the original hypotheses.
Meta-analysis refers to the analysis of analyses; the statistical analysis of a large collection of analysis results from individual studies for the purposes of integrating the findings. The application of statistical procedures to examine tests of a common hypothesis from more than one study. It connotes a rigorous alternative to the casual, narrative discussion of research studies which typify our attempts to make sense of the rapidly expanding research literature.It contributes in the creation of new knowledge synthesized from existing studies. The literature explosion has resulted in a massive amount of information that must be analyzed and summarized in order to be useful. Quantitative methods of integration of research results have been used for many years and have received a great amount of attention (Abraham et al., 1991).
Meta-analysis usually involves three major phases; the three “Ps”: preparation, performance, and presentation. This sequence is the same as for any other type of research. The project must be planned in advance, then systematically carried out, then followed by reporting of results (Abraham et al., 1991).
Any statistical procedure or analytic approach can be misused or abused. As Green and Hall (1984) aptly stated “Data analysis is an aid to thought, not a substitute”. Most of the criticisms of quantitative approaches to reviewing the literature are objections to the misuse or abuse, real or potential, of meta-analysis.
Meta-analysis is a statistical approach to the aggregation summarization of results from independent studies. It is systematic, thorough, objective, and quantitative. The essentials of this technique are to collect all the studies on the issue, convert the results to a common metric and average them to give an overall result. Procedures employed in meta- analysis permit quantitative reviews and syntheses of research literature that address these issues (Wolf, 1986). An epidemiologist has described meta-analysis as “a boon for policy makers who find themselves faced with a mountain of conflicting studies” (Mann, 1990).
Any meta-analyst has to address three problems that have been identified by Sharpe (1997) as the “Apples and Oranges”, “File Drawer” and “Garbage in - Garbage out” problems.
The “Apples and Oranges” problem refers to the idea that different phenomena are sometimes aggregated and averaged, where disaggregation may show different effects for different phenomena. The best way of dealing with this problem is to carry out meta- analyses, in the first instance, on narrowly defined phenomena and populations and then attempt to integrate these into broader categories.
The “File Drawer” problem means that studies producing significant effects tend to be published, while those producing non-significant effects tend not to be published and remain unknown in the file drawer.
The “Garbage in – Garbage out” problem concerns poor quality studies. Meta-analyses that include many poor quality studies have been criticized by Feinstein (1995) as “statistical alchemy” which attempt to turn a lot of poor quality studies into good quality gold. Poor quality studies are liable to obscure relationships that exist and can be detected by good quality studies. Meta-analysts differ in the extent to which they judge studies to be of such poor quality that they should be excluded from the analysis. Some meta- analysts are “inclusionist” while others are “exclusionist”, in the terminology suggested by Kraemer, Gardner, Brooks and Yesavage (1998). This meta-analysis is “inclusionist” in the sense that it included all the studies on the Progressive Matrices among school and university students that have been located if the strict inclusion criteria apply to them.
The next problem in the meta-analysis was to obtain all the studies of the issue in concern. This is a difficult problem and one that it is rarely and probably never possible to solve completely.
Although, there is no set pattern of the procedure of meta- analysis, still the widely used pattern is given below-
Steps to Perform a Meta-Analysis
- Define the meta-analytic research question
- Locate the relevant literature
- Calculate effect sizes and code moderating variables
- Analyze the meta-analytic database
- Report and interpret the results
Step One -Define the Meta-Analytic Research Question
The purpose of this step is to determine what hypotheses your meta-analysis will test and to estimate the strength of an effect. At this stage we also determine moderators of an effect and determine what types of studies one will include in his analysis
The goals for this step are:
– Hypotheses should have theoretical value
– Should have specific inclusion criteria to make locating studies easier
– Included studies should be appropriate for the hypotheses being tested
Step Two- Locate the relevant literature
The purpose of this step this step is to obtain the population of studies related to your research hypotheses and to modify the hypotheses and inclusion criteria of your analysis to better fit the literature .
- Goals for this step
– Should find every study that has investigated the effect of interest
– Make your hypotheses better address the questions that researchers have investigated in primary research
Step Three- Calculate effect sizes and code moderating variables
The purpose of this step is to determine what effects you will examine in each study, Compute a specific estimate of the size of each effect and to determine the value of your moderating variables for each effect
- Goals for this step
– Accurately determine effect size estimates and moderator codes
– Should try to have estimates for every effect
– Typically have two different people calculate effect sizes and code moderators so you can estimate reliability
Step Four- Analyze the meta-analytic database
The purpose of this step is to perform descriptive analyses to determine the overall strength and consistency of the effect and perform moderator analyses to determine if study characteristics influence the effect size.
- Goal for this step
– Analyses should be valid
– Analyses should directly answer the research questions
Step Five- Report and interpret the results
The purpose of this step is to summarize the results of your analyses, relate your analyses to the research questions and draw conclusions based on your analyses
- Goals for this step
– Verbally describe the implications of your analyses
– Report any limitations you see regarding your analysis
- Violation of assumptions
- Power
- Representativeness
Suggest areas of future research
Merits of Meta-analysis:
- It increases power and leads to stronger conclusions because more studies can be analyzed with statistical methods than the impressionistic literary review. Often this can bring effects into sharper focus, particularly when the results of all studies are not consistent (Higgins and Green, 2006).
- Meta-analysis does not prejudge or exclude some studies as unworthy because of their particular research designs, however weak. By empirically examining the 241 effects of research quality on study findings, meta-analysis is likely to be more objective than traditional literary reviews (Wolf, 1986).
- It can answer questions not posed by the individual studies (Higgins and Green, 2006).
- It can settle controversies arising from apparently conflicting studies (Higgins and Green, 2006).
Limitations of Meta-analysis
• It oversimplifies the results of a research domain by focusing on the overall effects and downplaying mediating or interaction effects. The better examples of meta-analyses built potential mediating factors into their designs rather than ignoring them. They do this by coding the characteristics of studies to empirically examine whether such interactions exist. In practice, many meta-analyses do not provide sufficient attention to possible interaction effects (Wolf, 1986).
• Meta-analysis of poor quality studies may be seriously misleading (Higgins and Green, 2006).
• Decisions regarding inclusion and exclusion criteria of studies are inevitably subjective. In some cases consensus may be hard to reach (Higgins and Green, 2006).
• Meta-analysis in the presence of serious publication and/or reporting bias may produce an inappropriate summary (Higgins and Green, 2006).
Meta Analysis refers to a research strategy where instead of conducting new research with participants, the researchers examine the results of several previous studies. This is done with the purpose of gaining greater confidence in the results because of the larger pool of participants, as long as steps are taken to avoid errors that may have existed in the original studies.