Guidelines for Critical Analysis of Published Research Article

Guidelines for Critical Analysis of Published Research Article

Evaluating Titles

 

1. Is the title sufficiently specific?
2. Does the title indicate the nature of the research without describing the results?
3. Has the author avoided using a “yes-no” question as a title?
4. If there is a main title and a subtitle, do both provide important information about the research?
5. Are the primary variables referred to in the title?
6. Does the title indicate what types of people participated?
7. If the title implies causality, does the method of research justify it?
8. Has the author avoided using jargon and acronyms that might be unknown to his or her audience?
9. Overall, is the title effective and appropriate?

 

Evaluating Abstracts

 

1. Is the purpose of the study referred to or at least clearly implied?
2. Does the abstract highlight the research methodology?
3. Has the researcher omitted the titles of measures (except when these are the focus of the researcher)?
4. Are the highlights of the results described?
5. Has the researcher avoided making vague references to implications and future research directions?
6. Overall, is the abstract effective and appropriate?

 

Evaluating Introductions and Literature Reviews

 

1. Does the researcher begin by identifying a specific problem area?
2. Does the researcher establish the importance of the problem area?
3. Is the introduction an essay that logically moves from topic to topic?
4. Has the researcher provided conceptual definitions of key terms?
5. Has the researcher indicated the basis for “factual” statements?
6. Do the specific research purposes, questions, or hypotheses logically flow from the introductory material?
7. Overall, is the introduction effective and appropriate?
8. Is current research cited?

 

 

Evaluating Samples When Researchers Generalize

 

1. Was random sampling used?
2. If random sampling was used, was it stratified?
3. If the randomness of a sample is impaired by the refusal to participate by some of those selected, is the rate of participation reasonably high?
4. If the randomness of a sample is impaired by the refusal to participate by some of those selected, is there reason to believe that the participants and non-participants are similar on relevant variables.
5. If a sample from which a researcher wants to generalize was not selected at random, is it at least drawn from the target group for the generalization?
6. If a sample from which a researcher wants to generalize was not selected at random, is it at least reasonably diverse?
7. If a sample from which a researcher wants to generalize was not selected at random, does the researcher explicitly discuss this limitation?
8. Has the author described relevant demographics of the sample?
9. Is the overall size of the sample adequate?
10. Is there sufficient number of participants in each subgroup that is reported on separately?
11. Has informed consent been obtained?
12. Are there probable biases in sampling (e.g., volunteers, high refusal rates, institution population atypical for the country at large, etc.)?
13. Overall, is the sample appropriate for generalizing?

 

 

Evaluating Samples When Researchers Do Not Generalize

 

1. Has the researcher described the sample/population in sufficient detail?
2. For a pilot study or developmental test of a theory, has the researcher used a sample with relevant demographics?
3. Even if the purpose is not to generalize a population, has the researcher used a sample of adequate size?
4. If a purposive sample has been used, has the researcher indicated the basis for selecting individuals to include?
5. If a population has been studied, has it been clearly identified and described?
6. Has the researcher obtained informed consent?
7. Are there probable biases in sampling (e.g., volunteers, high refusal rates, institution population atypical for the country at large, etc.)?
8. Overall, is the description of the sample adequate?

 

Evaluating Instrumentation

 

1. Have the actual items, questions, and/or directions (or, at least, a sample of them) been provided?
2. Are any specialized response formats and/or restrictions described in detail?
3. For published instruments, have sources where additional information can be obtained been cited?
4. When delving into sensitive matters, is there reason to believe that accurate data were obtained?
5. Have steps been taken to keep the instrumentation from obtruding on and changing any overt behaviors that were observed?
6. If the collection and coding of observations is highly subjective, is there evidence that similar results would be obtained if another researcher used the same measurement techniques with the same group at the same time?
7. If an instrument is designed to measure a single unitary trait, does it have adequate internal consistency?
8. For stable traits, is there evidence of temporal stability?
9. When appropriate, is there evidence of content validity?
10. When appropriate, is there evidence of empirical validity?
11. Is the instrumentation adequate in light of the research purpose?
12. Overall, is the instrumentation adequate?

 

 

Evaluating Experimental Procedures

 

1. If two or more groups are compared, were individuals assigned at random to the groups?
2. If two or more comparison groups were not formed at random, is there evidence that they were initially equal in important ways?
3. If only a single participant or a single group is used, have the treatments been alternated?
4. Are the treatments described in sufficient detail?
5. If the treatments were administered by people other than the researcher, were these people properly trained?
6. If the treatments were administered by people other than the researcher, was there a check to see if they administered the treatments properly?
7. If each treatment group had a different person administering a treatment, has the researcher tried to eliminate the “personal effect”?
8. Except for differences in the treatments, were all other conditions the same in the experimental and control groups?
9. If necessary, did the researchers disguise the purpose of the experiment from the participants?
10. Is the setting for the experiment “natural”?
11. Has the researcher used politically acceptable and ethical treatments?
12. Has the researcher distinguished between random selection and random assignment?
13. Overall, was the experiment properly conducted?

 

Evaluation Methods

 

1. What is/are the research hypothesis/es?
2. Is the method so described that replication is possible without further information?
3. What data was collected? Were the validity and reliability of the instrument addressed?
4. What was the level of measurement of each distinct category of data?
5. What was (were) the independent variable(s)?
6. What was (were) the dependent variable(s)?
7. What statistical tests were used in the procedure?
8. Were the statistics used with appropriate assumptions fulfilled by the data (e.g., normalcy of distribution for parametric techniques)?
9. What was the null hypothesis for each statistical test?
10. What were the results of each statistical test?
11. Did the statistical procedure answer the research hypothesis/questions?
12. Have statistical significance levels been accompanied by an analysis of practical significance levels?
13. Are the figures and tables (a) necessary and (b) self-explanatory?  Large tables of non-significant differences, for example, should be eliminated if the few obtained significances can be reported in a sentence or two in the text. Could several tables be combined into a smaller number?

 

 

Evaluating Results

 

1. Is the results section a cohesive essay?
2. Does the researcher refer back to the research hypotheses, purpose, or questions originally stated in the introduction?
3. When there are a number of statistics, have they been presented in table form?
4. If there are tables, are their important aspects discussed in the narrative of the results section?
5. Have the researchers presented descriptive statistics before presenting the results of inferential tests?
6. If any differences are statistically significant and small, have the researchers noted that they are small?
7. Have appropriate statistics been selected?
8. Overall, is the presentation of the results adequate?

 

 

 

 

 

 

Evaluating Discussion

 

1. In long articles, do the researchers briefly summarize the purpose and results at the beginning of the discussion?
2. Do the researchers acknowledge their methodological limitations?
3. Are the results discussed in terms of the literature cited in the introduction?
4. Have the researchers avoided citing new references in the discussion?
5. Are specific implication discussed?
6. Are suggestions for future research specific?
7. Have the researchers distinguished between speculation and data-based conclusions?
8. Overall, is the discussion effective and appropriate?

 

 

Putting It All Together

 

1. Have the researchers selected an important problem?
2. Were the researchers reflective?
3. Is the report cohesive?
4. Does the report extend the boundaries of our knowledge on a topic?
5. Are any major methodological flaws unavoidable or forgivable?
6. Is the research likely to inspire additional research?
7. Is the research likely to help in decision making (either of a practical or theoretical nature)?
8. All things considered, is the report worthy of publication in an academic journal?
9. Would you be proud to have you name on the report as a coauthor?