Evidence Type | Possible Methods of Evaluation |
Is there any evidence? | If not, ask for evidence. Reject the claim if no evidence is produced. |
If yes, determine the type of evidence and evaluate its credibility. | |
Anecdotes/observations | Has the observation(s) been recorded so it can be verified? |
Can the observation be verified through replication? | |
Is the observation falsifiable? ○ If the observation cannot be repeated, is there corroborating evidence? | |
What biases could have influenced the observer’s objectivity? | |
Associations | Evaluate whether the cause must logically precede the claimed effect. |
Have studies or observations with different measures and populations found the same results? | |
| What third variables might explain the association? |
Have third variables been controlled with multiple regression, quasi-experimental designs, or other methods? | |
For evidence with statistical results, ask whether a unique finding is reported, whether power is low, and whether the obtained probability is close to 0.05. Presence of all three criteria indicate a low probability of replication. | |
Is there evidence of bias in design or interpretation? | |
Experiments with Random Assignment | Identify the manipulated and measured variables so the direction of causality is clear. Was the obtained p-value close to 0.05, indicating low power and difficulty in replication? |
Does the study show a surprising result, or does it fit with pattern of prior similar results? | |
Has the finding been replicated? | |
Each replication adds to the strength of evidence the study represents. | |
Is there evidence of bias in design or interpretation? | |
Evaluate the strength of the causal argument and evidence. | Is it possible to identify a pattern of evidence, i.e., several studies/observations with different methods that support the claim? Does a meta-analysis show support for the causal claim? |
Does the evidence include a replicated study with random assignment, given more weight in the argument? | |
Studies of association require a greater number and variety of studies to validate the claim. Does the evidence include failed predictions which undermine the causal claim? Does the evidence include successful predictions which strengthen the causal claim? | |
Are there alternate interpretations of the findings that need to be tested? | |
Examine the findings for signs of bias in the design or interpretation. |