Evidence Type

Possible Methods of Evaluation

Is there any evidence?

Ÿ If not, ask for evidence. Reject the claim if no evidence is produced.

Ÿ If yes, determine the type of evidence and evaluate its credibility.

Anecdotes/observations

Ÿ Has the observation(s) been recorded so it can be verified?

Ÿ Can the observation be verified through replication?

Ÿ Is the observation falsifiable?

○ If the observation cannot be repeated, is there corroborating evidence?

Ÿ What biases could have influenced the observer’s objectivity?

Associations

Ÿ Evaluate whether the cause must logically precede the claimed effect.

Ÿ Have studies or observations with different measures and populations found the same results?

Ÿ What third variables might explain the association?

Ÿ Have third variables been controlled with multiple regression, quasi-experimental designs, or other methods?

Ÿ For evidence with statistical results, ask whether a unique finding is reported, whether power is low, and whether the obtained probability is close to 0.05. Presence of all three criteria indicate a low probability of replication.

Ÿ Is there evidence of bias in design or interpretation?

Experiments with Random Assignment

Ÿ Identify the manipulated and measured variables so the direction of causality is clear.

Ÿ Was the obtained p-value close to 0.05, indicating low power and difficulty in replication?

Ÿ Does the study show a surprising result, or does it fit with pattern of prior similar results?

Ÿ Has the finding been replicated?

Ÿ Each replication adds to the strength of evidence the study represents.

Ÿ Is there evidence of bias in design or interpretation?

Evaluate the strength of the causal argument and evidence.

Ÿ Is it possible to identify a pattern of evidence, i.e., several studies/observations with different methods that support the claim?

Ÿ Does a meta-analysis show support for the causal claim?

Ÿ Does the evidence include a replicated study with random assignment, given more weight in the argument?

Ÿ Studies of association require a greater number and variety of studies to validate the claim.

Ÿ Does the evidence include failed predictions which undermine the causal claim?

Ÿ Does the evidence include successful predictions which strengthen the causal claim?

Ÿ Are there alternate interpretations of the findings that need to be tested?

Ÿ Examine the findings for signs of bias in the design or interpretation.