Step 4: Evaluate the research evidence

Evaluate the research evidence:
Evaluate the reasoning that ties the evidence to the key point
Do the conclusions go beyond the evidence?
How good is the research evidence?
What kinds of problems are there with the research evidence?
Are there problems with the reasoning relevant to the research evidence?

Identify the source (citation)
Summarize the way the study was done (description of study)
The research results
Decide if you have a psychological fact or incomplete research evidence (Even when you label a sentence as psychological fact some important info may still be missing. )

Citation:

Who conducted the research and when?
How recent is it?
Is there more current research that might have modified the study?
Is any related research sited?

Subjects:

Who are the subjects?
What are their characteristics? Were different age groups studied?
How many subjects were observed? If only a few subjects were observed the results will likely produce not typical results and might produce misleading and unusual conclusions. Were too few subjects studied? Experiments usually use the smallest number of subjects. Surveys need more subjects to generalize to populations.
How were they selected from the population? Was the selection procedure unbiased? If random sampling is used is it explained? If a random process was not used who does the sample represent?
Are the subjects relevant to the central idea and key points?
Has the writer overgeneralized the results? From animals to humans? From college students to all humans? From males to females?

Procedure:

How was the study done?
How were the observations made?
What were the subjects told?

Dependent and independent variables

How were these variables defined and measured?
What factors have been controlled to produce reliable results
Was there a control group? If yes how were the subjects assigned to the experimental and control groups? Random assignments avoid bias.
Is the independent variable precisely defined?
Is the dependent variable precisely defined?
What are the limited circumstances?

Are the psychological terms defined?
everyday terms have a different meaning when used by psychologists. Writers of psych topics should define the terms. If the writer does not are the terms used consistently in the source. Look up unfamiliar terms in a psych dictionary.

Constructs - terms that refer to a variety of behaviors: Aggression, helping, cognitive dissonance, mental illness, intelligence and learning are broad psych terms called constructs. Constructs are used to pull related research for better understanding
Operational Definitions: Since constructs are broad terms they cannot be studied directly and the construct must be refined to state a specific aspect of that behavior. Researchers must define exactly what they are going to observe and how they will measure behavior. If researchers do not use operational definitions they might not be measuring the same behavior every time and other researchers will not know exactly what behavior was observed. Example:

Construct definition: Aggression is intended to harm another (Broad)
Operational definition: Aggression is the behavior of person A (the subject) pushing a button to deliver an electric shock to person B whenever B gives the wrong answer (Specific)

Possible problems dealing with procedure:

If there is control group - no cause and effect statement can be made. Misleading conclusion can easily occur when there is no comparison group
Was a reasonable control group used? Was the best control condition used? or should several be used?
Were subjects randomly assigned to the control groups? If not results might be due to differences in the group rather than the independent variable
If other variables were not controlled they may produce the observed change in behavior and not the independent variable
Did subjects get the identical independent variable
Was the measurement of the dependent variable consistent across subjects? If it is not reliable then the resulting variability can influence the results.

Research results:

Are the results clearly and fully presented?
How big is the difference among groups?
Is it a statistically significant difference?
Are specifics given?
Are results given for each group studied?
Is there an indication that the results were statistically significant
Are the results based on reported observations?
Are the results consistent with other research results on the topic?

Reasoning:

How is the psych evidence tied in with central idea an key points? Is it relevant?
Does the writer draw the correct conclusion from evidence?
Does writer generalize beyond evidence? (important to look at who the subjects were and how they were selected.
Was control condition used for comparison? If not approach research with great caution
Was a cause and effect relationship based on research evidence other than an experiment? A correlation does not mean a cause has been isolated
A change is stated but no measurements were made before introduction of the independent variable
A survey was conducted but the sample was not randomly selected from population.

	Created by Jc Kunkel almost 8 years ago

Notes on text book Evaluating Psychological Information