On the (Ir)Relevance of Psychological Research: Students versus Scientists and Implications for Teaching

Research articles are widely used in the training of undergraduate students. Editors and reviewers of the top scientific psychology journals influence the development in the field by publishing certain articles and rejecting others, probably assuming that the published articles are empirically sound and theoretically highly relevant. The current study investigated if published articles are indeed regarded as relevant by a sample of 393 psychology undergraduate students from a university in the Southeast of the United States. The students’ age ranged between 18 and 57 (M = 23, SD = 6.05) and 84% were female. Students received brief statements about potential research studies and rated them regarding relevance, not knowing that the summaries were from actual research studies published in peer-reviewed journals. Results showed that (1) overall, research articles were regarded as generally irrelevant, (2) applied articles were regarded as more relevant than basic research articles, (3) ratings did not differ based on gender or age, and (4) the more advanced students were in the Psychology program, the higher their relevance ratings were for applied research as compared to basic research. Results are comforting or disturbing; comforting, because students might not have the professional expertise to make such relevance judgments; disturbing, because results might indicate how specialized and insulated journals have become by not addressing topics relevant to a wider population. Results also have implications for teaching research methods and experimental psychology courses.


Introduction
Psychology is largely dependent on the advancement and publication of research in basic and applied areas.Basic psychological research uses scientific methods to investigate fundamental laws of mind and behavior.The goal of basic psychological research is to expand theoretical knowledge and might not be directly applicable to the real world; whereas, applied psychological research focuses on solving practical real-world problems (Bornstein & Meissner, 2008).We would like to acknowledge that this distinction of basic and applied psychological research is somewhat artificial and controversial, yet it is very common (e.g., Hoffman & Deffenbacher, 1993).Both types of research are frequently critiqued on their novelty, contributions to the field, and relevance (e.g., Gergen, 2001).Approximately, 10-20% of manuscripts submitted to the top journals in psychology are published (American Psychological Association, 2013).Thus, one would assume, most of these are innovative and relevant research studies which have a big impact on the field.
According to Becher (1994) disciplines are the primary determinant of membership and identification in academics.However, within specific fields, there are often conflicts over the importance and value of research outcomes.For example, researchers in both applied and experimental psychology will observe specific phenomena using varying approaches and frames of reference.Differences such as these inevitably influence students' beliefs about a field of study.The purpose of this study is to investigate whether students' views about research's contribution to science differ from that of editors and reviewers.
Constructivist theory asserts that an individual's current knowledge structure is constructed based upon context and the interpretation of previous experience (Resnick, 1991).Cognitive constructivist theory is concerned with active learning (Garrison, 1993); whereby, the learner creates meaning and understanding through dialogue with others.Therefore, it could be expected that psychology students are socialized into the research process of their chosen field of psychology through course assignments and lectures, and thus would share the values and beliefs of their professors.However, many students differ from faculty on topics of interest and their "appreciation" of courses focused on research methods and experimental design, which faulty members rated as best to instill scientific values in students (McGovern & Hawk, 1986).
According to Manning, Zachar, Ray, and LoBello (2006) students enter psychology with the common misperception that psychologists are clinicians or practitioners, rather than researchers.Due to this misperception, many students are overwhelmed by the concept of research and believe it is beyond their academic capabilities (Allen, Dorozenko, & Roberts, 2016).In fact, Manning el al. (2006) found that after being exposed to a research methods course, students lost interest in such scientific activities.Yet, skills related to research methods, such as problem-solving and critical thinking, are among the top 10 rated skills by students and employers (see Miller & Carducci, 2015).
Perhaps, as students are exposed to more psychology courses, they become further polarized and develop an interest in either applied or basic scientific activities, but usually not both.Holmes and Beins (2009) found that the 201 psychology students in their study reported greater interest in practitioner activities when responding to the Scientist-Practitioner Inventory.
Collectively, these studies suggest philosophical and practical differences between psychology students and instructors in regard to their view of psychology as a discipline.In the present study, we investigated student interest in basic and applied research topics in psychology, the relevance of these research topics to the world at large, and whether such studies are worthy of research/grant funding.Due to students' lack of appreciation for basic research, which is primarily learned in their more difficult methodology courses, it is hypothesized that students would rate applied research topics as more relevant and worthy of funding than basic psychology research topics.

Participants
The participants in this study were undergraduate and graduate psychology students from a Southeastern University in the United States.Participants were recruited via the university's online research database and received course credit for participation.Due to the demanding nature of the survey and to avoid fatigue, the survey was divided into even and odd-numbered sections, thus two separate surveys (each with 26 statements/questions) were used in this study.First, 25 participants of the 450 students who had fewer than 6 responses for the 26 questions were removed from the dataset.Then 32 participants who had answered more than 20 out of 26 questions in less than 3 seconds were removed.Thus, our sample consisted of 190 (of originally 200) participants for the odd-numbered survey and 203 (of originally 225) participants for the even-numbered survey.The analyses were conducted on the remaining 393 participants.The sample was 84% female and 16% male with ages ranging from 18-57 (M = 23, SD = 6.05).Regarding years in college, 8.7% were freshmen, 12.1% were sophomores, 42.6% were juniors, 35.9% were seniors, and 0.8% were graduate students.We did not include the graduate students in any of the following analyses.

Instruments
To assess the relevance of psychological research topics, two surveys were used, both consisting of 26 questions.Each question summarized a research topic in one sentence (see Appendix A).We randomly selected empirical articles from 13 journals.From each of the 13 journals, we randomly assigned four articles that were published between 2010 and 2013.We then summarized the research into one key research question including the key variables studied for each of the 52 articles selected (see Appendix).We attempted to be as close to the wording of the authors as possible.Initially, one undergraduate research assistant and the first author randomly selected the articles for each of the 13 journals.Then the undergraduate student who was not familiar with this study's hypotheses summarized the main research study hypothesis in one statement.Sometimes this statement did not include all the many variables assessed in a specific study, but our main goal was to summarize the main topic of the study in an understandable way.Then the first author read the statements without knowledge of the articles' content.Several statements did not seem clear and then were revised by the undergraduate student and first author together.
The research questions presented in the surveys were collected using 13 academic journals with the highest impact factors in their respective fields.Health Psychology, Journal of Abnormal Psychology, Law and Human Behavior, Organizational Behavior and Human Decision Making Processes).Following each summarized research question, a set of three questions utilizing a 5-point Likert Scale (1 = not at all; 5 = very much) were used to evaluate the understandability of the research question, the perceived relevance of the research, and the likelihood of funding to allocate for such research ("Is the research question understandable?""Is this a relevant research topic?"Relevance is here understood as meaningful and important in their view."Should funding be allocated for conducting this research?"

Procedure
Both versions of the survey were posted on an online system.All guidelines of the Institutional Review Board were followed and participants received first an Informed Consent Form outlining the purpose of the study and the rights of participants.Participants' privacy was protected.Participants were randomly assigned to one of the two survey versions.They saw one journal article summary statement at a time with the three questions below the statement.Once they selected their answers on the three Likert scales they clicked to proceed to the next journal article summary statement.They were not allowed to go back to previous statements and answers.At the end, they completed the demographic questions.Filling out one of the two surveys took an average of 20 minutes.

Data Analysis
Upon completion, the responses for each of the surveys were entered in Microsoft Excel.Answers that were given for the surveys in less than 3 seconds were removed from the dataset.Responses were removed because it was believed that at least 3 seconds were needed to read, process, and answer each question.Also, participants who had answered the understandability question with a "1" indicating that a particular research topic was not at all understandable, were also removed from the dataset (only for that particular question).We removed 213 answers marked with a "1" for not understandable which is 2% of all answers.We removed these participants because a research topic could not be adequately judged as relevant or worthy of funding if it was not understandable.

Comparison of Mean Ratings of the 13 Research Journals
The means for "Is the research question understandable?"ranged between 3.65 and 4.29 (Overall M = 3.99, SD = 0.50).The lowest mean was for Cognitive Psychology, (M = 3.65).The highest mean was for the Journal of Abnormal Psychology, (M = 4.29).The mean relevance ratings ("Is this a relevant research topic?") ranged between 3.04 and 4.18 (Overall M = 3.61, SD = 0.52).The two lowest means were those for Psychology and Aging (M = 3.24), and Journal of Experimental Psychology General, (M = 3.04).The highest mean was for the Journal of Abnormal Psychology, (M = 4.18).The means for "Should funding be allocated for conducting this research?"ranged between 2.80 and 3.90 (Overall M = 3.39, SD = 0.62).The lowest mean was for Journal of Experimental Psychology -General, (M = 2.80).The highest mean was for the Journal of Abnormal Psychology, (M = 3.90).
We calculated correlations based on "understandability" and "relevance" for the overall means of the 13 journals.The Pearson correlations ranged between .40 and .69,all ps < .001.We calculated correlations based on "understandability" and "funding" for the overall means of the 13 journals.The Pearson correlations ranged between .33 and .63,all ps < .001.We calculated correlations based on "relevance" and "funding" for the overall means of the 13 journals.The Pearson correlations ranged between .76 and .89,all ps < .001.These correlations show that higher ratings in understandability go hand in hand with higher ratings in relevance and funding.
Figure 1.The mean Relevance ratings for the six "applied" research journals to the left were significantly higher than the means of the seven "basic" research journals to the right

Comparison of Mean Understandable Ratings, Mean Relevance Ratings, and Mean Funding Ratings for Basic and Applied Research Journals
We conducted a paired samples t-test to compare the overall mean understandability ratings for basic (M = 3.92, SD = .56)and applied research journals (M = 4.01, SD = .57).As hypothesized, the mean understandability ratings for applied research journals were significantly higher than those for the basic research journals, t(388) = -4.99,p < .001.
Figure 1 shows the seven basic research journal ratings to the right and the six applied research journal ratings to the left.We conducted a paired samples t-test to compare the overall mean relevance ratings for basic (M = 3.50, SD = .58)and applied research journals (M = 3.74, SD = .59).As hypothesized, the mean relevance ratings for applied research journals were significantly higher than those for the basic research journals, t(388) = -9.65,p < .001.
We also conducted a paired samples t-test to compare the overall mean funding ratings for basic (M = 3.26, SD = .67)and applied research journals (M = 3.53, SD = .67).As hypothesized, the mean funding ratings for applied research journals were significantly higher than those for the basic research journals, t(388) = -9.96,p < .001.

Comparison of Mean Relevance Ratings and Mean Funding Ratings for Gender, Age, and Class Standing
Independent samples t-tests showed no gender differences for any of the 13 mean relevance ratings or any of the 13 mean funding ratings, nor in the overall mean relevance ratings for "basic" journals, t(383) = -.13 p = .90,and "applied" journals, t(383) = -.38 p = .71.
There was no significant relationship between age and mean relevance ratings in any of the 13 journals, except for the Journal of Abnormal Psychology, r = .11,p = .03.There was no significant relationship between age and mean funding ratings for any of the 13 journals, except for the Journal of Abnormal Psychology, r = .16,p = .002,and the Psychology and Aging, r = .12,p = .02.Considering Bonferroni adjusted p-values, even these correlations would not be significant anymore.
Regarding years in college, 8.7% of the sample were freshmen, 12.1% were sophomores, 42.6% were juniors, 35.9% were seniors, and 0.8% were graduate students.Again, we excluded the 4 graduate students.A mixed between-within subject's ANOVA was conducted to investigate the effects of class standing on mean relevance ratings for basic and applied research journals.There was a significant interaction between class standing and journal type, Wilks' Lambda = .97,F(3, 383) = 3.82, p = .01,partial eta squared = .03.The more advanced students were, the higher their mean relevance ratings for applied research journals compared to their ratings for basic Mean Relevance Ratings for the 13 Journals research journals (see Figure 2).
Figure 2.There was a significant interaction between class standing and journal type.The more advanced students were, the higher were their mean relevance ratings for applied research journals compared to their ratings for basic research journals There was a significant main effect for journal type, Wilks' Lambda = .91,F(1, 383) = 39.12,p = .01,partial eta squared = .09,but no significant effect for class standing, F(3, 383) = .08,p = .97,partial eta squared = .001.Due to the significant interaction, interpretation of main effects is not meaningful.
Then we conducted a mixed between-within subject's ANOVA to investigate the effects of class standing on mean funding ratings for basic and applied research journals.There was no significant interaction between class standing and journal type, Wilks' Lambda = .98,F(3, 383) = 2.25, p = .08,partial eta squared = .02.There was a significant main effect for journal type, Wilks' Lambda = .89,F(1, 383) = 45.26,p < .001,partial eta squared = .11,but no significant effect for class standing, F(3, 383) = .66,p = .58,partial eta squared = .005.

Discussion
The present study investigated the relevance of articles published in top scientific psychology journals, according to students.The results showed low overall relevance ratings and low overall suggested funding ratings, which is surprising considering that the topics are research published in the top psychology journals.One would expect relatively high means, but the means for relevance were between 3 and 4 for 12 of the 13 journals and between 2.8 and 4 in the 13 journals for funding ratings.Overall, students are not convinced about the particular relevance of these published articles.We do not believe this result could be due to a particular response tendency of students, for example avoiding the extreme scores on the Likert scale.After every semester, for example, students rate their professors' teaching performance on Likert scales, and students are not shy to select "4"s or "5"s on these Likert-scales ranging from 1 to 5 for many of their professors (see also Stanny & Arruda, 2017, for empirical data).
Additionally, students rate the relevance and suggested funding for the applied journal articles higher than those for the basic articles, which supports our hypothesis.This result is consistent with prior studies which have also shown greater practitioner interests among psychology students at both the undergraduate and graduate levels of education (Holmes & Beins, 2009).In addition to this finding, students with greater class standing rated applied psychology research topics as more relevant than basic research topics.This could be partly due to the fact that they are orienting themselves towards the workforce and are searching for a job and are therefore more confronted with the application of psychology.No gender and age differences were found regarding relevance and suggested funding ratings.
One limitation of the present study is the inclusion of only 13 academic journals.Perhaps, students' main areas of interests were omitted from the survey and therefore the research questions were viewed as relatively uninteresting.Furthermore, it is difficult to summarize a whole study in one sentence.Students might have interpreted the question about relevance in different ways: relevant to them personally or relevant to the field as a whole.Also, due to the survey sample consisting of undergraduate students, the students may have lacked the exposure to course material necessary to fully comprehend each research question.Although, students' self-reports indicated they "understood" the research questions and those responses that indicated no understanding of the research question were eliminated.
Future research could focus on evaluating student interests when first entering the discipline and measure changes in interests and experiences as the student progresses through their respective programs.By measuring students longitudinally, changes in interests, motivations, career aspirations, and attitudes after exposure to more rigorous research-based courses -and potentially controlling for students' research experience -can be more accurately assessed.
Additionally, researchers may also want to focus on creating effective methods for increasing student interest in the scientist activities performed by researchers.For example, instructors should openly express their research interests as well as current projects that students could become involved in because approaching faculty to ask about research can be a difficult task for many students.Furthermore, instructors may also want to actively promote local undergraduate and graduate research confrences, as this would allow students the opportunity to network with potential mentors and learn about the research process from theperspective of other students.

Conclusion
To conclude, results could be interpreted as either comforting or as disturbing; comforting, because students may not have the background yet and will later realize which areas are important and that the articles published in top psychology journals tend to target only a specific small audience; disturbing, because results might show how specialized and insulated the top journals have become by not addressing topics important to society.Have researchers, reviewers, and editors become so specialized and restrictive in their respective domains, that they are losing sight of what truly constitutes relevant research topics?
A second implication is related to the training of psychology undergraduates.Results of the current study and the previous research discussed show the difficulties undergraduate students have with empirical courses such as research methods and experimental psychology, regardless of their gender and age (see also Balloo, Pauli, & Worrell, 2018).Students, as the current findings have shown, are more interested in applied topics even in their junior and senior years.It is the responsibility of professors to make those empirical courses focusing on basic research more interesting by, for example, relating research methods to topics that are relevant and interesting for the students.
accelerated sexual development, promiscuity, and sexual risk taking?
• Can people's mental representation of progress function as a self-regulation mechanism that helps motivate continued effort towards a goal?-More precisely: Do individuals who have just started pursuing a goal think they are further ahead and individuals who are close to the goal think they are further away?Can this mental representation be a means of motivating themselves to elicit greater effort?
• Can aggression toward one's romantic partner be predicted by provocation severity and low relationship commitment?

Child Development (CD)
• Depending on their representations of attachment (secure-autonomous vs. insecure-dismissing), how do first-time pregnant women physiologically (heart rate, respiration rate) react to their own infant's cry?
• Do children when they see two dolls interacting show more situation attributions or more person attributions for behavior of the dolls?
• Do early developmental variations in the rate at which children acquire words help predict later vocabulary skill?
• Are there language-specific developmental differences comparing the speech productions of Englishand Japanese-speaking children to the speech produced by adult speakers?

Psychology and Aging (PA)
• Is mind-wandering frequency of older adults and sustained attention on a task related to falls?
• What are the relationships among social network (such as size and frequency of contact), social activity, emotional support, and different aspects of health and subjective well-being in older adults across 6 years of development?
• Can a training program increase openness to new experiences in later adulthood?And is such a training effect higher for participants with high control beliefs?
• Are older as compared to young adults less able to avoid misleading effects of context in a task where they "saw" a briefly flashed word that was preceded by a prime (Dirt -dirt or dart -dirt)?

Journal of Experimental Psychology -General (EP)
• Does the experience of social rejection stimulate creativity for people with an independent self-concept?
• Is an item in the focus of attention subject to semantic interference?Can we show through priming semantically related or unrelated words that the time to think of a cue will either be increased or decreased?
• Does positive affect broaden and does negative affect narrow the attentional focus?
• Are odd numbers associated with masculinity and even numbers associated with femininity? 7. Psychological Science (PS) • Do people's perceptions that they live in a harsh environment (e.g., an economic crisis) influence their food choices, i.e. they choose more high-calorie foods?
• Companies often provide incentives for employees to maintain healthy lifestyles.These incentives can take the form of either discounted health insurance premiums for healthy-weight employees ("carrot" policies) or increased premiums for overweight employees ("stick" policies).Do these different forms of incentives convey the same information to employees?
• Does an intervention which helps parents convey the importance of mathematics and science courses to their high school-aged children lead these children to take more mathematics and science courses in high school?
• Early in development, do linguistic labels function as category markers (e.g., dog) or are they no more than the features of the stimuli they present (e.g., barks, fur) before becoming category markers in the course of development?
Six of the journals focused mainly on basic research (Social Cognitive and Affective Neuroscience, Cognitive Psychology, Journal of Personality and Social Psychology, Child Development, Psychology and Aging, Journal of Experimental Psychology -General, Psychological Science) and seven journals focused on applied research (Journal of Applied Psychology, Journal of Consulting and Clinical Psychology,