Chatbot Development as a Digital Learning Tool to Increase Students’ Research Knowledge

The research aimed to: 1) develop the chatbot; 2) evaluate its effectiveness; and 3) investigate its effects on students’ research knowledge. The sample consisted of 36 Thai university students. The research instruments consisted of: 1) the chatbot; 2) an evaluation form; 3) an effectiveness questionnaire; and 4) research tests. Data analysis used was mean, standard deviation, content analysis and a t-test. The findings indicated that: 1) the chatbot was evaluated by experts with the applicability at a very high level ( = 4.67, S.D. = 0.08) with recommendation to add more research content and interactive learning. The pilot test was done with 14 non-target group of students. Students perceived the chatbot’s effectiveness at a high level ( = 4.43, S.D. = 0.35) with comments to add more examples and graphics to make the chatbot more interesting; 2) the 36 target group of Thai university students perceived the chatbot as an effective technology to use as a digital learning tool at a high level ( = 4.37, S.D. = 0.48). They thought that chatbot technology was easy to use, easy to understand, innovative and fun for learning. They could get answers instantly and be able to seek specific information without waiting for responses. However, in response to questions not matched keywords specified, further details of finding proper answers such as links should be provided instead of leaving those questions unanswered. Also, the chatbot only provided responses when typing correctly so there should be an option to choose from a list of questions or keywords; and 3) the post-test scores were significantly higher than the pre-test scores at the 0.05 level of significance. In conclusion, using chatbot technology in education settings to increase students’ research knowledge gave positive results as it led to positive learning outcomes and helped provide better personalized learning experience for students.


Introduction
In educational settings, the role of research has greatly increased in modern times (Kothari, 2004) because research is studied in some subjects and sometimes required by educational institutions as graduation criteria. However, the lack of basic research knowledge on research skills may cause students to feel inadequate or result in serious academic difficulties and not success in study (Sutasuwan, Sumalee, & Supsombat, 2016). Therefore, it is important to understand basic knowledge of research so that the translation of that research knowledge into research skills will be easily facilitated.
There are many possible means with technology support that can be integrated as a digital learning tool to help enhance student's research knowledge through personalized learning support due to the fact that technology for education nowadays is changing how students learn (Singh, 2018) and it can help students learn faster with a wider perspective (Havenstein, 2008), increase teaching's effectiveness (Ellison, Steinfield, & Lampe, 2007), offer guidance and use as information retrieval tool to give a positive and sometimes fun digital learning experience to students who have good adoption of technology-based learning.
Chatbots are considered as one of the latest and trendy learning technologies with artificial intelligence. A chatbot is programmed with text messages or speeches to simulate human conversation in a scripted way (Rouse, 2017;Hettige & Karunananda, 2015;Imran, 2015;Heller & Procter, 2010;Dryden, 2008;Wang, 2008;Bayan & Atwell, 2007;Heller, Procter, Mah, Jewell, & Cheung, 2005;Weizenbaum, 1975), engages with a computer, often, over the internet through text messages or speeches (Xu, Liu, Guo, Sinha, & Akkiraju, 2017), and can be trained with any language or some jokes to create fun learning. Only typing questions or keywords are needed to receive answers or explanation instantly (Shawar, 2005). Two types of chatbots are popular. Firstly, it is a rule-based or retrieval-based chatbot that uses preprogrammed responses to match with the text messages or speeches inputted. Secondly, it is a generative chatbot that generates as suitable a response as possible from the input through Natural Language Processing (NLP) and deep machine learning (Liu et al., 2016;Nieves, 2018).
With integrated artificial intelligence, chatbots can be used as a digital learning tool to ask questions, give answers to questions, retrieve information (Shawar & Atwell, 2007), visualize the contents (Bayan, 2005), explore online content (James, 2016), provide useful information (Brandtzaeg & Følstad, 2017), establish the knowledge that the students has to learn (Wenger, 1987), and propose possible solutions to the students individually (Singh, 2018) because several studies have shown that poor personalized support can lead to weak student learning outcomes while good personalized support can improve student learning outcomes (Pane et al., 2017;Hone & El Said, 2016;Brinton et al., 2015;Eom, Wen, & Ashill, 2006). Therefore, educational institutions should utilize chatbots to provide interactive e-learning environment for students (Kowalski, Hoffman, Jain, & Mumtaz, 2011;Bii, 2013) and sometimes fun learning. Responses to frequently asked questions can be programmed and instant responses can be provided (Cameron et al., 2017) while making it easier and more comfortable for students to ask questions and obtain immediate answers to questions. As a result, chatbots are able to support the learning process to be more engaging, interesting and entertaining for the students while facilitating the teaching process (Singh, 2018) to free up teachers from responding to repetitive questions and act as a virtual assistant or tutors (Garcia-Brustenga, Fuertes-Alpiste, & Molas-Castells, 2018) for students.
Using chatbot technology has many potential benefits. The advantages of using chatbots include low cost, less time in obtaining answers, better interaction, creative learning and improved efficiency when using in instruction (Llic & Markovic, 2016;Bii, 2013) because users find chatbots safe and easy to chat online (Cameron et al., 2017) with the ability to operate as a 24/7 support service, provide responses to repetitive or frequently asked questions that can easily resolved, and give access to learning contents (Garcia-Brustenga et al., 2018;Winkler & Söllner, 2018) when required. In addition, students can refresh their memory by using chatbots to help recall, revise and remember the knowledge studied. With chatbots, timely and efficient assistance or information can be obtained with reported motivation and curiosity in their entertaining, social and relational factors. Students also view chatbots as a novel phenomenon.
Furthermore, chatbots can play the role of teaching guide (Silvervarg, Kirkegaard, Nirme, Haake, & Gulz, 2014) and assistant throughout the learning process with wide range of functions of obtaining information for students, giving knowledge and enhancing its understanding with uninterrupted availability if learning through chatbot technology is properly designed. The teachers can also use questions asked to collect data, modify a knowledge base, and expand more knowledge by using chatbot technology to look for questions and add additional answers to those questions asked in its knowledge base. Most students prefer using chatbot technology because chatbots are able to give direct answers instead of links for further searching like using search and sort-based tools (Shawar & Atwell, 2007;Shawar, 2005).
Nowadays, chatbots can work on Facebook's instant messaging platform that is installed on most smart phones (Stalista, 2017) and becomes the main network where students and teachers have their own account users with no need to download any application and more comfort to use the application that they are already familiar with (Lachs, 2017;Singh, 2018;Preedasutti, 2010). Apart from that, it is not necessary to have programming knowledge to use chatbot technology as chatbots can recognize the language inputted by users, and recognize their intent to provide relevant responses according to the input (Cameron et al., 2017).
Chatbots have significant educational potential and positive impact on student learning and satisfaction through their personalized learning support (Winkler & Söllner, 2018). Although there were numerous studies related to successful implementation of chatbots (Dutta, 2017;Huang, Lee, Kwon, & Kim, 2017;Kerly, Hall, & Bull, 2007), only few of them have been used for educational purposes (Kowalski et al., 2011). At present, the use of chatbots in education was scarce as not much research has been done on the utilization of chatbot technology in classrooms (Baker, 2016;Goos et al., 1998;Bayan & Atwel, 2007;Gimeno, 2008;Wang, 2008;Torma, 2011;Govindasamy, 2014;Osodo, Indoshi, & Ongati, 2010). In Thailand context, there were some researches done on the use of chatbots to provide customer service (Santirattanaphakdi, 2018), to guide a system (Bungodchai, 2017), to act as performer agents (Lerdsahapan, 2015) and to diagnose a disease (Mokarat, Unchai, & Marpae, 2016) but there is little research being done in education although chatbot technology has high potentials as a digital learning tool for providing personalized learning support. Therefore, more researches are necessary to widen findings related to chatbot technology. ies.ccsenet.org International Education Studies Vol. 14, No. 2;2021 Due to the possible and potential of chatbot technology as a digital learning tool with many functions to use for education and the limited research on chatbot technology in educational settings, especially in Thailand, the current study tended to use chatbot technology as a digital learning tool on Facebook's instant messaging platform to increase research knowledge of Thai students. It is anticipated that the chatbot developed can be used as a digital learning tool in class to increase students' research knowledge and create positive personalized learning support through this digital platform.

Objective
1) To develop the chatbot as a digital learning tool with personalized learning support to increase students' research knowledge 2) To evaluate the effectiveness of the chatbot from student perceptions 3) To investigate the effects of the chatbot on student's research knowledge

Method
The experimental research was conducted with a mix method design. The quantitative research was related to the chatbot applicability, its effectiveness, and students' research knowledge before and after the chatbot use. The qualitative research was related to recommendations from experts and feedback on the chatbot technology from students. This research consisted of 2 main stages: 1) the development of the chatbot, questionnaires, and research tests and 2) the implementation of the developed chatbot to investigate the effectiveness and effects of the chatbot with related questionnaires and research tests.
Stage 1 involved the literature review of chatbot technology, the study of research content, and the development of the chatbot, an evaluation form, an effectiveness questionnaire, and research tests.
Stage 2 involved the evaluation of the chatbot and related research instruments by 3 experts and the pilot test with non-target group of 14 students. The results of the experts' opinions and the pilot study were utilized to improve the chatbot. Next, the chatbot was implemented with 36 Thai university students for one semester to investigate the effectiveness and the effects on students' research knowledge after the use of chatbot.

Population and Sampling Procedures
There were 60 Thai university students under the curriculum of Business and Computer Education at the Faculty of Education, Kasetsart University, who were required by the curriculum to conduct research as graduation criteria. 36 Thai university students were selected using purposive sampling method as they were entitled to conduct researches under the selected curriculum so they must have research knowledge in order to conduct researches effectively and successfully, were Facebook messenger users as Facebook messenger was the instant messaging platform where the developed chatbot was integrated, enrolled in classes which the researcher was the instructor, and were willing to participate in this study. Although most of participants did not have online learning experience with chatbot technology before, they were able to keep up with chatbot technology after the brief introduction of how to use chatbot technology was provided as they were already accustomed to the Facebook's instant messaging platform.

Instruments
The chatbot was designed and developed to explore the effectiveness of chatbot technology and its effects with three questionnaires, that is, an evaluation form, an effectiveness questionnaire and research tests. The details were as following.
The chatbot was developed from the literature review of chatbot technology and content of research knowledge with strong focus on the content, organization of the content and the application of chatbot. 3 experts evaluated it with an evaluation from and the pilot test was done with 14 non-target group of students, who needed to conduct researches as required by the selected curriculum and were Facebook messenger members with strong focus on the chatbot effectiveness in 9 areas, that is, quickness, flexibility, convenience, interaction, providing information, searching information, familiarity, learning and satisfaction.
The first questionnaire was an evaluation form based on the key components of chatbot technology in 3 areas, that is, content of knowledge with 4 items, organization of content with 3 items, and application of chatbot with 5 items. Therefore, there were 12 items on a 5 Likert scale basis (1=very low to 5= very high) with an open-ended question at the end of the form. The evaluation criteria of average scores () was established at a level of 4.50-5.00 as the highest, 3.50-4.49 as higher, 2.50-3.49 as average, 1.50-2.49 as lower, and 1.00-1.49 as lowest. The level of 4.50-5.00 was accepted applicability of the chatbot. This questionnaire was then utilized to obtain opinions related to the chatbot applicability from experts before its use. The results of scores and recommendations were reported ies.ccsenet.org International Education Studies Vol. 14, No. 2; 2021 47 and utilized to improve the chatbot before its use.
The second questionnaire was an effectiveness questionnaire based on the literature review of chatbot technology in 9 areas, that is, quickness, flexibility, convenience, interaction, providing information, searching information, familiarity, learning, and satisfaction. Each area consisted of 2 items. Therefore, there were 18 items on a 5 Likert scale basis (1 = very low to 5 = very high) with an open-ended question at the end of the questionnaire for further recommendation. The evaluation criteria of average scores () was established at a level of 4.50-5.00 as the highest, 3.50-4.49 as higher, 2.50-3.49 as average, 1.50-2.49 as lower, and 1.00-1.49 as lowest. This questionnaire was then utilized to obtain students' perceived effectiveness of the chatbot including feedback and comments on its use. The values of the index of congruence (IOC) ranged from 0.67-1.00 according to opinions of experts while the reliability coefficient was established at 0.89 according to the pilot study with 14 non-target group of students.
The third questionnaire was research tests. They were developed from the content of research using a pretest-posttest design with the same item order to investigate the effects of the chatbot on students' research knowledge before and after the implementation. There were two tests. Each test consisted of 15 multiple choice questions, each with 4 possible answers but only one correct answer to investigate the understanding of research knowledge. Each correct answer was worth one score. The total scores in each test ranged from 0 to 15 indicating the understanding of research knowledge. The maximum total score of each test was 15. Total scores of two tests were 30. The validity of each test was verified by 3 experts with the values of the index of congruence (IOC) ranging from 0.67-1.00. The pilot study was done with 14 non-target group of students with the reliability values of 0.83. The difficulty and discrimination of the two tests were analyzed and reported at a level of difficulty (P) ranging from 0.20-0.80 and discrimination (r) ranging from 0.20-1.00, respectively.

Data Collection
The target group of students took the research knowledge pre-tests in the first session before the use of chatbot. After completing the pre-tests, they used the chatbot as a digital learning tool to provide personalized learning support in order to recall, revise and remember research knowledge studied. At the end of semester, the post-tests were distributed to investigate the chatbot's effects on students' research knowledge and the effectiveness questionnaire to obtain student perceptions towards the chatbot's effectiveness including feedback and comments on its use. Finally, the completeness and accuracy of all instruments were checked before proceeding to data analysis.

Data Analysis
Data obtained from an evaluation form, an effectiveness questionnaire, research tests were analyzed. Quantitative data from opinions of experts and participants on the applicability and the effectiveness of chatbot technology were used. A 5 Likert scale basis (1 = very low to 5 = very high) was utilized to evaluate the chatbot applicability according to opinions of experts and the perceived effectiveness of chatbot technology from students. The t-test was used to determine if there was a significant difference between the means of students' research knowledge before and after the use of chatbot. Qualitative data from open ended questions was reported to summarize recommendations from experts and feedback on the chatbot technology from students. The data collected were analyzed using mean, standard deviation, content analysis, and a t-test with the p-value set at the 0.05 level of significance.

Results and Discussion
The research findings were as following.

Development of the Chatbot
The chatbot was designed and developed as a digital learning tool to provide personalized learning support according to the literature review of chatbot technology. It was a rule-based or retrieval-based chatbot to match text messages inputted with appropriate predefined responses to help increase research knowledge of students through chat conversation so that students can make understanding of the research knowledge when using this digital learning tool as their personalized learning support to recall, revise and remember the knowledge studied.
Facebook messenger, which students were familiar with, was used as an instant messaging platform over the internet and Chatfuel was the chatbot for Facebook messenger with no need to install as it was already installed on most smart phones and computers. Students asked the chatbot a series of simple questions through text messages or typed any keyword to interact with content of research knowledge via smart phones or computers. Predefined responses according to questions asked or keywords typed were provided in short time. The research knowledge covered research methodologies, research process, research instruments, population, sample group, variables, data collection, data analysis and research report. The chatbot was then verified by 3 experts with strong expertise and experiences in education and technology to check for its applicability and recommendation on its improvement so that it could be further improved to increase students' research knowledge and create a better personalized learning support. The chatbot applicability for use was demonstrated in Table 1.  Table 1 demonstrated that the overall mean was 4.67 and standard deviation (S.D.) was 0.08, which was in very high level. In particular, the mean and standard deviation of content of knowledge was high ( = 4.50, S.D. = 0.25) while the mean and standard deviation of organization of content and application of chatbot was very high ( = 4.89, S.D. = 0.19,  = 4.67, S.D. = 0.12, respectively). The experts recommended an addition of research knowledge and interaction between users and the chatbot for more interactive learning. The necessary changes were made according to experts' opinions. The chatbot was then tested with 14 non-target group of students with the high level of perceived effectiveness ( = 4.43, S.D. = 0.35). The students recommended an addition of more graphics and more examples to make the chatbot more interesting. The necessary changes were made according to the students to improve the chatbot prior to its use with the target group of students.

Students' Perceived Effectiveness of the Chatbot
The chatbot was introduced at the beginning of the selected semester and implemented over one semester with target group of students who were encouraged to use the chatbot as a self-help digital learning tool in inputting their research queries and/or searching for information needed to get replies instantly without waiting for long time. Students conversed and interacted with chatbots via smart phones or computers at any time, either in class or outside class for personalized learning support as many times as needed. They were given an effectiveness questionnaire at the end of semester to investigate their perceptions of the chatbot effectiveness. Student perceptions towards the chatbot effectiveness were demonstrated in Table 2. ies.ccsenet.org International Vol. 14, No. 2;2021 = 0.50, respectively). Students reflected that chatbot technology was easy to use, easy to understand, innovative and fun for learning. They could get answers instantly and able to seek specific information related to research knowledge without waiting for responses. However, in response to questions that did not match the keywords specified, further details of finding proper answers such as links for further action should be provided instead of inputting the new questions and left those questions unanswered. Also, the chatbots only provided responses when typing correctly so there should be an option for the students to choose from a list of questions or keywords without typing any texts to get the answers required.

Effects of the Chatbot on Students' Research Knowledge
Two research tests were assigned to students. Before starting the chatbot, students were given pre-tests. At the end of semester, post-tests were distributed to determine if there were changes in student's research knowledge. The paired sample t-test was then used to determine if there was a significant difference between the two means of students' research knowledge at the different times (before and after use of the chatbot). The researcher checked assumptions of normality before starting statistical procedures to ensure that there were no missing values and outliers in the differences between the two means for the same group of students. The dependent variables were normally distributed. Assumptions of normality were met for the two means and their testing showed no violation.   Table 3 demonstrated that the mean of the pre-test scores was 21.83 (S.D. = 1.54). After the chatbot use, the mean of post-test scores increased to 25.75 (S.D. = 1.76). The post-test scores were higher than the pre-test scores at the 0.05 level of significance. The findings indicated that the chatbot increased students' research knowledge because students with higher scores reflected that the chatbot made it easier to get answers needed and make understanding of those answers. At the end, students' research knowledge was significantly increased when students understood the knowledge clearly through using the chatbot as a digital tool to help recall, revise and remember the knowledge studied with personalized learning support.
The main findings indicated that the chatbot, when used as an effective digital learning tool to provide personalized learning support, was effective according to students' perceived effectiveness and increased research knowledge after the chatbot use. The findings were in line with Winkler and Söllner (2018) who reported that chatbots had positive impact on successful learning and student satisfaction when using as personalized learning support. The more chatbots learn from questions and keywords, the better chatbots can be further developed to provide better content knowledge, which was in accordance with Knill, Carlsson, Chi, and Lezama (2004) who found that chatbots helped the teacher see through asked questions where student had problems, gauge student learning and weaknesses so that personalized support can be provided. Also, the findings can help instructors better integrate chatbot technology to monitor student's knowledge and create better personalized learning support in the future. Therefore, using chatbot technology was effective in enhancing students' knowledge, as the research findings illustrated that there was significant difference in students' scores before and after the chatbot implementation. The current study can provide guidelines of using chatbot technology as a useful digital learning tool with personalized learning support in an educational context to monitor students' understanding and increase content knowledge. With possible and potential use of chatbot technology, chatbots can present content knowledge with texts, images, graphs, audio, videos or a combination of these to increase knowledge and motivate student learning through its personalized learning support.

Conclusion and Recommendation
In conclusion, using chatbot technology in education settings to give personalized learning support in order to increase students' research knowledge gave positive results as it led to positive learning outcomes and helped provide better personalized learning support through this digital platform. However, the sample size in this research was quite small. Therefore, it was suggested that the future research may use a larger sample size with a control and experimental group to extend results to a broader extent in using chatbot technology to create an optimistic personalized learning support for students. Furthermore, the version of chatbot used in this study consisted of some inflexibility and limitations as it was unable to handle unseen texts or keywords. Therefore, a more flexible chatbot platform, which can generate as suitable responses as possible, may be considered for next research.