Attitudes Toward and Implications of the Computer-Based Exams at Arab American University of Palestine

We examine the computer-based exams system at Arab American University-Palestine in terms of teacher and student attitudes as well as the quality of the test items. A three-pronged approach to data collection was used. First, we elicited answers to questionnaires from 704 faculty and student respondents. Second, we conducted eight individual interviews with students and instructors, as well as three focus groups—each comprising 8–10 students—from different majors in the university. Third, we had access to the records of the registrar on the grades of students in different years prior to and after introducing the system. We utilized descriptive statistics to examine the quantitative data and qualitative methods to analyze the interviews. The results suggested that the attitudes of instructors and students, as well as the quality of the exams were not adequately considered by AAUP when it introduced the computer-based exams system. We found significant differences between grades prior to and after adopting the system; yet the changes are not necessarily positive, at least from an academic point of view.


Introduction
Arab American University-Palestine (AAUP) has recently started a comprehensive upgrade of all aspects of its education system, with the ultimate goal of improving the quality of educational output. Students assessment system, particularly exams-the most important tools of evaluation at AAUP-is now being revamped. The new assessment system has been developed to facilitate the process, save time and effort, and reduce human intervention, and thus subjectivity. Now computer-based exams are conducted for many university obligatory and elective courses-as part of the phased process of modernization.
A quite large part of AAUP development effort is centered on endeavors to progressively replace the traditional exam-based assessments with the more authentic performance-based evaluation. However, there are some courses whose output cannot be genuinely assessed, particularly university obligatory and elective courses, which are taught in a lecture mode given the large numbers of students. For these courses, AAUP has built a computer-based testing system which would expectedly deliver positive results and reduce the downsides of paper tests.
With an eye to achieving objectivity and avoiding bias, the new evaluation model involves multiple choice as well as true-or-false questions, believing that such a system would make all students subject to the same evaluation criteria. Earlier, questions in the tests were both objective (no more than 20 percent of the weight of the exam) and subjective (no less than 80 percent of the weight). The latter method is based on personal opinion, interpretations, points of view, emotions, and sometimes prejudice (see, for example, Anakwe, 2008;Marriot, 2009). In multi-section courses taught by different instructors (such as university required courses), computer-based exams achieve objectivity in the sense that they defeat any personal judgment by instructors.
graduates with the skills particularly needed in the market. Together with improving quality of education, the new system would expectedly enhance cooperation among students. Naturally, computer-based exams encourage students to work in groups, study together and share experiences, which helps build social positive attitudes, a soft skill that is needed in different settings at college and even in the future work place. Computer-based exams have also been found to lower student anxiety, as students are quite sure that they do not need to worry about results objectivity or structure of the exams (James, 2016;Karaman, 2011;Liu et al., 2015).
The topic has garnered much attention from scholars. In a Taiwanese study designed to identify the attitudes of undergraduate students toward computer-based tests, Berg and Lu (2014) found positive perceptions. However, students complained about the inadequacy of time. The same positive attitudes were suggested by Win and Tasi (2006), Tayon and Alkayat (2017), Ozden et al., (2004), and Nikou and Economides (2013).
Some relatively positive as well as negative perceptions, however, were reported by other studies. Al-Omari and Eyadat (2016), measured the perceptions of the academic staff and students of computer-based exams at Al-Yarmouk University in Jordan. The authors found relative satisfaction and reported significant differences between the responses of instructors and students. Bayazit and Asker (2012), on the other hand, reported negative responses from students. According to students, computer-based exams needed more time than paper exams, as noise and screen-time harm adversely impacted understanding. Finally, as far as performance is concerned, the findings of a study by Hosseini et al., (2014) suggested no significant differences between computer-based exams and traditional tests.
With the number of students in some courses (especially university requirements and electives) reaching beyond the capacity of an instructor to adequately manage countless answers to subjective questions characteristic of paper exams, AAUP has built a computer-based testing system for those courses. Generally speaking, computer-based tests are effective and productive, as well. However, AAUP should have conducted a detailed study to assess the adequacy of those tests. The present study, therefore, sets to give insights into the attitudes of instructors and students toward the new system, on the one hand, and the outcome of the quality of the exams, on the other.

1) What is the attitude of AAUP students toward computer-based exams?
2) What is the quality of computer-based exams at AAUP?
3) What is the attitude of AAUP instructors toward computer-based exams? 4) How do AAUP students evaluate their first-hand experience with computer-based exams? 5) What are the implications of computer-based exams for student grades?

Data Collection and Reliability
We used three approaches to collecting data. A primary-data structured questionnaire had two forms, one for students (two sections: students' attitude & exam quality), and the other for instructors. The two forms have been developed based on our literature review. To ensure reliability, we used the Cronbach's Alpha equation values 0.794, 0.858 and 0.828 respectively for students' attitude, computer-based exam quality and instructors' attitude, with a coefficient total score of 0.887 for students and 0.828 for the instructors. The second approach involved individual and focus group interviews. In the third approach, we had access to the records of the registrar on the grades of students in different years prior to and after introducing the system.

Population and Sample Size
The population comprised AAUP students (around 11,000) and instructors (236). Krejcie and Morgan equation (1970) was used to determine the minimum sample size (371 students and 146 instructors). For the two forms of the questionnaire, we used a stratified random sample to ensure appropriate and equitable representation for the target population. For both individual and focus-group interviews, we selected students from across different faculties based on major, year of study and gender.

Test Questions
We based our assessment of responses in this section on the mean and the standard deviation of the obtained values. Likert scale was specifically used, as illustrated in the Table below (low: 1-2.33, moderate: 2.34-3.66, and high: 3.67-5). The computer-based exam environment is more highly conducive for examination compared to paper exams. Overall, students have moderate-level attitude (M=3.44, SD=1.286). Interestingly, the mean for individual items fell between the moderate minimum and maximum values (2.71-3.94). As is clearly shown, at 3.94 (or 78.8% in percentage terms), technical glitches (such as poor network speed and power outages) are the major impediments of the system. In the second place comes suitability for students with special needs, at 3.67, or 75.2%. At the bottom of the scale comes the belief that the computer-based exam results are better than paper exam results (with a mean of 2.91 or 58.2%). The findings are quite similar to the conclusions Al-Omari and Eyadat arrived at back in 2016. 4.1.2 Second Question: What Is the Quality of Computer-Based Exams at AAUP? Again, quality of the exams received a moderate evaluation (M=3.23, SD=1.29). The mean for individual items fells between the moderate minimum and maximum values (2.77-4.03). The majority of respondents believe that the computer-based exam instructions are clear and easy to understand (4.03 or 80.6%). On the other hand, the linguistic accuracy of the exams received only 2.77 evaluation (or 55.4%). The computer-based exams need more time of student preparation than paper exams.
3.97 1.158 High 3 Processing the computer-based exams in different forms is easier than with paper exams. This dimension was also perceived moderately, with an average value of 3.49 (or 69.8%) and a standard deviation of 1.03. All items fell between a range of 2.31and 4.33; i.e. between the upper lows and lower highs, with mean figures falling in the moderate category. Items 31 (Computer-based exams require unified syllabi for all instructors) and 40 (The course ILOs are explained to students at the beginning of each semester) received high scores (4.33 & 4.22, respectively). On the other hand, items 18 (Computer-based exams require instructors to allocate more time for writing the exam) and 22 (Computer-based exam questions can be leaked) came at the upper limit of the moderate classification, at 3.56 and 3.50, respectively, or 71.2% and 70% in percentage terms. By contrast, items 9 (Essay questions could be included in the computer-based exam) and 11 (Computer-based exams provide support during the exam) received low scores: respectively 2.31 (or 46.2%) and 2.33 (or 46.6%). The findings are somehow parallel to the results of an earlier study by Al-Omari and Eyadat (2016) Responses to this question were elicited by both in-depth individual interviews and focus group interviews. Answers were audio-recorded and then transcribed verbatim. Later, data were analyzed using content analysis. The interviews with the 14 students were classified into five main categories.

The First Category: Understanding
Answers we obtained from respondents suggest that computer-based exams depend on understanding rather than memorizing. Traditionally, students prepare for paper exams by memorizing material. They believe that computer-based exams save time and efforts, and they need less preparation, as well. Responses suggested preference of computer-based exams, especially for university obligatory and elective courses. Respondents from the Faculty of Arts, however, support paper exams in the major core courses. "When you do a course, you just read and understand rather than memorize," one student said. "I just understand the course because there's no need to write accurately," she added. "The use of computer-based exams in specialization courses is counterproductive. For humanities, a test needs to check the style of writing and other related technicalities. However, computer-based tests can still be effective in elective courses," said another.

The Second Category: Course Content
According to focus group respondents, the content for the same course is different insofar as different instructors teach the same course. It occurs that such instructors use their own material and demonstration. Responses elicited revolved around that particular aspect: "Instructors have different books and summaries." "Instructors present the course in different ways." "Because there is no printed material, we have to take notes while the Doctor is lecturing. Sometimes what we seen in the exam is somewhat different from our notes." "I had a stack of notes. Prior to the exam it was so messy that I didn't know what to focus on." According to James (2016), Karaman (2011) and Liu et al., (2015), in a computer-based test, the exam items should reflect the content taught in classroom.

The Third Category: Grade Assigning System
Respondents approached this category from three angles: Human intervention: Answers suggested that in a paper exam, an instructor can intervene to give more or less grades to specific questions based on his/her discretion; so that a student who doesn't do well in a particular question can have the opportunity to be given more grades in another question for which s/he provided an engaging answer. In a computer-based test, the instructor has no control over assigning different questions specific grades.
Questions style: According to respondents, in a computer-based exam, the system gives grades for the final right answer, so the probability is either a full mark or zero mark for a question. In an essay question-typical of paper exams, a student is given a mark (low or high) no matter the answer. Naturally, the instructor evaluates the student critical thinking skills.
Marking Criteria: Answers from students suggested that the marking criteria in computer-based exams are more reliable, accurate and transparent than in paper exams.

The Fourth Category: Cheating
Respondents believe that in computer-based exams there is more room for cheating. Students can easily take a screen shot and share it with other students via smart watches and mobile phones. Worse, some respondents reported instructors in the labs illegally helping students find the right answer to some questions. "There are jel.ccsenet.org Journal of Education and Learning Vol. 8, No. 1;2019 some instructors who illegally help students in exams. In an exam, I saw the instructor giving a student an illegal assistant," a student claimed.
The results in this category are quite different from the findings of Marriot (2009)-that cheating in computer-based exams is by far less than in paper exams.

The Fifth Category: Exam Questions
Answers from the interviews characterized four aspects of computer-based exams. First, the language of the computer-based exam questions is ambiguous. "The questions are sometimes unclear. The proctor cannot explain anything. So, we lose marks," a student told us.
Second, some questions are repeated in the midterm and final exams. Worse, sometimes the same question is repeated from a student in the same session, but in different format. Some students said that the same question, but in different form, is given to different students in the same testing room. Students who are smart enough to know that these different questions need the same answer are given the opportunity to cheat. "There are questions repeated in the midterm and final exams," one student said. "A difficult question is repeated more than once," said another. "I had the same question repeated four times in an exam," a student maintained. "We do know that some questions are repeated, but we don't dare ask the instructor about them," another student complained.
Third, the exams are imbalanced in terms of complexity. Respondents reported that for the same course and in the same exam session, some students had a very easy exam and others had a very demanding one. "Some students have easy questions and others have difficult ones. It is unfair when we have easy and difficult versions for the same exam," a student expressed her concern.
Fourth, students say that the computer-based exam questions do not reflect the course syllabus. Many topics are not covered in the exam. "We couldn't imagine that in the final exam we will have only two questions from the midterm material. We spent much time studying something, only to discover it has a meager weight in the final exam," a student said.
Results from James (2016), Karaman (2011) and Liu et al., (2015) suggested that computer-based exams are impartial in terms of the marking system, clarity of questions and content coverage.

The Fifth Question: What Are the Implications of Computer-Based Exams for Student Grades?
We took the Faculty of Arts at AAUP as a case study because it was the first faculty to adopt computer-based exams, and now it has the largest number of computer-based exams. As a service faculty, it provides 18 university required courses, 12 of which have already operated the computer-based exam test. We recorded the final grades of students in these courses before (2013-2017 academic years) and after applying the new testing system. Modern Arab Thought Note. Source: Own. There were no records for courses numbered 1, 9, and 12 in the Table, because the computer-based testing system was applied for these courses only at the time this study was being conducted, and the results have yet to be published.  Obviously, there are stark differences in the grades before and after introducing the computer-based exam system. For illustration of the results, we have taken a random range of A, C+, F and W marks: Overall, the figures suggest a decline in the 'A' range as well as the 'F' range. Deciding whether the results are satisfactory to an academic system or not is not, however, an easy job. Miller et al., (2009) maintain that when judging the effectiveness of grades, there are many factors that need to be taken into consideration, such as the level of the course (introductory or advanced) and the student skill (outstanding, average or low-learning).
On the other hand, the grades within the C, C+ and B-ranges have seen important improvements. It seems that the class of students who used to have a relatively low performance in the paper exams have managed to move into a higher-performing category. In some accounts, this is a pseudo representation of student performance, with adverse implications for the entire academic system in the university.
Our results are different from the findings reached by Bayazit andAsker (2012) andHosseni et al., (2014), whose studies suggested no remarkable differences in student grades in the computer-based and the paper exams.
While they helped many low-performing students raise their grades relatively significantly, computer-based exams might be a source of frustration for highly-performing students. Whereas most students managed to pass the courses, the overall results were moderate, suggesting that computer-based exams do not adequately respond to the intended learning outcomes of the courses taught, nor do they reflect the actual performance of students, particularly outstanding ones. Respondent students reported that computer-based exams couldn't properly assess their skills.

Attitudes and Quality
We used t-test examination to assess three dimensions of computer-based exams compared to paper exams: the students' attitude, the instructors' attitude and the quality of the computer-based exams. A sample t-test was used to measure students' attitude. Figures in Table 9 show that the students' attitude (mean=3.3896, SD=.67432) was not significantly different from 3.67 (the cut point), t=118.315, p=0.000. Bottom line: students' attitudes were not adequately considered by AAUP when it introduced the computer-based exams system. The above figures indicate that computer-based exams (mean=3.2304, SD=.99443) were not significantly different from 3.67 (cut point), t=76.461, p=0.000. Evidently, quality was not particularly considered when AAUP launched the computer-based testing system.

Conclusion and Recommendations
Our analysis suggested that the quality of the computer-based exams as well as the attitudes of instructors and students were not particularly taken into consideration when AAUP launched the electronic testing system. Assumptions of the negative implications of the computer-based testing system were adequately attested by the student grades. Records from the registrar show a decline in the 'A' grade. Whereas computer-based exams helped many low-performing students raise their grades, they adversely impacted the highly-performing students. It is true that computer-based exams might have helped many students pass the courses; however, the overall results were not up to the expectations of planners. On the other hand, the 'F' grades saw a dramatic drop. In a sense, those who might have failed a certain course-judging from their performance in other courses-have managed to achieve a pseudo success.
Reflections from student respondents on the computer-based exams fell within a framework of 5 categories: Understanding, Course Content, Grading System, Cheating, and Exam Questions.
Understanding: Responses suggested preference of computer-based exams, especially for university obligatory and elective courses.
Course Content: The exam items should reflect the content taught in classroom.
Grading System: Answers suggested that in a paper exam, an instructor can intervene to give more or less grades to specific questions based on his/her discretion; so that a student who doesn't do well in a particular question can have the opportunity to be given more grades in another question for which s/he provided an engaging answer. In a computer-based test, the instructor has no control over assigning different questions specific grades.
Cheating: Respondents believe that in computer-based exams there is more room for cheating.
Exam Questions: First, the language of the computer-based exam questions is ambiguous. Second, some questions are repeated in the midterm and final exams. Worse, sometimes the same question is repeated for a student in the same session, but in different format. Third, the exams are imbalanced in terms of complexity. Fourth, the computer-based exam questions didn't reflect the course syllabus.
With these results in mind, we believe it is quite applicable to run a computer-based exam system for university obligatory and elective courses, but not major core courses. For computer-based exams to run smoothly and effectively and to deliver the intended objectives, it is important to allow and help instructors build the computer-based exam system. Instructors who teach the same course have to take part in writing the exams. For the sake of transparency and impartiality, it is important to adopt a single syllabus for a multi-section course. To eliminate any forms of cheating, surveillance cameras need to be installed in places where the behavior of students is effectively monitored. Finally, further studies are needed, particularly a research on implementing the standard academic performance criteria in the computer-based exams at AAUP and other local higher education institutions.