Digital School Examinations : An Educational Note of an Innovative Practice

The aim of this study is was discuss the effectiveness of a digital school examination practice which was developed, and cultivated at a Norwegian university‟s business school over a 9-year period. In this innovative practice, we intended to align the digital school examinations constructively into the course design, and we crafted examination questions and problems aiming to motivate students to acquire a deep learning approach. To hinder cheating on examinations where students brought their own devices, they worked with semi-indivual exam papers. The issues were common, but the students worked with specific data sets. Consequently, no solutions were equal. Empirical indications of effectiveness was derived from multiple sources: a survey, grade distributions, exam scores on question/problem types, and strings from the examination marking. The results show that students were satisfied using spreadsheets on the final school examination, which also motivated them to utilize a spreadsheet in their day-to-day learning activities. Moreover, we found it reasonable to affirm that semi-individual examinations hindered digital cheating.


Introduction
In Norway, both primary and secondary school students utilise computers for learning and assessment and students progressing to higher education are, in general, digital literate (Ørnes, Wilhelmsen, & Breivik, 2011).This is due to a renewed interest in which Norwegian politicians stated, "In higher education the government wants students to be trained in using information and communication technology (ICT) as an integral part of their learning" (Report No. 17 to the Storting, 2006Storting, -2007, p. 57), p. 57).
Research indicates that the ICT revolution has, to a certain extent, replaced paper-based technology in higher education in Norway (Aure & Abelsen, 2011).However, when knowledge is assessed on a final written exam, students must take a technological step backwards in which the pen, paper, and calculator still apply even when the use of such traditional tools is becoming increasingly outdated (ibid.).
Reasonably, students have asked why it is so important to develop digital skills when they cannot actually use such expertise in their respective exams.This was seen, inter alia, in the "We want PCs on our Exam" student campaign in Norway, which occurred during autumn 2011.Realising that employers search for digitally competent candidates reinforces the idea that students should be allowed to use computers also for their exams.
One reason why students are denied the use of computers is the lack of facilities (computers and premises) that can simultaneously handle hundreds of students undertaking their final exams digitally.Another reason is the fear that students may be tempted to cheat when using their own computers, which are connected to the Internet (King, Guyette, & Piotrowski, 2009).However, due to the insistence of the students, majority of the universities in Norway are experimenting with digital exams under various models.In addition, certain initiatives have been taken to coordinate the work in a national project (eCampus Norge, n.d.).
Assessment has an impact on student learning (Becker, Geer, & Hughes, 1968;Gibbs, 2006;Rust, 2002), and its practice is the basis of the overall student experience since it influences student behaviour, and ultimately student learning (Brown & Knight, 1994;Kohn, 1994).It has been confirmed that the majority of students regard assessment as the most important aspect of a course, and consequently, they use it to guide and frame their learning (Price, Handley, Millar, & O"Donovan, 2010).However, assessment itself is a fairly complex issue.
Managing assessment requires an integrated approach in which it takes the context of the assessment into account.Therefore, there is no unified approach in the management of assessment (ibid.).This article"s purpose is to provide an original contribution to the current debate on digital school examinations by discussing how such practice was developed, and cultivated at a Norwegian university"s business school over an 9-year period.Three research questions have guided our work: 1) Did we succeed in aligning the digital school examinations constructively into the course design?
2) Did we succeed in fosterering deep learning through digital school examinations?
3) Did we succeed in hindering cheating on digital school examinations?
The next sections examines the pedagogical theory underpinning the examination practice, describes the digital examination procedure and discusses its overall strengths and weaknesses.Finally, we present some empirical indications of effectiveness.

Aligning a Digital School Examination Constructively
Learning outcomes, assessment, and teaching and learning methods are interdependent.To support student learning, assessment should not be disconnected from teaching and learning outcomes.Instead, assessment should be aligned with the overall learning process.According to Biggs" model of "constructive alignment" (1996) in curriculum design, well-crafted course designs include: (1) learning outcomes that are clearly identified; (2) appropriate assessment tasks that are designed to directly assess whether each of the learning outcomes has been met; and (3) learning methods that empower the students to master the assessment requirements.
In course design, both formative and summative assessment activities should be embedded (Knight, 2002b).Assessments that certify achievement include a feed-out function since the grades can be treated as a performance indicator for the students.Such assessment is often called "summative" or "assessment of learning" (ibid.).On the other hand, assessments may have a formative purpose that modifies the learners thinking or behaviour to improve overall learning (Shute, 2008).Formative assessment is known as "assessment for learning".Educational research emphasise the powerful influence on learning through formative feedback and assessment (Black & Wiliam, 2009;Hattie & Timperley, 2007;Shute, 2008).

Constructive Alignment in Practice
One key factor in our examination practice was to embed formative and summative assessment activities constructively into the overall course design which implies that the teaching materials, learning activities, and assessments are linked and balanced to achieve the expected learning outcomes (Biggs, 1996).Spreadsheet use is integrated into all of the practical activities, including formative and summative assessments, which ensures that the assessments are a consistent part of the overall course design.Figure 1 illustrates the design of the undergraduate finance course at our business school.

Figure 1. The course design
In this course, we arranged two mandatory assignments (FA/SA in Figure 1) that must be approved before the student is allowed to take the final exam (SA in Figure 1).The assignments include both a summative and formative purpose.The summative purpose assesses the learning and documents the learning outcome (Knight, 2002b), while the formative purpose also assesses the learning but it provides each student with a detailed written report that shows the strengths and weaknesses as well as provides constructive suggestions for improvement.Within a continuous assessment system, it is essential to deliver formative feedback at regular intervals (Shute, 2008) and such reports also offer useful guidance for further education (Price et al., 2010).Formative assessments can also help faculty to recognize where students are struggling and address the problems in their teaching.
Mandatory assignments can motivate students who are struggling to begin their academic work and deter absenteeism by ensuring that they regularly submit the necessary assignments (Gibbs, 2007, p. 6).An additional purpose of the assignments is to inspire students to start their work early and maintain a sustained effort.The assignments also provide examination training (scaffolding) since they are arranged by using the final examination procedure (Wood, Bruner, & Ross, 1976).

Fostering Deep Learning through Digital School Examinations
According to Knight (2002a, p. 107), assessment is "the Achilles" heel of quality".In higher education, we seek to encourage deep or quality learning.In contrast, surface learning is the opposite of deep learning (Gibbs, 1992;Marton & Saljo, 1984;Ramsden, 1992).When students can only achieve high marks through memorisation and drills, then the course design promotes surface learning (see Figure 2).Figure 2. Surface learning and deep learning linked to Bloom"s revised taxonomy (Anderson & Krathwohl, 2001) Bloom"s revised taxonomy (BRT) defines the cognitive learning process in the following format: "the student will learn to: verb, noun" (Anderson & Krathwohl, 2001, pp. 4-5).The verb describes the cognitive process, and the noun depicts the unit of knowledge.The two axes in Figure 1 above, provide an overview of BRT"s main categories.The arrows in the figure connect the conceptual pair (surface and deep learning) to the cognitive processes and knowledge dimension in the taxonomy.
Surface learning is restricted to remembering processes such as the memorisation of facts, and concepts as well as learning by drilling.At the undergraduate level, learning is about understanding, applying, and analysing facts, concepts, and procedures.However, it is also necessary to remember pertinent information to acquire in-depth knowledge.In Figure 1, the further the arrow moves toward the bottom right-hand corner, the more depth-oriented the learning becomes.In this regard, a significant portion of our examination practice is to craft examination questions and problems that challenge students to include a deeper learning approach.

Fostering Deep Learning in Practice
The final examination is perhaps the most important tool that encourages deep learning.Consequently, we should design examination questions and problems so that they are difficult to solve for those students who possess only surface knowledge of the topic.Through taxonomic analysis, we also strive to ensure the quality of the assignments and the final examination (Bertheussen, 2012).To this end, we use BRT for the cognitive domain, as illustrated in Figure 1.Table 1 provides an overview of the four general questions/problem types that form the basis of the required assignments and the final examination.1).

30%
2 Implementing a single procedure Applying procedure knowledge (see 3C in Figure 1).10% 3 Implementing multiple procedures in an established structure Applying and analyzing procedure knowledge (see 3C and 4C in Figure 1).

20%
4 Implementing multiple procedures in a structure established by the students Applying, analyzing, evaluating, and creating procedural knowledge (see 3C, 4C, 5C, and 6C in Figure 1).

40%
The exam tests the students" ability to remember facts and understand concepts (Question/Problem Type 1 in Table 1).However, multiple-choice questions cannot test the students" capacity to apply knowledge and skills to practice-related problems (Schuwirth & Van der Vleuten, 2004).Nevertheless, in basic professional courses, it is important to have a practical approach.Since the students will eventually become professionals, we emphasise their capacity to implement disciplinary procedures on a spreadsheet (Question/Problem Types 2-4 in Table 1).
In the process of surface learning, the learner remembers and acts mechanically by memorising facts and drill procedures.In addition, Questions/Problems Types 1-3 in Table 1 can also be solved by students using a surface learning approach.
However, for the exam, students must also have the ability to apply knowledge and procedures to new problems.Question/Problem Type 4 in Table 1 tests the capacity of students in which they must independently establish a structure for the problem to be solved.These types of problems can be difficult, especially for those students who rely only on memorisation and drills.In practice, this requires developing a financial spreadsheet model from scratch (Bertheussen, 2012).As shown in Table 1, solving this type of unstructured problem received the highest score (40%).

Hindering Cheating on Digital Exams
Cheating on exams has been a considerable problem in the education sector for more than a century (Anderson, 1998) and the research literature with regard to cheating on traditional exams is extensive (McCabe, 1999;McCabe, Trevino, & Butterfield, 2001;Michaels & Miethe, 1989;Whitley, 1998).However, the literature is considerably sparse in relation to online cheating (Watson & Sottile, 2010).In a study by King et al. (2009), approximately 74% of the students stated that they found it easier to cheat in an online course compared to a traditional class.Nevertheless, this does not necessarily imply that all students, in reality, cheat more online.
In fact, the results of studies on online cheating have been contradictory.Grijalva, Nowell, and Kerkvliet (2006) found that there was no significant difference in cheating between traditional and online exams with approximately 3% of the students cheating on both examination types.Lanier (2006) found that students cheated much more in online courses while Stuber-McEwen, Wiseley, and Hoggatt (2009) concluded that students cheated less in online courses.
In addition, students taking a surface approach to learning are more likely to cheat compared to intrinsically motivated students (Carroll & Appleton, 2001).Therefore, it is important to develop digital exam practices that foster deep learning approaches.

Hindering Digital Cheating in Practice
The final exam is not an open-book examination and the students are not allowed to open or copy materials from files since this is regarded as cheating.We also consider any communication with fellow students as cheating.To reduce the potential for cheating, students are given common problems to address, but they include individual data sets.In other words, the examination papers are semi-individual since the problems are similar but the data sets are personal.This implies that the students should solve the problems by applying the same procedures, but they must implement the procedures in his/her own way.
In the data sets, the values of the variables differ, which results in various solutions.Moreover, the variables used in calculation procedures (formulas) are inserted in randomised cells on the spreadsheet, and as a result, the formulas created by the students are different.Therefore, information concerning another student"s solution, or formulas leading to the solution, is of no value.The fact that students develop different solutions, as well as different formulas leading up to the solutions, makes it more difficult and time consuming to cheat.
Nevertheless, this is not a guarantee against cheating.Prior to the exam, the students may work with financial issues solved in a specified way and may have saved many of the solutions on their computers.Since we have limited opportunities to enforce the ban on cheating, students can still open these spreadsheets and compare them with the exam questions during the examination.
Furthermore, by analysing the structure of an exercise that was previously stored on the computer, a student can find helpful guidelines that may aid in working out the solution.However, since the individual data sets are different, they cannot simply copy and paste elements from the previous exercises into their examination paper.
Comparing and discovering the underlying structure of the two problems require sophisticated abstract thinking.
According to BRT, evaluating is, along with analysing and creating, a higher order cognitive process (see Figure 1).Students who are capable of this type of thinking process are more likely to resolve the original exam questions and problems.
However, we cannot rule out that students are searching for solutions to exam questions and problems by opening, analysing, and evaluating material stored on their computers.In addition to being knowledge-intensive, this operation requires students to have completed the exercises in the course and saved the solutions.Indeed, this will be a time-consuming method for solving an exam question or problem.During a digital examination such as ours, time is a limited resource and the students receive approximately half of what they would have received on a similar manual examination.

Automatic Scoring of Digital Exam Papers
A computer can quickly and automatically mark the assignments and examination workbooks and, in most cases, students believe that it is credible and fair.When marking an examination paper manually, we do not simply assess the result, but we also examine the problem solving process.The same practice is utilised with automatic marking.For example, students receive credit for each sub-problem that they manage to solve and they also obtain credit for solving part of the sub-problem.
However, automatic marking is complicated by the risk of consecutive errors in a calculation (Bertheussen, 2014).A consecutive error is an error that occurs early, thus propagating and causing errors in subsequent calculations.In this regard, our marking algorithm detects and corrects consecutive errors so that students are not successively penalised for the same error (ibid.).

Empirical Indications of Effectiveness
The innovative digital examination procedure described in this article included several activities.First, we attempted to prepare our students for the final exam through continuous formative and summative assessment activities that were aligned with the course design (see FA&SA in Figure 1).Second, we aimed to construct exam questions and problems that challenged and measured more than surface subject knowledge (see "Deep learning" in Figure 2).Third, we prepared semi-individual examination papers to hinder possible cheating (see the paragraph "Hindering digital cheating in practice" on p.8).
In this section we will present some empirical indications of the effecitiveness of the digital examination procedure.This issue is open and complex, but we do have some indications that we are pursuing a fruitful track.

Research Question 1: Did We Succeed in Aligning the Digital School Examinations Constructively into the Course Design?
6.1.1Indication of Success: After Nine years, the Digital Exam Practice Is Still Ongoing The digital exam procedure has been through classroom iterations for nine years at our business school.Currently, more than 100 students take the course each year and both the assignments and final exam are completed using the students" computers in a large auditorium (with supervisors).A total of 800 students have passed the exam in this financial course by using their own computers to solve semi-individual exam questions and problems.Each repetition has provided us with new opportunities to make improvements, so that over time, we think the procedure has evolved into a well-developed practice.
6.1.2Indication of Success: The Students Were Satisfied with the Practice Now, we will discuss students satisfaction and motivation from using a spreadsheet for mandatary assignments and the final exam.A questionnaire was distributed electronically (QuestBack) to 131 students who completed the final digital exam in fall 2013 and we received 83 responses (63%).Two claims were formulated and designed to capture the content of the question in a valid and reliable manner.The responses were reported on a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree).The claim that students were satisfied by being allowed to use a spreadsheet for the assignments and the final exam included a significant mean score of 4.61.Realising that the use of a spreadsheet on the assignments and exam was required motivated the students to learn how to use this artefact (significant mean score of 4.64).
Since spreadsheet use is integrated into all practical learning activities in the course (see Figure 2), the students should be allowed to use a spreadsheet when being assessed.If not, the course design would become inconsistent (Biggs, 1996).The students reported that they were satisfied with being allowed to use a spreadsheet for their mandatory assignments and the final exam (see Table 2).They were also motivated to develop spreadsheet skills since they could use such skills for the assignments and the final exam (see Table 2).The results are, in agreement with Brown and Knight (1994), stating that the examination exerts great influence on what students give priority.

Indication of Success: The Grading Distribution Is Approximately Normal
Now, we will discuss whether the grading distribution for digital examinations differ significantly from the normal distribution or the national average.During the examination, the students document some of their learning outcomes (Ash & Clayton, 2009).Figure 3 illustrates the cumulative distribution of grades for the 2005-2013 time period.During this period, 38% of the students received either an "A" or a "B", while 11% failed the exam.Table 3 compares our distribution of the grades (see Figure 3) with the normal distribution and the national average for an undergraduate finance course.As indicated in Table 3, an additional 6% achieved an "A" or a "B" on our digital examination compared to the national average, while an additional 2% received an "A" or a "B" using our examination procedure compared to normal distribution.Our failure rate was 4% lower than the national average.
For a digital school examination to be trustworthy, an external examiner approves the exam questions and problems before the final examination.Moreover, the grading distribution (based on 800 students over an 9-year period) should not differ significantly from the normal distribution or the national average.Our students scored higher than the national average and slightly better than the normal distribution (see Figure 3 and Table 3).In addition, our failure rate was somewhat lower than the national average (see Table 3).
One reason for these findings may be that our course was more motivating since the use of a spreadsheet was integrated into all of the practical activities, which satisfied and motivated the students (see Table 2).But there might be many other reasons why students score higher than the national average.For example, the teachers may have set less ambitious goals for the course, or the exam problems may have been easyer to solve for the students.

Research Question 2: Did We Succeed in Fosterering Deep Learning Through Digital School Examinations?
A significant part of our examination practice is to craft examination questions and problems that motivate students to acquire a deep learning approach (see Table 1).In Table 4, we aim to elucidate this claim.On the right side of Table 4, we provide the average score and standard deviation of the four question/problem types presented to 104 students who completed the examination in fall 2013.Question/Problem Types 1-3 (Table 4) can be solved by both students with a deep learning approach and surface oriented learners.The average score of these three question/problem types is approximately 70%.However, to solve Question/Problem Type 4, students must develop their own financial spreadsheet model from scratch, which requires a deeper learning approach (see Figure 1).The average score of this problem is, as indicated, lower (52%).To earn an "A", 90% of the examination problems must be solved correctly.If students score 100% on all of the "easy" exam problems (Question/Problem Types 1-3), then they still need to solve 75% of the "difficult" problems to receive an "A".
We hope that this fact will encourage students to aquire a deeper learning approach.In higher education, students should not be able to earn high marks simply by memorising and drilling (Ramsden, 1992).Since we have invested heavily in digitising school exams, we risk reinforcing a problem that already exists.To prevent this issue, it is of utmost importance that we challenge students on higher order cognitive processes (see Figure 1).
In addition, the final examination is perhaps the most important tool that encourages deep learning.In our digital exam, there was always at least one task that required a student to independently establish a structure for the problem to be solved.In practice, this means developing a financial spreadsheet model from scratch.Upon examination, solving this type of unstructured problem achieved the highest score (40%, as shown in Table 1).

Research Question 3: Did We Succeed in Hindering Cheating on Digital School Examinations?
The grade distribution in Figure 3 does not indicate cheating.We would expect the grade distribution to be influenced by cheating since it enhances their performance.Therefore, more candidates would receive an "A" or a "B" compared to grades achieved through the normal distribution and the national average.The distribution in Figure 3 is slightly skewed to the left since a greater percentage of students received either an "A" or a "B" through digital examinations compared to the normal distribution (37% vs. 35%) and the national average (37% vs. 31%).However, behind the national average, there is a significant variation among institutions.
The automatic marking algorithm creates detailed marking strings for the students by counting their individual scores for each question and sub-problem.If several students achieve a total score with identical sub-scores, then there may be a reason for concern.After the marking, we compare the students" marking strings to expose instances of cheating (see Figure 4).Neither the four students who received 91 points, nor the three with 90 points had the same string in Figure 4. Students obtained the same total score by adding different sub-scores, which indicated that they achieved their scores honestly.However, it does not exclude them from the possibility of cheating.Upon examination of the grade distribution and the combined marking strings, it is not unreasonable to affirm that semi-individual examinations hinder digital cheating.However, we did experience cheating in which one student submitted another student"s digital examination paper as his own.This type of cheating was revealed through the examination procedure.

Conclusion
Students progressing to higher education in Norway are, in general, digital literate (Ørnes et al., 2011).Nevertheless, when knowledge is assessed on final exams, they are still required to apply pen, paper, and a calculator even though such traditional tools are becoming increasingly outdated.To comply with both the students" request to use computers on an exam (ibid.)and educational objectives to utilise ICT as an integral part of the learning process (Report No. 17 to the Storting, 2006Storting, -2007)), we have at our business school designed a finance course in which students solve problems by using a spreadsheet on their own computers rather than a calculator.In addition, the spreadsheet is also used for the mandatory assignments, as well as the final exam.In this paper, we reflect on what we believe is an innovative educational practice that utilises digital school exams.Three research questions guided our work.
First we questioned if we had succeeded in aligning the digital school examinations constructively into the course design.In the empirical part of the study we discussed three indications of success.One such indicator was that after nine years, our digital exam practice is still ongoing, and currently, more than 100 students take the course each year.Both the assignments and the final exam are completed using the students" computers in a large auditorium with supervisors.Another indication giving support to research question 1 is that the students were satisfied with the assessment procedure and motivated to improve their digital skills (see Table 2).This result was based on a survey which recieved 83 responses.A total of 800 students over an 9-year period have passed the exam in this financial course by using their own computers to solve semi-individual exam questions and problems.According to Figure 3 and Table 3, the cumulative distribution of grades was approximately normal, which is the third and final indication supporting research 1.
In the second research question, we asked if we succeeded in fosterering deep learning through our digital school examination practice.For students to achieve high grades on the digital school examinations (Table 4), they were challenged also on higher order cognitive processes (analysing, evaluating and creating).We trust that this fact encouraged them to aquire a deep learning approach.
In our third and final research question, we asked if we succeeded in hindering cheating on digital school examinations by submitting semi-individual examination papers in which the presented materials were the same, but the semi-individual data sets ensured that none of the solutions were identical.Upon examination of the grade distribution (Figure 3) and the combined marking strings (Figure 4), we found it not unreasonable to affirm that semi-individual examinations hindered digital cheating.
Although we believe that our digital examination procedure is an educational innovation, it still includes some limitations.The examination procedure was not generic since it related to exam problems that were only suitable for spreadsheet modelling.In addition, the problems had to be simple in structure and limited in scope.However, this is a typical feature of examination problems, especially in current business economics subjects at the undergraduate level.Therefore, the empirical evidence reported in this study is restrictive and not representative.
To eliminate contextual effects, future research should be significantly expanded to include studies that cross class, institution and geographic lines.

Figure 4 .
Figure 4. Strings from an examination marking

Table 1 .
Exam questions that challenge different cognitive processes

Table 2 .
Student satisfaction and motivation from using a spreadsheet for the mandatory assignments and the final exam (n = 83)

Table 3 .
Comparison of the grade distributions