Engaging with Graduate Attributes through Encouraging Accurate Student Self-Assessment

This work was part of an ALTC Funded Project - Facilitating staff and student engagement with graduate attribute development, assessment and standards. The project team would like to acknowledge the work of the teaching team of subject (Dr Peter Docherty & Mr Harry Tse) for their contribution to this study. Abstract Self-assessment can be conceptualised as the involvement of students in identifying assessment criteria and standards that they can apply to their work in order to make judgements about whether they have met these criteria (Boud, 1995). It is a process that promotes student learning rather than just grade allocation. However, self-assessment does not have obvious face validity for students; and many students find that making an objective assessment of their work difficult (Lindblom-ylanne, Pihlajamak & Kotkas, 2006). Previous business education research has also found that self-assessment does not closely reflect either peer or instructor assessments (Campbell, et al., 2001). The current study aimed to explore: (a) the relationship between self-assessment grading and teacher assessment; and (b) the effect of self-assessment in engaging students with graduate attributes, in order to explore the tenets of self-assessment This process of self-assessment was investigated through application of an online assessment system, ReView, to encourage more effective self-assessment in business education. Data collected from two groups (student and teacher) demonstrated that: (1) initial self-assessment results between the teaching academics and the students’ self-assessment, were significantly different with students overestimating their ability on every criterion; (2) however, the variation diminished with time to the point that there was no significant difference between the two assessments; and (3) students’ awareness of the graduate attributes for their degree program increased from the beginning to the end of the subject (Note 1).


Introduction
This study (Note 2) focused on developing an approach to engage academics and students with graduate attributes in the business curriculum using self-assessment to assist with the process. It has long been acknowledged that assessment is a key driver in student learning (Ramsden, 2003). Assessment defines what students regard as important, how they spend their time, and how they come to see themselves as students (Brown, Bull & Pendlebury, 1997). Therefore, one way to change student learning is through methods of assessment designed to facilitate these changes (Lawson & Fazey, 2000) and explicitly articulated to students so that their perceptions of the assessments are accurate (Biggs, 2003). In all forms of assessment there are two key stages: firstly, an appreciation of the criteria and standards required for succeeding in an assessment task, and secondly, the ability to make a judgement about whether work meets these criteria and standards. This ability to make accurate judgements can be demonstrated through self-assessment, which can be described as, "the involvement of students in identifying standards and/or criteria to apply to their work and making judgements about the extent to which they have met these criteria and standards" (Boud, 1991, p.5).
One of the core purposes of education is to develop students' capacity in to make judgements about their own work. Such self evaluation is needed both to enable effective study, so that students can focus on the most important aspects of their work they need to improve, and secondly for students to build their skills. Although self-assessment is critical in learning, and it is natural for students to self-assess, it is usually completed in a random, unstructured fashion and as such is not effectively aligned with formal assessment procedures and therefore does not directly impact on student performance (O'Donovan, Price & Rust, 2008).
When self-assessment is fostered in a manner that facilitates both the understanding of criteria and standards, as well as accurate self-judgement, then it contributes to a variety of central goals of university education (Boud, 1995). A person who has the capacity to make good judgements about their work will be able to know how their work can be improved. It can also create awareness of the scope of practice and when they will need to refer to and involve others, as well as recognising areas for further development. Development of the ability to realistically assess one's own performance and taking responsibility for decision making relates to autonomous learning, which is a skill for lifelong learning, as well as improving the student's level of understanding in the course being studied (Black & Wiliam, 1998). Unless students are able to make effective judgements about the quality of their own work beyond their university degree, the assessment to which they are subjected within their degree programs is not sustainable (Boud et al, 2010).
The benefits of students capable of demonstrating effective self-assessment has been documented in many studies of self-assessment (see Falchikov and Boud 1989). It is recognised that high achievers tend to underestimate performance and low achievers over-estimate performance. However, Boud, Lawson and Thompson's (2011) data suggests that students who are both accurate estimators (mid-range performers) and those who tend to underestimate their performance (high performers), improve their academic performance over successive tasks. However, over-estimators, who tend to be poor performers, do not appear to learn how to improve their performance over time.
Students often struggle to engage with the standards expected of them and the meaning of the criteria when they self assess. These difficulties can arise if understanding the criteria is downplayed by the educators; when engagement with these factors of assessment involves considerable critical thinking, a skill that students need support in developing; if the student's lack ability to manipulate criteria in ways which allow them to be applied to their work; and due to difficulties learners have in articulating standards and criteria without prior learning and experience (Boud, 1995). Despite of these challenges, students are keen to better understand assessment standards (O'Donovan, Price & Rust, 2001). However, evidence suggests that many students are confused about what is being asked of them and often resort to guessing as a way of interpreting assessment standards (Hinnett 1995;O'Donovan, Price & Rust, 2008).
Students are more capable of making a judgement about the standard of their work. Carlson and Smith-Howell (1995) found that oral presentations could be graded validly and reliably by inexperienced and relatively untrained peer and self markers. Campbell, Mothersbaugh, Brammer and Taylor (2001) also established high reliability between academics and self-assessors for holistic grades but when asked to score disaggregated performance, there was a substantial difference between the scores for the content element. Falchikov and Boud (1989) argue that self-assessing is still valuable even when there are discrepancies in judgement saying, "even in the absence of significant agreement between student and teacher, self-assessment can provide potent feedback to the student about both learning and educational and professional standards" (p.427).
Self-assessment is viewed as an important part of students' learning. However, difficulties students face with understanding criteria and standards and the problems that occur with some elements of judging performance against these standards point to a need to develop practices that can foster effective learning experiences using self-assessment. These practices should acknowledge that business education is changing, and the role of the lecturer is moving from expert/judge to coaches/facilitators who aid students in being able to identify the criteria and standards for assessment so that they can be self critical (Gopinath, 1999). In this respect including self-assessment processes can impact on teaching.
Faculty members often express dissatisfaction over grading procedures, which are often seen as largely subjective and requiring considerable care to balance perceived student expectations of their grades with what the instructor believes they deserve . . . Sharing the assessment process with the students can potentially be attractive as well as efficient and can release faculty time for more educationally worthwhile pursuits (Gopinath, 1999, p.10) The project outlined in this paper attempted to bridge the gap in understanding of graduate attributes by embedding these attributes into teaching through alignment of assessment criteria and provision of feedback on these attributes, fostered by the use of self-assessment. Graduate attributes are often mentioned in curriculum documentation but the effective integration of these into developmental approaches in the classroom has proven to be somewhat elusive. In concert, the consistent alignment of graduate attributes with assessment processes is not widespread across the higher education sector (Chalmers & Thomson, 2008). Institutional support for the development and assessment of graduate attributes has been inconsistent and not without its problems (Hoban, Lefoe, James, Curtis, Kaidonis, Hadi, Lipu, McHarg, & Collins 2004). Barrie (2004, p. 261) claimed that, "it is apparent that Australian university academics charged with responsibility for developing students' generic graduate attributes do not share a common understanding of either the nature of these outcomes, or the teaching and learning processes that might facilitate the development of these outcomes." Attempts to integrate graduate attributes into teaching and assessment have met with responses ranging from reluctance and resistance to full adoption (Rust, O'Donovan & Price, 2005).
Academics' resistance is understood in terms of their expectations that assessment is based only on discipline-specific content and that assessment of 'additional' attributes is a distraction or unnecessary extra work (Thompson & Treleaven, 2008). Also "business lecturers do not value the development of their students' profession-specific skills or knowledge creation capacity as highly as the development of their theoretical knowledge. Therefore in general academics have been less motivated to adopt social constructivist teaching methods" (Hanson & Sinclair, 2008 p183).
Graduate attribute engagement and assessment was facilitated through use of an online system, ReView. The aim was to use self assessment to assist the students' learning process in making the connections between graduate attributes and assessment. To achieve this aim a social constructivist approach to graduate attributes was adopted, (Rust, O'Donovan & Price, 2005;Kember & Leung, 2005) whereby assessment processes, criteria and standards were framed within an active engagement and participation of both students and their business educators. Using the approach (1) the relationship between student self-assessment grading and teacher grading; and (b) the effect of using self-assessment in engaging students with graduate attributes, were explored.

ReView: An Online Marking & Self-Assessment System
ReView (Note 3) facilitates the development of self-assessment in learning processes, by engaging academics and students in a deeper understanding of assessment criteria in relation to graduate attributes, and providing a mechanism to guide students through making judgements about their work using these criteria.
The initial steps of using an online software system incorporate a systematic 'review' procedure whereby assessment criteria are linked to graduate attribute categories and the criteria are clearly worded to ensure the qualities, knowledge and skills that are valued in student performance are explicit. Through this process, academics have the opportunity to clarify and demonstrate an unequivocal alignment of assessment tasks to learning objectives and graduate attribute development across subjects and levels of a program of study.
This initial step was designed to make assessment criteria more transparent to students to aid their understanding of the expectations of their assessment tasks. Students are able to view these assessment criteria before submitting their work and can visually see how the criteria align to the graduate attributes through the colour and shape coding system (see Figure 1).
<Insert Figure 1 Here> The next step engages the student with making judgements on their performance in relation to the assessment criteria. Students are prompted to self-assess their work for each of the criteria using a sliding grading scale for each criterion divided into fail, pass, credit, distinction and high distinction categories (See Figure 2).
<Insert Figure 2 Here> Academics mark assessments directly online using ReView's data sliders in a similar fashion to the student self-assessment process. After entering their assessments, the academics are able to see how students self-assessed their work (see Figure 3). Where there is a large variation between a student's self-assessment and the academics' grading, the academics are able to use this difference between the student's assessment and their own as a guide for feedback.
Students were then able to access the academics' gradings for each criterion and overall, as well as the written feedback for the assessment task via their personal student view. In this view the students were able to examine the academics' assessment of their work in comparison to their own self-assessment.
<Insert Figure 3 Here> A: Selectable list of students (obscured here for ethics de-identification), B: Colour-coded symbols next to the criteria represent one of five attribute categories in this particular university example, C: Data 'sliders': The black bar is the academics' slider. Triangles on the top edge of the data sliders are students' self-assessments (done prior to academics marking and not visible until academics have marked), D: 'Total' data slider: the black bar can be dragged causing the marks and bars on other criteria to move in proportion for benchmarking purposes.

Methodology
The study (Note 4) involved 239 undergraduate student participants studying a second year economics subject in an undergraduate business degree. These students volunteered to complete the pre and post survey and to self-assess their work using ReView.

Self-Assessment
The subject was developed by two academics and consisted of four assessment tasks. The first two assessments were scaffolded so that the skills and knowledge developed for task one were built on for the second task. This meant that the feedback from task one could be directly applied to student performance in task two. ReView was used for online marking and student self-assessment in these first two tasks. The third and fourth tasks were exam based and so ReView was not used for these assessments. The students were introduced to how the assessment criteria for first two assignments related to the graduate attributes using the colour code system in ReView. The students were encouraged to self-assess for each criteria using ReView prior to submitting their assignments. They were then provided with feedback on their performance which was directly aligned to the assessment criteria. In order to encourage attention to this feedback and grading markers for the assessment were published in ReView two days before the student's percentage marks were released. This meant that students were able to compare how they had self-assessed their work in relation to the academics (see Figure 3), whilst seeing the feedback comments alongside. This process of self-assessing at submission, receiving feedback and then receiving the academics mark was then repeated for the second assignment. The data collected on the students' performance from both the self-assessment scores and academics marks for the two assessments were then analysed.

Engagement
The methodology also included the administration of a pre and post questionnaire to the students. The initial questionnaire that was given to students during the first week of the subject was to obtain information on student awareness of graduate attributes before the subject began. This questionnaire contained items about the students' awareness of the subject content; understanding of graduate attributes; perceptions of self-assessment; recognition of what was required to achieve a high distinction. The post subject survey was administered at the end of the subject when the student had submitted and received feedback and academics marks for their first two assignments. It was used to measure the graduate attribute awareness at the end of the subject, asking for views on what elements impacted on any changes of awareness. This post subject survey also further explored how engagement with the graduate attributes had aided the students' learning.

Self-Assessment
The initial task logged in the online software system showed a significant difference between the academics assessment and the students' self-assessment (t(238,1)=-2.411; p=0.017), with students overestimating their ability on every criterion. This variation diminished with time and by the second task there was no significant difference found between academics and student marks (see Figure 4). This indicates that students aligned their expectations and standards to the standards applied by the academics as a result of the self-assessment and feedback system. <Insert Figure 4 Here> Although student and academics marks vary in the initial task, with students overestimating their grades, the data does predominantly follow the same pattern (see Figure 5) with significant reasonably strong correlations found in the results for each of the criteria between the academics and student scores (r= 0.518 -0.614; p<0.01). This means that although students are not necessarily accurate in rating their ability they are able to judge which criteria they have performed better in than others. These correlations, although not as strong, were found again for each criteria in the second task (r= 0.305 -0.334; p<0.01) (see Figure 6).

Engagement
The pre subject student survey indicated that students' graduate attribute awareness was very low at the beginning of the subject. There was a reported increase in the number of students who were familiar with graduate attributes by the end of the subject. The impact of the activities in this subject in aiding students to understand the assessment criteria was evident from the post subject survey questions relating to the assessment. The students reported they were able to clearly see how the assessment criteria linked to the graduate attribute, emphasised by the colour coding system (79.6%) and that this meant that the assessing criteria was easier to understand (76.4%). This increased understanding the criteria would have aided students in their ability to understand and respond to their criteria referenced feedback.
The impact of the self-assessment process made an impression on the students' understanding of what was expected of them in relation to the assessment criteria and the level that was required. As a result of completing the subject, a reasonably large proportion (60.7%) reported they would like the opportunity to self-assess in all their subjects of study.

Discussion
The aim of this study was to develop an approach to teaching and learning that would aid students in understanding assessment criteria and standards, in order to be able to accurately make judgements about their performance for the assessment criteria. An online system, ReView, provided a structured environment for self-assessment. By aligning assessment criteria to graduate attributes; providing a self-assessment mechanism for students and a platform for feedback which is linked to each assessment criteria; an opportunity to compare self and teacher marks for assignments was created. The academics were encouraged to establish integrated learning objectives with the assessment criteria through their interaction with the ReView system. This meant that subjects were constructively aligned between their graduate attributes and the assessment tasks and provided a structure for students to both identify and recognise the connections between the way they were being assessed and the objectives of both the subject and the degree in general.
The study found that students' marks could be better aligned with academics' scores through support in understanding expectations and progressive exposure to self-assessment. This phenomenon could be explained in several ways. The students were more experienced in self-assessing by the second task. The students understanding of the criteria and standards could have also improved as the subject progressed. The opportunity to compare their self assessment scores with the academics in the first task would have provided an opportunity for the students to understand the standards expected more clearly. Additionally, the feedback, which was linked to the criteria, provided for the first assignment would have contributed to the students understanding of the criteria and standards. This feedback was released before the actual scores and students' attention to the feedback was higher, thus this might have elevated the impact. This is supported by the post implementation data where students reported that the ReView system helped them to gain a better understanding of the criteria and standards expected within the assessments.
These results show that with support students can gain a better understanding of the assessment criteria and standards involved in assignments and as a result they are able to make sound judgements about the level of their work, which according to previous research (see for example, Boud, 1995;Black & Wiliam, 1998;Boud et al, 2010;Falchikov and Boud 1989;Boud, Lawson & Thompson, 2011) impacts on student performance at the time and post-graduation.
The correlation analyses showed that for the first task, even though the students overestimated their performance, the marks they awarded followed the same pattern as the academics. This relationship showed that where students' self-assessment marks indicated a higher achievement in a criterion, the academics marks also showed that these as stronger components of the student's work. This mirroring pattern was evident for all the criteria in both the first and second assessment tasks. This indicates that students do have a self-awareness of how they are performing, that is where their strengths and weaknesses are, but until they have a benchmark to gauge from gaining a true understanding of the standards expected, they are uncertain how to accurately score in these criteria. The important support to give to students therefore is not only in making judgements about the elements of their work but how to rate these in line with the expected standards for the assessment.
When reviewing the impact of this process on students' engagement with the graduate attributes for their degree. The pre and post questionnaire results showed that student's awareness increased over the duration of the subject. The colour coding helped to make the links between the assessment and the graduate attributes clearer, which meant the assessment criteria was easier to understand. This simple step was seen to make a difference in students' understanding of how assessment criteria align with graduate attributes and can be implemented easily in any assessment design.

Conclusion
The results from this study show the ability of students to be able to understand criteria and standards for assessing, and so are able to make valid judgements on their work, a skill which is beneficial to both academics and learning in the educational setting, and beyond into the workplace. Although this study implemented the online system ReView to foster this development in a highly aligned structured fashion, it is important to note that it is the design of the assessment that is the key to these developments and not restricted to self-assessment using ReView. The important elements that need to be considered in designing assessment tasks are: to provide opportunities for students to understand the criteria and standards; to make the relationship between criteria and graduate attributes explicit; to include feedback that relates to the specific criteria, as well as focusing feedback on the criteria where there are large variations in gradings between academics and students; to design scaffolded O'Donovan, B., Price, M. and Rust, C. (2001). The student experience of the introduction of a common criteria assessment grid across an academic department. Innovations in Education and Training International, 38, 74-85. O'Donovan, B., Price, M. and Rust, C. (2008 Thompson, D., Treleaven, L., Kamvounias, P., Beem, B., and Hill, L. (2008). Integrating graduate attributes with assessment criteria in Business Education using an online assessment system. Journal of University Teaching and Learning Practice, 5(1), 34-48.

Notes
Note 1. The word 'subject' is used to represent an individual unit of study within a degree program.
Note 2. This project derived from work on an Australian Teaching and Learning Council (ALTC) funded project.
Note 3. ReView evolved from an innovative approach, originating in the Faculty of Design, Architecture and Building at University of Technology, Sydney Note 4. Ethics clearance was obtained for this study from the University Ethics Committee. Published by Canadian Center of Science and Education 11 Figure 3. Academics' view of a marking screen from a subject of study A: Colour-coded symbols next to the criteria represent one of five attribute categories in this particular university example, B: Data 'sliders': The black bar is the academics' slider. Triangles on the top edge of the data sliders are students' self-assessments (done prior to academics marking and not visible until academics have marked), C: 'Total' data slider: the black bar can be dragged causing the marks and bars on other criteria to move in proportion for benchmarking purposes.   Figure 6. Graph to show the relationship between students and academics marks over each criterion in task two