Student Awareness and Use of Rubrics in Online Classes

The design, development and deployment of online instruction has become standard practice. The focus of the study was on student perceptions of course rubrics and not on the rubrics, themselves, or the instructors. In order to improve student engagement online we conducted an exploratory study of the awareness and perceptions of course rubrics. Fifty graduate students completed an online survey at the end of the semester about their awareness and perceptions of course rubrics. All students reported that they were aware that course rubrics existed. They indicated that they had learned about this information through the course syllabus, professor announcements via email and posts to LMS. Most students reported reviewing rubrics prior to submitting an assignment. One of the key findings from this study was that students see rubrics as a mechanism for scaffolding their performance, and thus, instructors need to focus more effort on designing rubrics to accomplish more than student assessment.


The Problem
In spite of all the findings that report that rubrics are useful tools for students and that students use them in various ways, the current authors observed that initial submitted assignments submitted in their courses fell short of the stated criteria based on the course rubrics that were available to students from the beginning of the semester. Even at the end of the semester, some submitted assignments failed to meet many of the stated criteria. Thus, the authors were interested in whether students were aware of existing course rubrics and whether they used them in some capacity. Again, the focus of the study is on the student perspectives of course rubrics and not on the actual course rubrics or the course instructors. The current study serves as an exploratory study and is designed to answer the following research questions: Were students aware that rubrics were available in their course?
How did students know that rubrics existed? Did students review rubrics prior to submitting an assignment?
In what ways did students use rubrics?

Relevant Scholarship
The National Center of Educational Statistics (NCES) has continued to show that online learning is not only expanding but is in fact becoming the instructional delivery choice for many students. This new reality has prompted a re-definition of distance education. The US Department of Education now defines distance education simply "as a formal education process in which the student and instructor are not in the same place" (Parsad, Lewis, & Tice, 2008, p. 1). The availability of online learning is becoming commonplace for most institutions of higher education. Bowers and Kumar (2015) stated that "Online courses offer numerous advantages to students such as convenience, flexibility and access to education. Due to their busy lifestyles, students are looking for convenience for when to read, when to complete the assignments, when to watch videos, part time workers are on 'stand by' for hours and many students have two jobs, the need for time is absolutely essential" (p. 1). Surveys across degree-granting higher education institutions show large increases in student enrollment in online courses (up to 21%) during a time period when overall enrollment growth in higher education dramatically decreased (to 2%) (Allen & Seaman, 2010). Distance education learning programs at postsecondary institutions expand to meet student demand (Parsad, Lewis, & Tice, 2008). Bell and Federman (2013) suggested that the primary reasons why post-secondary institutions are increasing the number of online courses offered include "a need to generate new revenue streams, expand access, offer students greater scheduling flexibility and the freedom to work at their own pace, and curb increasing costs" (p. 166). Clearly, online classes have potential benefit for both the learner and the institution of higher learning.
Even though there has been a dramatic increase in the number of students enrolled in online courses (Bell & Federman, 2013), major issues, including higher attrition rates and quality of delivery, exist and need to be addressed. Every effort needs to be made to assure student success in online courses, and rubrics are one instrument for informing design and improving online delivery of courses.
However one of the issues with online instruction is the lack of sustained engagement with a given course. Typically, in face-to-face courses the performance expectation is negotiated with an instructor over time. The student has numerous opportunities to ask questions in order to refine their understanding of an assignment. Given the remote nature of an online course students do not maintain consistent involvement with the material and are at times lost as to what level of performance is expected (Keengwe, Adjei-Boateng, & Diteeyont, 2013).

Student Engagement
The key to improved performance is engagement. Carini, Kuh, and Klein (2006) note that "Student engagement is generally considered to be among the better predictors of learning and personal development. The premise is deceptively simple, perhaps self-evident: The more students study or practice a subject, the more they tend to learn about it" (p. 2). In this paper we examine the use of rubrics as tools that promote student engagement in course content on several levels.
One view of engagement is exerting effort over time to accomplish a task (Pintrich & Schrauben, 1992). However, engagement is not simply performance but also requires a cognitive commitment. Video games designers have known for a long time that there is more to sustaining game play than simply colorful graphics. Most games provide not only a challenge but also the means to meet that challenge. The player has to stay mentally aware in order to acquire the necessary skill or treasure to proceed in the game (Sylvester, 2013). A simple way to support the cognitive factor is through the use of rubrics, especially in an online environment. It is a clear and present mechanism that helps to guide students in performing a task.
Rubrics have been used in traditional classrooms for assessment for over 20 years (Dirlam & Byrne, 1978;Popham, 1997). With the increase of online classes, rubrics have been incorporated into many, if not most, online courses. In fact, rubrics are required by the Quality Matters online guidelines (Pollacia & McCallister, 2009;Sener, 2006) and match the best practice guidelines related to improving student assessment (Grant & Thornton, 2007). Andrade and Du (2005) examined student perspectives on rubric-referenced assessment with 14 undergraduate teacher education students who had completed an education course where there was regular use of rubrics. Through focus group sessions, these researchers reported that students were "consistently positive" about course rubrics. From the discussions, the most frequently reported purpose of rubrics was to "communicate the teachers' expectations". These students also reported that rubrics "help identify strengths and weakness in their work" (Andrade & Du, 2005, p. 3). Atkinson and Lim (2013) reported improved assessment in online courses after using the LMS Rubrics tool when a university-wide change was implemented. They gathered feedback from both students and teachers. Student feedback included both positive and negative comments. Some students believed that the rubric provided clarity of teacher expectations and solid feedback to improve on subsequent assignments. Generally, educators accept that "a rubric is a scoring tool that teachers use to assess student learning after a lesson. Using a set of criteria and standards (directly tied to the stated learning objectives), educators can assess each student's performance on a wide variety of work, ranging from written essays to class projects" (Lewis, 2016). Assessing student learning can be a difficult task that many instructors dread, because of their viewing assessment as making judgments about the adequacy or acceptability of student performance (Cizek, 2009;Guskey, 2003;Popham, 2009). Use of rubrics that are shared with learners as part of the assignment guidelines helps make the rules of measurement part of the learning and understanding experience (Nitko & Brookhart, 2011;Russell & Airasian, 2012).
Two approaches were assessed to support students' improvement including a rubric-based self-assessment and the teacher's oral feedback. These researchers concluded that the rubric-based self-assessment was found to be worth hes.ccsenet.org Higher Education Studies Vol. 7, No. 1; investing a student's time; while the teacher's oral feedback required time from both the student and teacher (Barney, Khurum, Petersen, Unterkalmsteiner, & Jabangwe, 2012).
A review of rubric use across 20 articles in higher education (Reddy & Andrade, 2010) focused on student perceptions of rubric use. These researchers concluded that students' positive response to rubrics might be related to the fact that rubrics were made available before an assignment was submitted. Generally, these authors stated that studies of rubrics in higher education have been undertaken to increase student achievement, improve instruction, and for program evaluation. As Reddy and Andrade (2010) suggest a successful, valid, and reliable rubric can help guide instructors to address on-going course improvements. In this way rubrics help to inform instructors about areas where the course needs improvement.

Participants
Fifty graduate students (15 males and 34 females) enrolled in either online Educational Psychology or Instructional Design and Technology courses at West Virginia University served as participants. Students were invited to participant through email, LMS, and class announcements. The participants were provided with an online link to the survey which was developed using Qualtrics. Survey questions are presented in the Appendix.

Sampling Procedures
The graduate students were contacted twice within the final weeks of the semester. Ninety-eight students were enrolled across the six graduate courses, and 50 students (51%) responded to the survey. Procedures were approved by the University's Institutional Review Board for the Protection of Human Subjects. Participants were identified as white which corresponded to the racial and ethnic profile of the university profile for 2014 and 2015 enrolled students as reported by the university where Caucasians made up 91% of the student population in both years (West Virginia University Institutional Research, 2017). Information about ethnicity and socioeconomic status (SES) was not reported by the participants.

Materials and Procedures
Participants were asked to complete a short online survey that asked about student awareness of existing rubrics offered by instructors in all six graduate courses. Based upon discussions among the three instructors, conversations with students, and instructor observations of submitted student assignments, the current investigators wanted to survey the following questions: 1) Were students aware that rubrics existed in the various courses? and 2) If they were aware, were they using the course rubrics and, if so, how? Survey questions were developed to primarily solicit information concerning these two issues, but the survey also included open-ended items to gain further insights pertaining to the use of rubrics from students. For example, students were asked on the survey "If you used the course rubric in some other way than those listed [in the survey], describe how you used the rubric".

Results and Discussion
Results showed that all 50 graduate students indicated that they were aware of various rubrics in their courses. Given that all rubrics were available to students from day one of the class, and that all instructors regularly reminded students of the available rubrics prior to an assignment being submitted and then again along with feedback on a graded assignment, it is not surprising that students reported being aware of the various course rubrics. It is noteworthy to point out that students responded to this question at the end of the semester; after receiving numerous reminders from course instructors about the course rubrics. A future study might ask about awareness of the course rubrics after the first week of the semester and/or after the first course assignment has been submitted.
Findings on how students knew that rubrics existed are presented in Figure 1. When asked how they knew that rubrics existed, the graduate students indicated that they knew because of the course syllabus (n=38 or 76%), from professor announcements (n=35 or 70%), from professor emails (n=29 or 58%), and from professor posts to online discussion threads (n=27 or 54%) (see Figure 1). Since participants could respond to multiple options, the total numbers within and across the four options do not equal the number of participants. Of 50 participants, 48 indicated that their professor reminded them of the rubrics (96%) with two individuals indicating that their professor did not remind them of the rubrics (4%). An important take-a-way from this finding is that instructors should use several online and course features to inform students about the existence of class rubrics. When asked if the student reviewed rubrics prior to submitting an assignment, all 50 graduate student participantresponses indicated that they reviewed the rubrics prior to submitting an assignment. When asked in a separate questions if they reviewed them more than once, 46 of the 50 individuals (92%) indicated that they reviewed them more than once, and 4 participants (8%) reported that they did not review rubrics before submitting assignments. There is an obvious contradiction across the responses to the two questions. Without follow-up questioning, it is unknown as to why four students answered positively in one question while responding differently in the second question.
The course professors involved in this investigation observed that not all students reviewed rubrics prior to submitting an assignment especially during the early weeks of the semester. Criteria related to required content in assignments, specific formatting requirements (e.g., references must be in APA style), length requirements, etc.
were not met by all students in the early assignments. By the end of the semester, it appeared that students were reviewing the grading rubrics prior to submitting assignments based on improved assignment scores.
According to the participants who were asked how they used a rubric to complete an assignment, most of the students (94%, n=47) reported using the rubric as a checklist to make sure they complied with the required elements in the assignment (see Figure 2). A majority of the students (86%) also indicated that they used the rubric to structure the assignment. Finally 60% of the respondents indicated that they used the rubric as a self-assessment tool prior to submitting the assignment.  Vol. 7, No. 1; Graduate student survey responses illustrate how the rubric was used to help structure and self-monitor an assignment during the process of starting, reviewing, and self-assessing before submission. Thus, the survey data show how rubrics offer students the flexibility to use this tool to assess for learning as well as a means of assessment of learning-supporting both formative and summative assessment processes (Moss & Brookhart, 2010;Stiggins & Chappuis, 2011).
The comment from this student reinforces this view of rubrics as serving both formative and summative assessment purposes-depending on the assignment and the student response to the task: Sometimes I look at the rubric religiously to structure/plan my assignment and to assess my assignment before submission. Other times, I rely on memory of the first time I looked at the rubric (if my level of comfort with that particular assignment is high) during the planning and assessment stages of completing an assignment. It often depends on my level of anxiety over the assignment whether I will look at the rubric once and then be done with it (which I almost always look over at least once, no matter what), whether I will use it simply to check my assignment before submitting it, whether I will use it carefully, etc. In fact, I would offer the suggestion of providing more options than just "yes" or "no" for the above questions in this survey. I could almost rate on a scale the ratio in which I will attend to a rubric or attend to it less, and how often I use it for certain purposes.
A final survey question asked the participants to rank order the uses of rubrics in order of how, in their opinion, they are most effective (see Figure 3). The survey item included the following three options: To help structure an assignment, as a checklist before submitting an assignment, and to self-assess my assignment. As presented in Figure 3, 23 graduate students picked "To help structure as an assignment" as their first choice, 10 graduate students selected the same option as a second choice, and 1 graduate student selected that option as a third choice. In response to the option, "As a checklist before submitting an assignment", 6 graduate students selected this as their first choice, 15 graduate students selected it as a second choice, and 13 graduate students selected it as a third choice. In regard to the option, "To self-assess my assignments", 5 graduate students selected this option as a first choice, 9 graduate students selected it as a second choice, and 20 graduate students selected it as a third choice. It is clear from the data that the participants reported using course rubrics to help structure an assignment more than any other provided option. This is similar to the Atkinson and Lim (2013) finding that students stated that rubrics provided clarity about teacher expectations. Participants were asked to add any personal thoughts about using rubrics in classes, and several of those responses are presented verbatim within this paper.

Summary Comments and Future Research
This investigation served as an exploratory study to gain better understanding of how students perceived and used rubrics, so that use of rubrics could support improved design and delivery of online instruction. The findings fit with the course professors' observations. Having graded many assignments across the semester, it was apparent that these students became aware of the rubrics and began to use them as a checklist prior to submitting subsequent assignments. The current survey findings confirm that the course professors' efforts to make the students aware of the rubrics worked. The various ways that students used the rubrics are similar to what other students have reported (e.g., Andrade & Du, 2005;Atkinson & Lim, 2013). No gender differences were found across any of the survey questions for the participants. This fits with what other studies reported (Andrade & Du, 2005;Hafner & Hafner, 2003).
Rubrics as documented are a very good assessment tool (Panadero & Jonsson, 2013). However, what was found in this study was that rubrics served as an excellent scaffolding mechanism. The students would use the rubric as a checklist as well as a way to structure the assignment. As one student noted: I heavily rely on rubrics as a guide to help structure my assignment. It helps clarify expectations and enables me to focus on relevant details and reduces time spent on details that are not pertinent to the important concepts. Rubrics are even more helpful when atypical terminology is specifically described. For example, in a previous semester the rubric included a section that was referred to as the "skeletal system". Not having a background in Education or Instructional Design, I incorrectly assumed the term was referring to the outline and organizational flow. I occasionally struggled to apply terminology that was somewhat esoteric in nature.
This is especially true for online students. Rubrics help to increase student presence by helping students understand what they need to do for the assignment.
Student survey data also supported the role of rubrics as contributing to the building of an online social presence between the instructor and students as a community of learners. Student ratings of rubrics and their reflective comments show that a rubric helps to blend together the rules, expectations, and guidelines for an assignment into a coherent social contract between the instructor and student that can serve as both a formative or summative assessment tool for a given assignment (Roblyer & Wiencke, 2003).
However students also reported that not all rubrics were helpful. As one student wrote: I find that some rubrics are more helpful than others. Sometimes they don't provide enough information and can be somewhat ambiguous, leaving me wondering what the professor is looking for. Other times there seem to be items within the rubric in which it is hard for me to make the connection as to why that item is relevant (why is this important? can I picture what it would look like if I were to fully address this item in my assignment to my professor's satisfaction? what am I supposed to learn from this?). It is as if sometimes I did not receive enough prior knowledge to access the connection I am supposed to be making by looking at the rubric, or certain items on the rubric. I will also add that I have had excellent class rubrics that made a big difference for me in my ability to conceptualize and see the relevance of an assignment.
Rubrics are useful tools, but they need to be carefully designed and integrated into the instruction. Consider the following student comment: Rubrics are helpful as long as they match the other information provided. It is frustrating to have information provided in the assignment instructions only to find out that the rubric lists different/additional information after you have already been working on the assignment.
Generally students were aware of the rubrics. However, what was discovered in the present investigation is that students use rubrics as a way to scaffold and guide their efforts in completing their assignments. Some of the responses to the open-ended survey items suggest that there may be a possible disconnect between the presented rubric with the specific assignment outcomes. In many courses, online students are using rubrics in lieu of in-class student-teacher exchanges where questions about assignments are asked and answered. What this study shows is that instructors, educators, and designers need to understand that rubrics are more than a simple hes.ccsenet.org Higher Vol. 7, No. 1; mechanism to score assignments. The rubric should not be viewed simply as an assessment tool but should be integrated directly into the overall design of the course. Further research might explore ways to design rubrics to provide scaffolding and guidance. Perhaps better training is needed for online instructors to assure that rubrics not only allow instructors to assess student learning but also that rubrics better meet the needs of students.