Using Digital Tools to Assess and Improve College Student Writing

Employers have continually indicated that writing instruction is much needed in higher education across all majors. It has become more imperative now than before to better prepare our graduates for professional success in an age of increasing writing necessity, data analytics and reporting, and technical sophistication. Writing assessment in a class setting has learning goals and needs to be differentiated from a mass testing evaluation context. When learning to write well, especially relating to subject-specific content, feedback is necessary. Performing analysis and evaluation, then providing explanation and recommendations takes time. Newer digital tools can provide formative feedback; and therefore transparency about grading as well. Among teaching tasks, grading assignments consumes the majority of online faculty time. This study identifies what type of online grading could take up the majority of faculty time and specifies estimates of time needed for such grading. Faculty workload is high in adopting an optimal combined formative and summative assessment model. Results of the study might help develop more sound policies of academic support. Faculty might use the study‟s information for better curricula planning and improved utilization of student assistants.


Introduction
Employers will increasingly emphasize soft skills particularly communication, team work, and critical thinking.When hiring new graduates, employers place the highest importance on soft-skills and the lowest importance on academic reputation (CBI/Pearson surveys 2014 and 2016; European Commission 2011; Finch et al., 2013).The 2014 GMAC corporate recruiters" survey determined that, on average, employers ranked communications skills twice as important as managerial skills.For all majors, among the communication skills, both employers and educators emphasize the importance of good writing skills, but they continue to bemoan the lack of such skills (Clare, 2002;Shellenbarger, 2012;Simkin et al., 2012;Skapinker, 2013;Stevens, 2005).At least two major recent studies concluded that it is possible to earn a four-year undergraduate degree without ever doing much writing (Arum and Roksa, 2011;Glenn, 2011;Roksa and Arum, 2011).It is now widely recognized that enhanced writing instruction and academic support are much needed in higher education.This paper highlights the importance of considering time in any deliberation to rethink conceptions and procedures to address college writing instruction.
Offering tutor assistance and remediation courses are still prevalent academic support practices but have been proven not enough.There are common features in academic writing that can be taught generically (Etherington, 2008).However, while generic writing instruction focuses on surface linguistic factors, college writing practices need students to both construct and reflect knowledge (Paxton and Frith, 2014;Somerville and Crè me, 2005).Perhaps the biggest challenge in writing instruction is the view that a student need not use correct punctuation or sentence structure if the overall information is somewhat organized and clear.Its advocates urge using what Educational Testing Service (ETS) adopted in the 1980s; an approach used since to score thousands of essays for its standardized tests used for school admission criteria.Called "holistic scoring", it is now a methodology widely used -because of its virtues of "simplicity, efficiency, and equity" (Dyrud, 2003, pg. 79).What is not mentioned is that ETS and others who popularized its holistic evaluation strategy were performing only summative evaluations of thousands of hand-written essays in hard-copies without returning feedback.instruction-learning purpose.There are also validity and reliability concerns related to holistic scoring (Boettger, 2010).When learning to write well, feedback is necessary because, according to Flower et al. (1986), then students practice the cognitive aspects of response and revision.They suggested that adequate feedback should include analysis and evaluation, followed by explanation of recommendations.Providing such levels of feedback takes time; their suggestion also assumes a college-appropriate English language fluency in student writing.It is no surprise then that faculty resisted incorporating writing into their courses because of (1) time spent on grading, (2) annoyance with errors in spelling and grammar, (3) frustration with reading papers without logical organization, (4) time spent investigating plagiarism, (5) trouble with subjective grading, and (6) late submittals by students (Boice, 1990).So the challenge remains as to how to instruct college students in both the communication aspects of discipline-based writing as well as the linguistic components integral to writing ability.
As learning management systems become more pervasive and even imperative, faculty might find it helpful to use online tools to assess student writing.These tools allow for formative assessment and feedback at various levels and also transparency about grading criteria.This essay reports from a research project to enhance student writing using digital tools.Specifically, this article reports the time requirements to grade student papers online and provide the needed feedback.It contributes to the emerging research on online instruction by identifying what type of online grading could take up the majority of faculty time and specifying estimates of time needed for such grading.Mandernach et al (2013) reported that, among all teaching tasks, grading papers and assignments consumed the majority of online faculty time (36.93%)followed by facilitating discussion threads (14.73%).However, they do not specify what types of activities and assignments were involved nor what level of grading.Similarly, Van de Vord and Pogue (2012) reported that faculty teaching online courses spent more than three times as much as faculty in on-ground courses in grading but there is lack of assignment and grading details.One can assume similar time demands for on-ground faculty grading online assignments in learning management systems that institutions increasingly prefer faculty to use.
In a literature analysis of 966 articles on education and educational technology relating to 'time', Barbera et al (2015) argue how it is necessary to make use of and be aware of the importance of the time-factor in educational practice and investigation.Summarizing the American Chemical Society's revised guidelines for bachelor's degree programs, Wenzel and Landis (2015) noted that the revisions include limiting of teacher-student interaction so that faculty and instructors have enough time for grading, and professional and curriculum development.They also observed the importance given to online and virtual instruction; and that "online activities that are substitutes for classroom instruction should be assigned at least the same instructor contact hour value as equivalent face-to-face class-room experiences".
There is a growing understanding that online instruction takes just as much time, if not more in many cases, than previously believed.As instruction and learning become increasingly multi-modal, research needs to incorporate a temporal dimension to provide a clearer picture on requirements to provide quality online writing instruction.Briefly, this research aimed to answer the following questions: A. What kind of assignment takes up the most grading time for faculty using online instructional tools?B. How long were the written assignments?C. What might be approximate grading time required to provide a helpful level of feedback for writing instruction?

Methodology
Similar to other studies that measured the time spent in various online teaching activities, this research examined time logs maintained by three graduate student assistants over three terms in two upper-level marketing courses.Each term, the student assistants helped to review between 120-130 essay assignments.Only graded assignments from students were included in this study.There are a variety of writing assignments used in learning assessment; in this study, the assignments were all essays because in the real workplace, all business majors are expected to produce essay-type written products such as reports and plans.
In the Business Communication course, students produced five graded essay-type assignments requiring incremental writing skills and course content complexity.The assignments ranged from 2 to 4 pages of single-spaced, Times-Roman, 12-point font size plus any graphics and references list.In the Principles of Marketing course, students produced four graded essay assignments.The assignments ranged from 2 to 4 pages of single-spaced, Times-Roman, 12-point font size plus any graphics and references list.
The graduate assistants were provided with orientation to the course content, curriculum design and management, sequence of assignments and assessment plan, and their responsibilities.Two of the three assistants had prior editorial/proof-reading experience.One student was an English major and the other a Business major.The third student was a Business major and had no prior editorial experience.All three had strong written English language competency.All assistants were trained in providing feedback relating to both surface language factors as well as subject content, using prior graded assignments.They were also trained on how to use the assessment tools in the learning management system.Each orientation and training session lasted an hour and half to two hours, and was conducted by the faculty leading the course.Faculty also assessed the student assistants" reviews on the first four-five assignments to ensure grading requirements and feedback standards were followed.
Briefly, the written assignment review process included reading (often more than once), analysis and evaluation, followed by explanation of comments and recommendations.Following this step, the assignments were graded using a rubric.Throughout the term, faculty concurrently evaluated the quality of feedback and grading, albeit randomly, to prevent any validity or reliability issues.Any question and improvements needed were promptly conveyed to the assistants.Faculty also randomly verified the total grading time registered in the learning management system, with assistants" time logs maintained by the university"s payroll office --for time recording accuracy.
Online text review tools have become quite sophisticated.They offer faculty the ability to explicitly advise on how to correctly prepare written products using language, rules, and conventions of their discipline.Faculty can also specifically instruct in linguistic aspects of writing, such as sentence construction, spelling, and punctuation, those egregious errors that can mutilate any essay beyond comprehension.For instance, in business or science, students need to know the difference between a proposal and a research report and how to prepare each using language and conventions of their field.Furthermore, writing as a communication skill in the workplace requires students to identify their purpose, know their audience, research the content, select a way to organize their ideas and information, and use appropriate vocabulary and grammatical conventions.
The learning management system used at the study site is a commercially available one.As part of its package, the evaluation tool for faculty use includes a small sample of ready-made comments for surface linguistic features.Called Quickmarks, individual faculty can add to the list and consequently also customize the kind of comments they want to provide.For example, in addition to sentence construction mistakes, faculty may identify and detail internal logical coherence deficiencies or the need for evidence to bolster a claim made.Additionally, faculty can opt to allow auto-grading for common grammatical and spelling mistakes; this feature may be used with caution because machine-learning is not yet there to recognize and comprehend individual creativity or knowledge-construction.Still, on the whole, students can see clearly where there is a pattern in the repeated mistakes they tend to make.
Although greater efficiency and effectiveness might be presumed in using the digital tools, the grading faculty still needs to read and review the product before providing relevant and appropriate feedback and a grade based on a rubric.Thus, the time logged by the graduate assistants covered the entire time needed to review a student"s paper from reading, reading over again (more usual rather than an exception), providing comments on surface linguistic factors as well as providing feedback on expected subject content coverage, and finally grading using a rubric and offering any other summary remarks.Their time logs included a standard practice recommended 15 minutes break after every 1.5-2 hours of review.

Results
Research question A: What kind of assignment might take up the most grading time for faculty using online instructional tools?
Based on the gap in the literature and informal observations, this research focused attention on essay-type written assignments produced in two courses.In one marketing principles course, this study looked at the time taken to assess four essays.In this course, the essays centered on reporting personal study and making recommendations in response to four assigned marketing research topics.The first two essays mainly required summarizing and applying textbook concepts whereas the latter two essays demanded research, analysis, and reflection to support recommendations.Additionally, this study examined the time taken to assess five essays in two sections of a business communication course.In this course, the first essay was a series of short-answer responses to questions based on a case-study.Following this preparatory assignment, the remaining essays were full-length essay-type assignments, ranging from informative descriptive content to preparing a persuasive recommendation plan for a business.
Research question B: How long were the assignments?
In the marketing principles course, each essay was two pages minimum to four pages maximum, single-spaced plus any appendices.Each student wrote four essays and the class had between 55-60 students.In the business communication course, each essay was two pages minimum to four pages maximum, single-spaced, plus any appendices.Each student wrote five essays, and each class had 35 students; together, the two sections of this course had a total of 70 students.
Research question C: What might be approximate grading time required to provide a helpful level of feedback for writing instruction?
From the total time taken to grade all four essays in the marketing principles class each term, the time logs across three terms revealed that assessment took between 96 -120 hours to review and give a helpful level of feedback.This translates into 24 -30 minutes to review and provide comments on each paper (averaging 12 -15 minutes minimum each page).
From the total time taken to grade all five essays in the business communication classes each term, the time logs across three terms indicated that assessment took between 67 -90 hours to review and give a helpful level of feedback.This translates into 22 -30 minutes to review and provide comments on each paper (averaging 11 -15 minutes minimum each page).
The table below summarizes the time needed to grade and review written assignments using digital tools included in the university learning management system (QuickMarks, rubric, Turnitin plagiarism check, and additional minor tools).Altogether, the tools helped faculty provide customized comments regarding language use, writing conventions, as well as subject-matter to help individual students.The time logs do not include time taken to resolve technical issues or system down-time.Takes 67-90 hours to review and give a helpful level of feedback Takes 22-30 minutes to review and provide comments on each paper (11-15 minutes minimum each page)

Discussion
This paper showed how, on the instructional side, time-factors need to be considered seriously.For too long, the time-factor had been ignored because time is there and it is simply taken for granted.Acknowledging the time requirements could provide an excellent opportunity to rethink and to make explicit conceptions and procedures by which to optimize temporal teaching and learning issues (Barbera et al., 2015;Bates et al. 2010).In using technology for written assignment review, the often mentioned benefit for students is the possibility of receiving more immediate, integrated, and continuous feedback that is cumulatively helpful.However, the reports do not include important questions concerning time needed to provide the immediate, integrated, and continuous feedback.This article draws from data collected for a larger study that demonstrated the possibility of providing detailed evaluation of student writing that triggers response, reflection, and revision.Analysis of multiple data points indicated that the technological innovation used for writing assessment enhanced students" overall ability to make revisions, their overall English writing, and even their ability to give thoughtful feedback to peers.Analysis of assistant time logs revealed how much time was needed to accomplish those deliverables.
This study is likely the first to explore in concrete terms the average time needed to review and grade junior and senior level student essay papers.Three different graduate assistants helped in the research.They did not know each other or meet each other.Analysis of their separate time logs provided important insights and confirmation of the time-intensive demand on faculty assessing college student writing using technological tools.The average time of 22-30 minutes to assess a two-page, single-spaced student essay corroborated many regular faculty"s personal experience.Given the class size typical of business courses, reviewing writing skills in addition to content knowledge is complex and time-consuming.Excluding technology glitches, faculty workload is high in adopting a combined formative and summative review process model.Innovation adoption is not for everyone because it involves risk-taking and resources.This is especially true of pedagogic innovations.As Everett Rogers, author of Diffusion of Innovations (1962) commented, "An innovative practice has consequences for both innovator and recipient".In order to promote technology-enabled review as a better practice for all faculty seeking to improve student writing in their particular subject area, academic support is much needed.There is no empirical study or existing literature that assesses the time-investment needed by faculty to test, prepare, and adopt new instructional technology to support student writing instruction.
Results of the log analyses might help develop more sound and humane policies of academic support.In teaching-oriented institutions, faculty in large classes (larger than 35) may be provided graduate student assistants.However, it is more common that their assistance is of limited value since often graduate assistant provides support to 2-3 faculty, as practiced at the study site.Findings of this study might be helpful in more feasible allocation of resources.The study"s findings would also be helpful to develop policies relating to graduate assistants at the institution"s writing center.Reviewing writing for formative and summative assessment purposes requires more time than "holistic scoring" assessment.
Faculty might use the study"s information for better curricula planning and improved utilization of student assistants.For instance, at first, assignments were planned for every two weeks.However, based on preliminary analysis of evidence from the first two quarters, faculty reduced the number of assignments in one course.The situation may be different under a semester system; but it is also important to note the total number of students enrolled.While the semester term may provide more time than a quarter system, a larger number of students does not expand the limited number of hours in a work day.Furthermore, based on student assistant feedback and mid-term review of student performance, faculty arranged individual student consultations to adjust for differentiated learning pathways.Such timely additional interventions benefited students in offering relevant and specific evidence-based guidance adjusted for pace of learning.
The present study presented several limitations.Faculty and graduate reviewers assessed student essays that had already passed one-two levels of review for improvement.In other words, students underwent preparatory work beforehand and work submitted for assessment represented their best effort.Future research might examine how much time was expended to revise student essays before those are submitted for grading, what kinds of revisions were mostly recommended, and compare the information on how much more time is needed to provide feedback during grading.
Another limitation is that the study includes only overall time logs maintained by three graduate assistants.The assistants were not bound by time limits, such as five-ten minutes per page.They were reminded to be thorough and thoughtful.Based on their suggestions and faculty analyses of student performance, one-on-one meetings were also arranged to help certain students.How much time did faculty spend in analyzing student performance trends and specific areas needing improvement?How much "real time" was spent in the one-on-one meetings?Were follow-up face-to-face meetings needed?How consequential were these meetings on ensuing assignment performance?In focusing on time use and consequently time efficiency, improvements in learning and pace of learning as expressed in temporal processes emerged as important topics deserving further study.
In instructional pedagogy, the implementation of technology-mediated review is generally found to be time consuming as user-reviewers need opportunities to gain expertise with the innovation (i.e.new for the user) and to explore their questions.To complicate matters, mid-course, the institution/vendor changed the appearance and some content of the learning management system.Thus, woven into the need to examine myriad temporal dimensions in online instruction are many questions relating to technology and the assumed performance stability.
As increasing numbers of students seek higher education and technology continues to transform the global economy, pedagogic thinking and high impact practices in writing teaching and learning would be more effective.Faculty performance and productivity, as well as student performance and learning, can be enhanced by digital tools but those cannot be assumed to be static, stable, and without temporal processes and events that undergird all the activities.

Table 1 .
Student Writing Assessment Time Taken Per Term