Refining a Competency Model for Instructional Designers in the Context of Online Higher Education

This study investigates the instructional designers (IDs) competencies essential for the context of online higher education, and has selected an instruction design unit in a research university as a case of investigation. To identify and compare IDs competencies at organizational and individual levels, this study employed a mixed method to collect and analyze data based on a validated IDs competency model by the International Board of Standards for Training, Performance and Instruction (ibstpi) as a framework. Throughout the study, IDs’ expected jobs/tasks and currently performed jobs/tasks were systematically analyzed, and the applicability of the ibstpi model in this specific context of online higher education was verified. Based on the empirical findings, this study proposed a refined competency model to improve IDs performance in human resources development and management practice.


Introduction
The rapid development of information and communication technologies (ICT) has enabled educational content to be delivered via various ways customizable to learners and their circumstances, and has drastically changed the landscape of higher education, with online education playing an increasingly important role.For example, Allen and Seaman (2014) conducted a survey on more than 2,800 colleges and universities in the United States, and found that the number of students taking at least one online class in 2013 had increased to an all-time high of 7.1 million, accounting for 33.5 percent of the total student population (Allen & Seaman, 2014).In addition, about 66 percent of academic leaders in 2013 considered online education as critical to the long-term strategy of their institutions (Allen & Seaman, 2014).New forms of online courses, such as Massive Open Online Courses (MOOCs), also contributed to the fast expansion of online higher education in recent years (Allen & Seaman, 2013, 2014).
The changing landscape of higher education in the online context also places new demands and challenges on instructional designers (IDs), as it often requires updated and/or different ID competencies as compared to IDs in face-to-face contexts.For example, IDs might need more specialized technical skills to deal with diversified educational technologies (e.g.different devices, operating systems, platforms, browsers) used for online learning (Koszalka, Russ-Eft, & Reiser, 2013;Sims & Koszalka, 2008).However, those specialized IDs competencies seem to be insufficiently addressed in the literature, and the existing competency models fail to differentiate instructional design (ID) practice between the face-to-face context and the online context.One exception is the competency model developed by the International Board of Standards for Training, Performance and Instruction (ibstpi), which specifies a set of competencies for e-learning specialists.However, ibstpi's model is still too generic for the organization-specific context of online higher education, and does not reflect organizational characteristics such as strategy, vision, and mission.
Given that online higher education is projected to continuously grow and expand, this study reports on the development of a competency model for IDs in an institute (the institute) that designs online curriculum/programs for a four-year public research university (the university) in the northeastern region of the United States, by analyzing jobs/tasks, skills, knowledge, and abilities (SKAs) at organizational and individual levels.The real names of the institute and the university were removed from this paper to protect the anonymity of the IDs and the personnel working there.This study employs the mixed method approach to validate the proposed IDs competency model, and asks the following research questions: 1) At the organizational level and individual level, what competencies, skills, knowledge, and abilities are expected for IDs to carry out their jobs/tasks at the institute?
2) Compared to ibstpi's IDs Competency Model, what ID competencies are commonly and/or uniquely found at the institute?
3) What are the key domains and competencies for the context-specific IDs competency model in this study?
Based on the results, we have proposed a competency model unique to our case study context that emphasizes the changes in online ID practice.The proposed competency model can be used to inform practitioners and/or department of human resource development and management (HRD/HRM) to improve ID performance through conducting needs assessments and implementing training and non-training interventions.

Literature Review
Competency models and job/task analysis are two methods that are widely used in various professions to achieve organizational goals.Despite an essential difference between them, specifically that a competency model is future-oriented while job/task analysis is focused on present facts, they are closely related when dealing with SKAs and other characteristics that are required for desired performance.As a result, the literatures reviewed in this section cover the following topics: definitions and processes to conduct a competency model and a job/task analysis.Furthermore, the integration of job/task analysis and competency models was also explored focusing on how they supplement each other in informing and improving HRD/HRM.

Competency Model
First of all, the term "competency" has various definitions.McLagan (1997) defined competencies as a set of work-related and people-related characteristics including knowledge, skills, attitudes, tasks, outputs, and results; Gupta, Sleezer, and Russ-Eft (2007) had a similar definition that described competencies as observable and measurable knowledge, skills, attitudes, or behaviors that enable one to perform the activities of a given occupation or function at the standards expected in employment.Clark and Armit (2010) further pointed out that in some cases competencies could lead to superiority or competitiveness in working performance.
A competency model, in essence, is an organization of competencies, which offers a conceptual framework based on a certain job, job family, or multiple job families that provides information on knowledge, skills, attitudes, capabilities, and jobs/tasks required to perform job responsibilities (Gupta et al., 2007;Klein, 2004;Koszalka et al., 2013;Marrelli, 1998).According to Rothwell (1996), competency model is "the systematic process of discovering and analyzing important human performance gaps, planning for future improvements in human performance, designing and developing cost-effective and ethically justifiable interventions to close performance gaps, implement the interventions, and evaluate the financial and non-financial results (p.3)".
In terms of the usefulness of competency models, they enable employers to determine strategic competencies for present and future needs in performance at an organizational level (Murphy, Hanchett, Olmsted, Farber, Lee, Hass, & Streed, 2012;Koszalka et al., 2013;Lucia & Lepsinger, 1999).Specifically, by assessing whether an organization's performance aligns with its strategic vision and mission (Cekada, 2010), human resource development and management (HRD/HRM) interventions can be administered in areas such as recruitment, assessment, mentoring/coaching, succession planning, training curriculum design, career development, performance management, and provision of feedback (Bock & Ruyak, 2007;Iqbal & Khan, 2011;Koszalka et al., 2013;Lucia & Lepsinger, 1999;McLagan, 1996McLagan, , 1997;;Sanghi, 2007).At individual-level an individual level, competency models can be used to inform on potential employees' preparedness and current employees' professional and personal development (Iqbal & Khan, 2011;Koszalka et al., 2013).Lucia and Lepsinger (1999) outline two conceptual frameworks for developing competency models: starting with a validate competency model and starting from scratch.Starting with a validate competency model is an approach that creates a model based on an existing competency model that includes generically required competencies.This approach can be appropriate when a competency model has been developed for professions that have general roles, functions, and positions involving limited technical skills and knowledge (e.g.leadership and management).
The other method, starting from scratch (Lucia & Lepsinger, 1999) is a way to reflect context of professions in organizations.Starting from scratch is an approach that develops a competency model by internally carrying out data collection and analysis of jobs, functions, or roles in an organization.As a result, a competency model following this approach should exhibit context-based specification and uniqueness of competencies of a certain profession in a specific organization (Lucia & Lepsinger, 1999).
However, determining whether a competency is essential should be context-based, as using a generic competency model even for the same profession would not fit all organizations.It is necessary to develop a tailored or customized competency model covering not only the generic competencies but also the organization-specific competencies (Lucia & Lepsinger, 1999).

Job/Task Analysis
Job/task analysis, referring to "a systematic process for collecting and analyzing information about a job especially focusing on the work environment where the employees function (Prien, Goodstein, Goodstein, & Gamble, 2009, p. 11)", is intended to identify employees' views on jobs and ways to perform their jobs at work (Alliger et al., 2007).Methodologically, a critical incident method can be applied to collect employees' behavior-related data by using a Job Analysis Questionnaire (p.258) and Competency Interview Worksheet (p.268) (Gupta et al., 2007).Therefore, following the job analysis process, it is possible to discover how individuals view work activities or roles, quality and quantity of job performance in which they engage, knowledge, skills, and abilities required to perform the job, and current social and physical environment or conditions (Brown, 2002;McClelland, 1995;Prien et al., 2009).Data sources for job/task analysis include self-reports, observations, interviews, document review (e.g.job descriptions, standard operating procedures, performance standards, literature on best practices), questionnaires, and surveys (Brannick, Levine, & Morgeson, 2007;Cekada, 2010;Noe, 1999;Prien et al., 2009).
The immediate outcomes from job/task analysis are (1) job description that describes tasks required for a job; and (2) job specifications that specify education, knowledge, skills, or physical or mental abilities required to adequately perform a job (Alliger et al., 2007).Furthermore, job/task analysis can also improve HRD/HRM in areas such as job evaluation, job/team/system design and redesign, workforce planning, performance appraisal, performance management, training needs assessment and training, recruitment, and candidate selection (Alliger et al., 2007;Brannick et al., 2007;Brown, 2002;Prien et al., 2009).

Integrating Job/Task analysis in Competency Model Development
As described above, both competency models and job/task analysis are procedures to identify and describe critical job requirements for the purpose of improving individual and organizational performance (Shippmann, Ash, Battista, Carr, Eyde, Pearlman, Prien, & Sanchez, 2000).On one hand, a competency model focuses on an organization's business objectives and strategies, and tries to define future job requirements, starting with top executives (Lievens et al., 2004;Sanchez & Levine, 2009).In other words, it deals with how objectives are fulfilled or how work is achieved (Alliger et al., 2007;Campion, Fink, Ruggeberg, Carr, & Phillips, 2011;Sanchez & Levine, 2009;Shippmann al., 2000).
On the other hand, job/task analysis focuses on describing actual job/task performance.Therefore, it focuses more specifically on task-related information and subject matter expertise, and mainly deals with questions regarding "what" -what tasks are accomplished by the performers (Lievens, Sanchez, & Corte, 2004;Shippmann et al., 2000).Given that, while a competency model is intended to influence performers' job behaviors to align with organizations' strategies, a job/task analysis is to describe performers' job behaviors (Sanchez & Levine, 2009).Using a competency model helps employers actively fulfill HRD/HRM as well as broad and dramatic organizational change and/or development, whereas using job/task analysis allows employers to refine and/or improve practices (Campion et al., 2011).
In conclusion, in order to improve organizational performance, it is recommended to integrate job/task analysis in competency model development, and thus take advantage of both approaches.Specifically, when detailed job/task statements are included in a competency model, the model becomes more rigorous and informative for HRD/HRM by providing sufficient task statements incorporating the organization's business strategy and goals (Sanchez & Levine, 2009;Shippmann al., 2000).

ibstpi's IDs Competency Model
The conceptual framework in this study is the IDs Competency Model developed and validated by the International Board of Standards for Training, Performance and Instruction (ibstpi) (Koszalka et al., 2013).With 22 competencies and 105 performance statements classified in five domains (see Table 1), the ibstpi model covers various instructional contexts including professional training, higher education, and K-12 education, and can be applied to both face-to-face and online learning environments.Furthermore, the ibstpi model addresses the needs for specialization and professionalization of instructional design in online learning settings, and has identified competencies specific to the profession of e-learning specialists, including ID specialists, Analyst/Evaluator, ID Manager, and e-Learning/Instructional Technology Specialist (Koszalka et al., 2013).

Research Context
By starting the nation's first correspondence course in 1892 and implementing a pioneering accredited online education program in 1998, the university has been a frontrunner in distance education in higher education.The case selected for this study is the institute, which has played a pivotal role in the expansion and development of online education at the university.The institute helps the College's five academic units to design, create, and sustain professional certification and degree programs offered online since 1999, operating more than 120 courses in 14 programs.In addition to accredited programs, the institute also creates open education resources (OER) that are freely available for non-commercial use under a Creative Commons license, and has delivered three MOOCs on the Coursera learning platform since 2013.
Working with faculty members affiliated with the five academic departments, the institute envisions the following three goals: serving lifelong learners through distance education, preparing undergraduate students for lifelong learning, and contributing to the global knowledge commons.In order to achieve this vision, the institute has a learning design team consisting of 14 people who specialize in five different roles including learning designer, videographer, analyst, research associate, and multimedia specialist.

Data
Data collected for this study were from two major sources: (1) official documents at the organizational level, and (2) survey results at the individual level.The 20 open-ended questions were developed referring to the Job Analysis Questionnaire and the Competency Interview Worksheet suggested by Gupta et al. (2007).The 105 Likert-scale questions were created based on the 105 ID competencies of the ibstpi model, which ask IDs to rate the criticality of those competencies based on their personal working experience in the institute.The rating of "5" represents the highest level of criticality while the rating of "1'' represents the lowest level of criticality.

Participants
Analysis of data and the reporting of the results of those analyses are fundamental aspects of the conduct.In this study, seven IDs in the institute's Learning Design Team participated in the survey.To protect confidentiality and prevent any potential damage of the IDs' job security, the responses to the survey were collected anonymously using the online survey tool Qualtrics.All personal identifiable information such as demographic and career/professional experience was excluded from the data analysis.

Analyses
This study primarily employs the starting with a validate competency model approach suggested by Lucia and Lepsinger (1999) to develop an IDs competency model for the online higher education context (Lucia & Lepsinger, 1999), adding components from starting from scratch for data collection.In addition, using the two different types of data and analysis approaches in accordance with types of data, it was able to triangulate findings from each method so that the validity of the study results could be increased (Greene, Caracelli, & Graham, 1989).
Particularly, the ibstpi model was used to quantitatively measure criticality of each competency in the context of the institute.More precisely, in order to examine the criticality of the IDs competencies identified by the ibstpi model as perceived by the institute IDs, mean values were calculated from the 5-point Likert scale questionnaire results, which reflected the perceived importance of the competencies in the ibstpi model.In this study, we consider the mean value of 4.0 (out of 5.0) or higher as the criterion for selecting essential competencies for the proposed competency model.
The model was also used as a conceptual framework to analyze qualitatively-collected data from organizations and IDs to identify specifically required IDs' SKAs of the institute to carry out their performance.To be specific, using textual data analysis, jobs/tasks found from data collected at organizational and individual levels were categorized and conceptualized by job context (Brannick, Levine, & Morgeson, 2007).First, by using ibstpi's model as pre-determined structural codes, key jobs identified from official documents and survey responses were categorized with similar attributes so that each category would comprehensively characterize a concept.Second, the categories were conceptually classified into broader domains to allow comprehension of the data as a whole (Hennink, Hutter, & Bailey, 2011).Third, the domains were compared with competency domains of ibstpi's model, which enabled the domains to be identified as generic or Institute-specific competencies.Based on the results, an IDs competency model was specifically constructed for the institute's context.The required competencies to be included in the competency model were informed by the frequency of their appearance in the documents and the mean values of their perceived criticality.For example, when competencies were frequently found in the documents at either organizational or individual level, or their mean values were higher than 4.0, they were classified as essential competencies for IDs at the institute and are included in our IDs competency model.

Conclusion
Corresponding to the increasing importance of IDs roles along with rapidly expanding online education in higher education, this study examined competencies critically required for IDs at an instructional design institute affiliated with a four-year research university.This study employs the ibstpi's generic IDs competency model as a conceptual framework to develop the context-specific competency model for instructional designers working in online higher education settings, and draws key conclusions from the results of contextual data analysis and descriptive analysis of Likert-scale questionnaire.
The job/task analysis has identified: (1) the jobs/performance expected by the university and the institute to achieve their strategies; and (2) the main responsibilities and practices perceived by the IDs.By measuring the criticality of competencies in ibstpi's model, critical IDs competencies in the institute context were identified.
Integrating results from the two methods, this study has suggested seven domains to be included in the institute-specific ID competency model, including: Professional Foundations, Planning & Analysis, Design & Development, Evaluation & Implementation, Management, Education, and Research.This model exhibits unique characteristics of the online instructional design practice in higher education: first, research-related competencies are highly valued by IDs at the institute as a result of the university context; second, IDs were expected not only to provide service to faculty, but also to educate them about effective design principles and practice.

Implications for Practice
The refined competency model in Table 3 is informative for the institute leaders and HRD/HRM practitioners.First of all, the model itself would be an opportunity for them to consider competencies and performance.By casting a question on the status of IDs' currently-retained competencies in connection with their performance level, the leaders and HRD/HRM practitioners might think more seriously about whether they need to align the model with the IDs' work, consequently followed by defining the right competencies to the right extent to be adopted from the model.On the other hand, with respect to Research and Education, which are perceived to be significant at both individual and organizational levels, non-training intervention, especially HRM-related intervention, can be applied.Specifically, the institute can officially add related components to current IDs jobs/tasks so that not only would IDs be able to actively engage their responsibilities, but also the institute could offer strategic ways to develop IDs competencies.For instance, it would be recommended to re-define an IDs job description by removing competencies irrelevant to the IDs working context or the organization's values, in order to make competencies consistent and explicit between IDs and the organization/institute.Also, by being re-positioned to projects purposefully designed to embed the required competency components, IDs would be able to improve their competencies by carrying them out.
The proposed model also can be used as an assessment tool to identify the gaps in terms of level, extent, and types of domains between IDs competencies expected by the organizations (the university and institute) and the retained competencies perceived by the individuals.
Different from the open-ended performance evaluation form currently used, the context-specific self-performance assessment form should elicit the following key information from the IDS: (1) IDs' perceived importance of a performance statement; (2) IDs' self-evaluation of their current performance level; and (3) IDs' perceived training needs to close the gap between (1) and (2).The findings from gap analysis are informative for both the institute leaders and HRD practitioners in planning proper interventions to align IDs' performance with the organization's strategy and expectation.

Table 1 .
ibstpi's IDs Competency Model Note.Summarized from Instructional Designer Competencies: The Standards (4th ed.), by T. A. Koszalka, D. R. Russ-Eft, and R. Reiser, 2013, Charlotte, NC: Information Age Publishing.Copyright 2012 by ibstpi®, www.ibstpi.org The Job Profile for IDs at university level provides a definition of IDs and includes 54 work activities in five levels categorized in accordance with their types of tasks, levels of difficulty, and scope of responsibility.The Quality Assurance e-Learning Design Standards provide a measure of quality assurance for online courses in order to serve the e-learning needs of students, covering different areas in course design such as: Syllabus, Learning Objectives, Learning Activities and Assessment, Copyright Requirements, and Course Functionality.The institute's Strategic Plan was used to identify organization-level expectation for IDs performance, roles, and responsibilities.This analysis focuses on the Vision & Mission and Design Goals sections of the Strategic Plan, which list the specific descriptions of the organizational goals, elaborate on the foci for ID, and lay out the desirable outcomes that the institute expects to gain from its performance.The institute-specific ID Job Posting specifies the essential requirements for the ideal candidate who is able to contribute to the achievement of the institute's vision and mission.Those requirements include the applicant's academic background, work experience, knowledge, and soft and hard skills.
The documents collected at the organizational level are: the university-wide Job Profile for IDs, Quality Assurance e-Learning Design Standards, the Strategic Plan of the institute, and the institute's Job Postings to recruit IDs.Data at individual level were collected using a questionnaire consisting of 20 open-ended questions and 105 Likert-scale questions.

Table 2 .
Organizational and Individual Level, What Competencies, Skills, Knowledge, andAbilities Are Expected for IDs to Carry out Their Jobs/Tasks Throughout job/task analysis, this study has identified six domains, 13 competencies, and 42 specific performance descriptions at both the organizational and individual levels (see Appendix 2).In terms of key SKAs at the organizational level, the university specifies competencies such as having a Master's degree in education, applying instructional and learning theories (e.g., adult learning, backward design) in practice, knowledge of learning management systems (e.g., Angel, Drupal), and technical skills.In general, the institute agrees with the SKAs identified at university level, but tends to have more specific qualifications, particularly in the areas of technical and interpersonal skills.The qualifications specified by the institute include: technical skills in handling learning management systems (e.g.Angel Drupal); creating HTML5 web pages, managing databases, multimedia skills in creating and editing digital content (e.g.videos, graphics); and communication skills to educate and collaborate with course instructors.Data collected at individual level present self-reported information such as job identities, abilities, and attitudes associated with tasks.Specifically, IDs at the institute defined themselves as liaisons between instructors and students, educators, co-workers of faculty, helpers of faculty, and instructional designers.In addition, IDs identified organization, problem solving, attention to details, multi-tasking, time management, project management, leadership, and team-building as general abilities.Being goal-oriented and having good-listening skills were stated as desired attitudes Finally, competencies for Research and Education were identified as unique competencies throughout the university's IDs Job Profile, Institute's Strategic Plan, Design Goals, and Job Profile.The competency domain of research includes competencies to assist faculty in conducting online learning-related research and establishing best practices in online learning.The competency domain of education emphasizes the competency to develop and provide appropriate opportunities for faculty to learn latest trends, skills, technologies, and information associated with online learning through training, workshops, seminars, and conferences.Institute's IDs Competencies and Domains Compared to ibstpi's Competency Model Note.* are summarized from Instructional Designer Competencies: The Standards (4 th ed)( Koszalka, Russ-Eft,& Reiser, 2012), ** indicates that components related to the competencies are found in the data source.***indicatesthatmeanvalues of competencies are above 4.0 4.3 What Are the Key Domains and Competencies for the Context-Specific IDs Competency Model in This Study?At the individual level, our analysis of IDs' responses to the open-ended survey questions has revealed four domains as important competency domains for IDs at the institute: Professional Foundations, Planning & Analysis, Design & Development, and Project Management.On the other hand, Evaluation & Implementation was rarely mentioned by the IDs.The results from the Likert-scale survey questions show that competencies in the domains of Professional Foundations (Mean = 4.20) and Design and Development (Mean = 4.12) were rated as highly important.On the contrary, certain competencies in ibstpi's model such as conducting needs assessment, implementing interventions, and business skills were perceived as less critical(Mean = 3.67, 3.45 and 3.22, respectively).It is also interesting to note that a few competencies emphasized at the organizational level were not valued at the individual level, such as conducting needs assessment, determining instructional content, and revising instructional and non-instructional solutions (Mean = 3.67, 3.93, and 3.81 respectively).In conclusion, an institute-specific competency model includes a total of seven domains.The domains of Professional Foundations, Design & Development, Education, and Research were emphasized at both the organizational and individual levels, while the domains of Planning & Analysis, Evaluation & Implementation, and Management were emphasized at the organizational level only.Also, Education and Research were competency domains that emerged from our analysis specific to Institute context but not included in ibstpi's model.The refined competency model is presented in Table3.
4.2 Compared to Ibstpi's Ids Competency Model, What ID Competencies Are Commonly or Uniquely Found at the Institute?Table2presents the combined findings from the job/task analysis and the Likert-scale survey at both the organizational and individual levels.Overall, the analysis results suggest that the key competencies in ibstpi's model are required at the organizational level, but the university and Institute also prefer certain ID competencies that are not mentioned or emphasized in ibstpi's model, such as initiating or participating in educational research, publishing in the field of ID or online education, educating faculty about optimal design, mentoring junior IDs, and conducting workshops/seminars/training sessions.Also, some of the competencies were added to the domain of ibstpi's model: to abide by copyright requirements in Professional Foundations; to find and suggest solutions for technical issues/learning environment in Planning & Analysis; to do coaching/mentoring, to manage resources, facilities, and learning systems, and to keep up with e-Learning market trends in Management.

Table 3 .
The refined competency model for instructional designers in online higher education Secondly, it would be helpful to choose the appropriate means of interventions, either training or non-training, for competency development.For competencies of Planning & Analysis and Evaluation & Implementation, which are not highly valued at the individual level but considered significant at the organizational level, the institute can offer organized training-related interventions to educate IDs about the importance of competencies.