Development of Physical Therapy Practical Assessment System by Using Multisource Feedback

The purposes of the research were (1) to develop the physical therapy practical assessment system by using the multisource feedback (MSF) approach and (2) to investigate the effectiveness of the implementation of the developed physical therapy practical assessment system. The development of physical therapy practical assessment system by using MSF was determined by nine experts in physical therapy. Suitability and feasibility of the system for each sub-group were investigated. Five input factors, two process factors, two output factors, and two feedback factors were involved in the system. Level of suitability and feasibility of elements in each sub-group presented at high to the highest level. In system testing, 40 physical therapy students were participated. Raters consisted of clinical educators, students (self-assessment), friends (students who in the same practical group), and patients. Twice assessments during the period of clinical practice were performed. Data analysis for generalizability coefficient (G-coefficient) was evaluated by the EduG program. Quality of the system was evaluated 4 aspects including the utility, feasibility, propriety, and accuracy. These were calculated by mean ( ̅) and standard deviation (SD). The values of G-coefficient for absolute and relative were 0.86 and 0.88, respectively. In addition, quality of the system showed value at high to the highest level in all aspects.


Introduction
Teaching and learning in higher education, there are many processes to facilitating knowledge, skills, and behaviors.Clinical practice is one of the most important processes to develop the students to be professional.This is the highest level of competency that students must achieve prior to graduation.Students are required to integrate and apply their knowledge and practical skill in classroom to real-life situations.This is very challenging for clinical educators to arrange the suitable situation and adapt or develop methods to motivate students to achieve the professional competence.In addition, the clinical practice management should consider several elements enriched in encourage student's experience effectively, such as being a good role model of the clinical educator, teaching preparation, facilitate reflective learning, adequate learning resources, assessment system that consistent with the objective, and giving and receiving good feedback.Especially, feedback in clinical education should be specific which had provided information for narrowing the gap between actual and expected performances.The purpose of all is to promote students to success in clinical practice (Archer, 2010;Cunningham, Baird, & Wright, 2015;Ernstzen, Bitzer, & Grimmer-Somers, 2009;Janicik & Fletcher, 2003;Jette, Nelson, Palalma, & Wetherbee, 2014;Ramani & Leinster, 2008).Physical therapy is one of the health professional team that provides health care for all people.Teaching in clinical practice are different in classroom learning because it occurs in real situation.Students are divided into small group of 3-5 persons and entrusted to the care of patients under clinical educator responsibility.
However, not everyone can achieve the requirement by clinical practice.So, it is necessary to give students an opportunity to improve and develop their practical skill.In order to get the appropriate information for improving of student's performance, the appropriate assessment method is required.Moreover, quality of clinical practical assessment is the most importance part which has to be concerned.Because clinical practical assessment is a key to succeed in health professions in the future, the formative assessment is benefit to help students to get valuable feedback and use this information in planning and developing their own practice in time.
Moreover, formative assessment can identify students learning to result in academic achievement in the desired direction (Andrade & Cizek, 2010;Black & Wiliam, 2009;Kahl, Hofman, & Bryant, 2013).In addition, these can be sharing attitude of teachers and students during teaching.But the most recent assessment in clinical practice is often used to judge performance of student's practice (summative assessment).Formative assessment was only used in term of verbal or informal that can lead to misunderstand between clinical educators and students.These issues can be found in studies that reported perception of clinical educator providing feedback.The research suggested that the understanding of clinical educators were always providing feedback to their student frequently.On the other hand, students reported that the receiving feedback from clinical educators was rare.It might be due to an assessment was used that not enough to students perceive and understand (Archer, 2010;Liberman, Liberman, Steinert, McLeod, & Meterissian, 2005;Van de Ridder, Stokking, McGaghie, & Cate, 2008).
Multisource feedback (MSF) is one of the assessment tools for collection of detailed information from multiple sources which called the assessors.The MSF can be used to improve performance of an examinee in the clinical practice of medical education.The MSF can be applied in two manner including formative and summative assessments, but mostly used in the form of formative assessment.The application of MSF was usually used to assessment and report informal.The process of MSF will get valuable information of feedback which impacted to student's performance directly.Therefore, there were several studies suggested that MSF was a tool for improving student's competences such as communication, teamwork, patient management, and professional development (Bracken, Timmreck, & Church, 2001;Davies, Ciantar, Jubraj, & Bates, 2013;Violato, Worsfold, & Polgar, 2009).There are many elements involved when using MSF such as design and select instrument, select raters, collect data, analyze data, report feedback etc.These assessment can help students to know their strength and weak performances, similarly, the teachers can use the result to plan the teaching and learning for students.
Furthermore, MSF will be received the result of assessment from the assessors more than one person.The assessors will consist of individuals who have relationships with the student's practice e.g.clinical educator, peer, and patient.This is in line with the authentic situation that there are many people involving student's practical experience.The different views of assessors will provide important information on the development of student's performance.For examples, clinical educator views can identify strength and weak performance.Peers views who practice in same group that can explore behavior practice routine.In addition, patient views can give another valuable feedback because patients are an importance stakeholders in the future.This assessment will assist the students to ready for work and also serve as reflection of the curriculum quality.Moreover, MSF will promote the student to be one of the assessors (self-assessment).This is very useful for the students to assess themselves more precise and encourage student lifelong learning (Cox & Irby, 2007;Davies & Archer, 2005;Davies, Ciantar, Jubraj, & Bates, 2013;Overeem et al., 2010;Reinders et al., 2011;Wall, Singh, Whitehouse, Hassell, & Howes, 2012).However, there was no study about the application of MSF in formative manner for improving student performance in the clinical practice for physical therapy education.Therefore, as the first step, researchers interested in developing the physical therapy practical assessment system by using MSF and investigate the effectiveness of the developed system in order to improve quality of clinical education.The purpose of the research was (1) to develop physical therapy practical assessment by using MSF and (2) to investigate the quality of the implementation of the developed physical therapy practical assessment system.

Method
There were 2 steps of developing the system as described in the following; Step 1: Developing of the physical therapy practical assessment system by using MSF; the assessment system was developed by a synthesis of documents related to the assessment in clinical practice and MSF approach.Then, investigation of suitability and feasibility of the developing system were performed by nine experts.
Step 2: Experiment of the developed physical therapy practical assessment system; after completed developing, the quality of the assessment system was evaluated.The participants in this step were mentioned as examinee.There were consisted of 40 physical therapy students of the Faculty of Physical Therapy, Mahidol University.The raters consisted of clinical educators, students (self-assessment), peers (students who in the same practical group), and patients.The instrument used for assessing students practice was a checklist format questionnaire.The topics of assessment were competence requirements in physical therapy profession, such as professional behavior, communication, and patient management.Validity and reliability of the questionnaire were tested and found to be high (0.87 for validity and 0.91 for reliability).
Data analysis for generalization was using EduG program.In addition, quality of the system was evaluated in 4 ies.ccsenet.aspects fol standard d

Results
The   Scott (2008).Group of elements had inter-related to function to making goals success.This study found that the system consisted of five input elements, two process elements, two output elements, and two feedback elements.The system suitability and feasibility evaluation by experts indicated that each sub-group had values in high to the highest levels.These indicated that the essential elements had been prepared when using this system.The input and process groups of system were support by studies as follows.
The study of Mizikaci (2006) reported that the essential elements for quality in higher education were clarify of the aims and objectives of assessment.Research by Donabedian (1966) cited in Jette, Nelson, Palalma, and Wetherbee (2014) indicated that duration of experiences, evaluation of student's performance, and model of supervisory were considered in quality of clinical education.Besides, the study by Calman, Watson, Norman, Redfern, and Murrells (2002) presented that the assessing practice should be prepared of assessors and assessment method.In addition, student academic achievement always put in the output of evaluation in educational system.As the study by Jette, Nelson, Palalma, and Wetherbee (2014), which indicated the essential outcomes, should be concerned in student's competencies and stakeholder satisfaction.It supports that the output of the system has appropriate.
The results of G-coefficient for absolute and relative were 0.86 and 0.88, respectively.As the description by Shavelson and Webb (1991) explained that G-coefficient was used to interpret of reliability coefficient across various facets of the study.The results of the study indicated this system had reliability in high level.This was supported by Cunningham, Baird, and Wright (2015) that mentioned the assessment in clinical education has to adequate reliability.
In addition, G-coefficient for absolute and relative in D-study which provided information for make decision in the most stable and efficient measurement procedures (Shavelson & Webb, 1991).The results of D-study presented that generalization for absolute (ρ 2 Abs ) were 0.7566, 0.8614, 0.9031, 0.9256, and 0.9491, respectively.The generalization for relative (ρ 2 Rel ) was 0.7919, 0.8839, 0.9195, 0.9384, and 0.9501, respectively.So, the appropriate implementation of this system should be used in assessment about two or three times because small different in G-coefficient was found since the third assessment.So, more than 3 times of assessment is not necessary.This is supported by Jette, Nelson, Palalma, and Wetherbee (2014), which presented the frequent of assessment, should be at least two times which usually performed at mid-point and final week of clinical practice periods.Excessive assessments may be waste in time and resource and resulted in unexpected outcomes.These could be found in previous studies that mentioned about negative impacts to the students.For example, it can make stress and uncomfortable that lead to disturb student's learning in clinical experiences (Changiz, Malekpour, & Zargham-Boroujeni, 2012;Razaee, Esmaeili, & Habibzadeh, 2015).
According to the joint committee on standards for educational evaluation (1994) recommended that the system effectiveness should be evaluated standards of utility, feasibility, propriety, and accuracy.This study found that the effectiveness of the developing of physical therapy practical assessment system by using MSF were at high to the highest levels of all aspects.Therefore, this system had adequate quality for implementing future physical therapy practical assessment.
This study was included participants only in the Faculty of Physical Therapy, Mahidol University.Because there were differences in pattern of clinical practice and evaluation in each physical therapy institutes in Thailand, future study will be challenge to investigate this system used in other institutes or other health professional educations that required clinical practice.