Training Improvement through Subjective Work Analysis: The Example of Radial Puncture

The aim was to test a method developed in nuclear industry applied to a simple activity, the radial artery puncture, and to assess its capacity to improve performance. The method involved digital ethnography based on the Square of PErceived ACtion model applied in real operating situation and first-person perspective video for post-analysis of the activity in order to improve training design. Two types of training sessions were compared in terms of trainees’ performance, one of them taking benefits of the digital ethnography analysis (restructured session) and the other without (classic session): medical students trained in the anesthesiology department of the university (N=24) were summoned for training in the framework of their university curriculum. Data obtained were used to restructure the training session and to elaborate an evaluation grid to assess performance in both sessions. Trainees’ motivation was assessed through the Motivated Strategies for Learning Questionnaire. The restructured session showed significantly higher overall performance (increased by 13%), improvement of every criterion assessed and no alteration of motivation. The improvement obtained for radial puncture matches this observed in nuclear industry. The improvement is two folds: at the level of the training efficiency and at the level of trainees’ performance.

of the subjects' activity in real operating situations. This is followed by an interview based on the Square of PErceived ACtion model or SPEAC model (Fauquet-Alekhine, 2016a, 2020Fauquet-Alekhine & Lahlou, 2017) involving workers in a reflexive analysis of their activity. The so-called "SPEAC method" provides a refined description of training needs (matrix of competencies) necessary to perform the activity and is used to develop teaching and training. This method has shown its capacity to identify more fields of competencies (explicit and especially tacit) for the studied activity when compared to other methods (Fauquet-Alekhine, 2016a, 2020. In that it was considered as more incisive than the descriptive method applied in the present case.
The activity chosen for the test was radial artery puncture (to provide a sample of blood for the arterial blood gas test for example), identified as needing improvement (as observed elsewhere: Tavakol et al., 2012;Hernández-Padilla et al., 2016a, 2016b and presenting the advantage of being simple and short, limiting the cost of the experiments.
The aim of the study was to assess to which extent the SPEAC method would improve simulation training efficiency and thus trainees' performance for radial puncture. Such improvements would contribute to reduce patients' pain, complications and related costs. This was undertaken in the perspective of applying the method to more complex activities with the same objectives.

Design
The research was led by a Behavioral Psychologist of the SEBE-Lab/Dept. of Psychological & Behavioural Sc. of the LSE (UK) and the head manager of the Health Simulation Training Center in 2017, and was conducted as follows: 1) An experienced anesthesiologist performed radial puncture in real operating situation and the activity was analyzed using the SPEAC method.
2) A matrix of competencies (list of knowledge and know-how necessary to perform radial puncture) was elaborated on the basis of the analysis of the activity, used to define the training needs and to design an evaluation grid for job performance assessment as done elsewhere (e.g. Hernández-Padilla et al., 2016b).
3) The training session was restructured based on the matrix of competencies. For easier understanding, the training session existing before conducting the research is called "classic session" in the following; the training session obtained after using results of the matrix of competencies is called "restructured session", 4) Classic session and restructured session were then compared in terms of performance with two groups of trainees.

Analysis of the Work Activity
Analysis of the radial puncture in real operating situation involved a volunteer anesthesiologist.
The SPEAC method applied for work activity analysis was developed, assessed and validated for workers in nuclear power plants (Fauquet-Alekhine, 2016a, 2020Fauquet-Alekhine and Lahlou, 2017). It is based on the SPEAC model presenting four interacting poles describing competencies in action ( Figure 1). It supports the fact that four poles are required for competencies to be put successfully in action: Having to act, Knowing to act, Wanting to act and Being able to act.
Having to act is a result of the order (manager, patient) and the pre-definition of the task. Knowing to act is that the worker will know to implement whilst performing the task; this pole is a result of learning, of professional training and experience. As it refers to knowledge and know-how, the wording used for the pole is Knowing to act. Should it have only referred to know-how and experience, the wording would have been Knowing how to act. Wanting to act refers to the workers's personal commitment and motivation. Being able to act reflects logistical resources and material means, social conditions and work organization and also the subjects' capacities from the physiological and psychological standpoints. The schema of the SPEAC method was the following: • Before undertaking the activity, the anesthesiologist was informed of the aim of the study so that to sign an informed consent, carried out a risk assessment (Fauquet-Alekhine, 2016b;Fauquet-Alekhine et al., 2018) regarding the use of subcam and was equipped with it (preparation phase, 5 min.).

•
The anesthesiologist performed his activity which was video recorded from the first-person perspective by means of a subcam worn at the eye-level (capture phase, about 20 min).

•
After the activity, the subjective video was pre-analyzed by the Behavioral Psychologist to select video relevant sequences.

•
The anesthesiologist was met individually by the Behavioral Psychologist to undertake the interview (analysis phase) based on self-confrontation (Von Cranach, 1982;Theureau, 2002), explicitation techniques (Vermersch, 2006) and SPEAC model (Fauquet-Alekhine, 2016a, 2020. A post-analysis of the interview (audio recorded) was undertaken by the Behavioral Psychologist; this provided the output data related to competencies (post-analysis phase).

•
Finally, the Behavioral Psychologist and the anesthesiologist discussed resulting output data for validation (validation phase).
The digital equipment used for the first-person recording was made up of three parts linked with cables: i) a micro digital self-powered recorder (size of a mobile phone), ii) a subcam mounted on safety glasses, iii) a lavaliere microphone.

Matrix of Competencies
Knowledge and know-how necessary to perform the activity were identified and described through the post-analysis per fields of competencies (the matrix of competencies), one field per column.
For each column of the matrix (Figure 2), knowledge and know-how were listed, regarding both the individual and collective aspects. Knowledge and know-how supposed to be tacit were highlighted in yellow. The first line addressed the fields of competencies designating a general know-how. They were written starting with "knowing how to" followed by an action verb at the infinitive form. So as to simplify the matrix, "knowing how to" was omitted. The individual or collective knowledge referring to information (lessons, books, demonstration…) kept in memory by the anesthesiologist was written using the gerundive form of "to know" followed by direct object. As knowledge fosters know-how (Fauquet-Alekhine, 2020), knowledge was thus to be followed by a goal of action expressed by "in order to know how to" and an action verb at the infinitive form, expressing know-how. However, "in order to know how to" was omitted so as to lighten the matrix. These items were identified during the analysis phase, the validation phase and the post-analysis phase. An excerpt of the matrix is provided on Figure 2. Note. Highlighted in yellow = tacit knowledge and know-how.

The Training Sessions
The sequences of the classic and restructured training sessions (two conditions of the experiment) were identical and carried out with the same simulators ( Figure 3): introduction, theoretical lecture, simulation training, simulation debriefing (Fanning and Gabba, 2007;Fauquet-Alekhine and Boucherand, 2016), assessment in practice, session debriefing.
For the restructured session, only the content, the structure of this content and the way it was discussed were adjusted.
The whole sessions lasted 90 min.

Classic Session and Restructured Session Comparison
Trainees were summoned to attend the training sessions. It was a blind experiment with two conditions: the classic session and the restructured session. The dependent variable was the trainees' performance, assessed using the evaluation grid. Two independent variables were controlled because they were fixed by the choice of trainees' profile: academic background (medical students in fourth year) and age. The other independent variables were gender, past simulation training, past professional experience, and type of simulation training (classic or restructured). Gender was partly controlled by making an equal proportion in samples of trainees. The effect of past simulation training and past professional experience was assumed to be not significant in first approach (trainees never trained for radial puncture), subject to confirmation through appropriate statistical tests of experimental data. The influence of the two values of the independent variable "type of simulation training" was analyzed. Trainees' performance assessment (dependent variable) was undertaken at the end of each training session.
The training team was a physician-trainer and two Referent residents.

Sample/Participants
Regarding the analysis of a radial puncture performed in real operating situation, an experienced anesthesiologist (male, 41 years old and 13 years of experience in the position) was volunteer.
Regarding the two-condition experiment, trainees were volunteer medical students in fourth year trained in anesthesiology department of the university in view to adopt Anesthesiology specialty (N=24; Table 1), summoned for training in the framework of their university curriculum.

Data Collection
On the basis of the matrix of competencies, an evaluation grid was elaborated, taking into account specifications of the simulator, the duration of the training session and the trainees' profile (novices). Evaluation criteria concerned clinical examination, technical skills during procedure application, compliance to procedure and communication with patient, assessed on a binary scale as done elsewhere (e.g. Tang et al., 2006). The items were weighted by coefficients (from 1 to 3) according to their importance for the activity; the maximum possible score was 68. The performance of each session was given by the averaged trainees' score.
The validity of the evaluation grid was assessed individually by three experts as done by others (Bausch et al., 2013) on the basis of three statements applied to each item of the grid and evaluated on a five-level Likert scale (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree): • Relevancy: this item is relevant to assess performance.
• Observability: this item is observable whilst trainee performs the activity.
• Binary assessment feasibility: this item is easy to assess on a binary scale (done, not done).
Job performance assessment of trainees was performed by a physician-trainer of the Medical Training Centre and the Behavioral Psychologist viewing the video recording of their performance.
Immediately after each session, trainees were asked to fill in a form for socio-demographic data and motivation assessment.
The motivation assessment was made up of three scales of the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich et al., 1991): the Extrinsic Goal Orientation scale (degree to which participants perceived themselves engaged in a task due to grades, rewards, performance, evaluation by others, and competition), Task Value scale (participants' evaluation of how interesting, important and useful the talk was), the Self-Efficacy for Learning and Performance scale (expectancy for success) and self-efficacy. These scales were chosen among the six motivation scales because of their relevance regarding the experiment; the remaining scales dealing with long term academic courses or anxiety were not selected as non-relevant. Two recent in-depth analyses confirmed the reliability of the MSLQ (Kivinen, 2003;Taylor, 2012).

Ethical Considerations
Informed consents were obtained from subjects after informing them of the general purpose of the study.
This study received ethical approval (code of approval: DSP/RS/PFA-4) of the Ethics Committee of the Dept. of Social Psychology (LSE, London, UK) and has therefore been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

Data Analysis
Performance scores were calculated per trainee and averaged per cohort. Improvement from one training condition to the other was examined through a t-test of Student. The consistency of the experts' assessment of the evaluation grid and the consistency of the cohorts' answers were evaluated through Cronbach coefficients. Sample size estimation N est for a cohort was calculated in the aim of comparing two mean performances. For a confidence interval of 0.05, a statistical power of 0.80, a difference between mean performances expected at 3% of the maximum score (i.e. 2) and a standard deviation expected at 5, we found N est =10.80. Each group was sized at N=12. The influence of the independent variable "past simulation experience" and "past professional experience" was assessed through Pearson correlation coefficient, a χ 2 -test and a Kolomogorov-Smirnov test.

Subjects
The final number of participants was N=23: one of the restructured session subjects was rejected due to a lack of commitment observed by trainers and researchers, objectified by MSLQ data and resulting in the lowest performance score of the cohort.

Work Activity Analysis
The analysis of the work activity led to enhance five points during training and provided a 28-item evaluation grid used to assess the trainees' performance (see Appendix).

Motivation
The motivation scales being assessed on a Likert scale from 1 to 7, we found that all subjects presented a score for each selected motivation scale higher than the middle value 3.5 (Table 2) except for the rejected subject. The Cronbach coefficient was α=0.69 for the first sample of subjects and 0.80 for the second denoting a good consistency of the answers. The MSLQ scores (individual and averaged per sample) showed an effective commitment of the subjects. No significant correlation was identified between performance and motivation scales. No significant influence on motivation was identified neither from gender nor from age. Table 3 gives the comparative performance results with scores averaged per cohort (maximum possible score: 68). Results showed an acceptable consistency for subjects' performance with a Cronbach coefficient α=0.72.

Performance
The variable "past simulation experience" had no influence on performance as it was zero for all trainees (no previous simulation training). Regarding the variable "past professional experience", the distribution of the number of times the trainees dealt with radial puncture as observers in real operating situation was drawn for each cohort. It was varying from 0 to 6 for trainees of the classic session and 0 to 7 for the other. Kolmogorov-Smirnov test and χ 2 -test showed that distributions were similar: resp. D(N=5)=0.15 and χ 2 (df=3)=0.58. The correlation coefficients between the variables "past professional experience" and "performance" were not significant (r(df=21)<0.27), validating the assumption of absence of influence made in first approach.
The correlation coefficients between "performance" and the other variables (age, gender) were not significant (r(df=21)<0.24) A significant improvement in the performance of the restructured session was found when compared with the classic session: • average score increased: +13,7% (t(df=21)=3.98; p<0.001), • standard deviation was divided by 2, • trainees whose score was over 90% of the maximum score were 5 times as many.
Furthermore, two points are of interest: i) the items achieved with 100% success in the classic session were achieved at the same level in the restructured session, ii) the score of the other items were higher in the restructured session.

Performance
To summarize the results from the classic session to the restructured session for this pilot study: • Significant improvement in the performance of the restructured session was obtained: increased score by more than 13% (p<0.001), • Same motivation score was observed on MSLQ, • Average performance score was increased for the restructured session with a standard deviation divided by 2 and a percentage of trainees whose score was over 90% of the maximum score was increased by a factor of 5.

•
Knowledge and know-how that need improvement were clearly identified through the evaluation grid. For example, the SPEAC method pointed out that patient's consent and identity had to be enhanced during training: these points were improved; furthermore, some points not emphasized were improved too: striking successfully at once and the fluidity of gestures. The restructured session was thus more efficient overall than the classic session.
Regarding the contribution of the work analysis, the SPEAC method undoubtedly enriched the content and the structure of the training session. It helped to describe competencies in action as an interacting system of four poles (Figure 1) required for competencies to be put successfully in action: Having to act, Knowing to act, Wanting to act and Being able to act. Therefore, the SPEAC method contributed to give a refined description of the knowledge and know-how than that of the descriptive method. This finding confirmed previous results (Fauquet-Alekhine, 2016a, 2020; Fauquet-Alekhine & Lahlou, 2017).

Limitation
Results show that however a few points remain to be improved. Some of these residual issues might be corrected in future training sessions using a full-scale simulator with an actor as patient; this might contribute to enhancing the relationship physician-patient and would contribute to force the improvement of the physician-patient exchanges including identification and consent.
Another limitation addresses the evaluation levels of the training session. This limitation is discussed on the basis of the Kirkpatrick four-level pattern (Kirkpatrick & Kirkpatrick, 1994, 2005: it is commonly applied to evaluate training programs at different stages (e.g. Bell & Marais, 2015): trainees' satisfaction, learning, behavior and results (performance in real operating situation). In the present study, the first level was assessed qualitatively through the debriefing undertaken at the end of each training session. The second level was assessed through job performance assessment using the 28-item grid. The third and fourth levels were partly assessed. While behavior and performance were assessed on simulator, it was not evaluated in real operating situation. This was not possible because trainees were from different parts of the region; therefore, the assessment in real operating situation would have implied a lot of trips on the road for a medical expert in charge of the assessments. Even if assigned to local experts, there was a high probability that they would not be available at the moment the radial puncture is performed.
Finally, being a pilot study, its results should not be generalized without further considerations.

Conclusion
The aim of the study was to assess to which extent the SPEAC method would improve simulation training efficiency and thus trainees' performance for radial puncture. The results have shown its possibilities to improve training session in terms of trainees' performance (more than 13% for radial puncture) at exit of the simulation: all the criteria assessed were improved when compared to those obtained after using a descriptive method for the design of the training session. The method shows it is a possible solution to deal with medical problems encountered during radial artery puncture in hospital and thus reduce the subsequent economic implications (Wade et al., 2015) when considering the significant quality improvement of the medical act. Application to other medical acts must be considered.