Assessment of Training Effectiveness Adjusted for Learning (ATEAL) Part II: Practical Application

Safety training programs are a popular method, in industry globally, to increase awareness of risks to employees and employers and plays a critical part in reducing safety incidents. The most frequently used method to assess the effectiveness of the training is to have the participants answer Multiple Choice Question (MCQ) and True/False (T/F) questions after the training. The metrics used to report the outcome of the assessments have drawbacks that make it difficult for the trainer and organization to easily identify the concepts that need more focus and those that do not. The goal of this research study is to compare how the methods used to measure training effectiveness of concepts in Level 2 post training assessment differ in how they assess training effectiveness using actual training results. Preand Post-training assessments were administered to the participants in 3 different utility industries and were analyzed for training effectiveness using the traditional metrics as well as using ATEAL method. The results were then compared and detailed recommendations of the best and least learned concepts by industry are presented based on these comparative analyses. The ATEAL method is further used to quantify the opportunities for improvement in the training programs based on the participant prior knowledge and any negative training impact observed. Results of the comparison of the various methods show that the proposed ATEAL method provides a quick, accurate and easy way to assesses the effectiveness of the training of concepts and the method identified that for 40% of the concepts trained a higher percentage of participants exhibited more prior knowledge than positive learning and for 6% of the concepts a higher percentage exhibited negative training. These results also provide a directional guide on the improvements that can be made to improve the training effectiveness of the programs. Additionally, it also shows that the ATEAL method can be used in any learning environment where there is a pre-/post-test evaluation of the change and is not limited in application to MCQ and T/F questions.


Introduction
Workplace training, globally, is an important way for organizations to increase the knowledge of their employees and it has been reported that organizations invest approximately $55.3 billion to $200 billion annually (Salas & Cannon-Bowers, 2001) on employee training. Brunello and Medio (2001) observed that different countries invest differing amounts in employee training based on tenure, and there is an overall approach globally to increase the knowledge of employees in an organization using formal training methods. With this level of fiscal and time investment being made in training it is important to ensure that the training is effective and will result in the expected changes in behavior among the participants.
Of the various topics that employees are trained on, safety training is particularly important due to the impact of poor safety practices (Campbell-Kyureghyan & Cooper, 2012). According to the Bureau of Labor Statistics, the number of fatal work injuries in the US for 2018 was 5,250, an increase of 2% (5,147) from 2017. Similar statistics have been reported by Ho and Dzeng (2010)  ods used to m ts and no easy the best an lly, the current to improve the raining on con ges in safety be mpanion paper methodology y adjusting for he results of t nt model and d ed.   (4), measures the proportion of all the participants who needed to learn the concept (responded incorrectly or IDK in the pre-test assessment) who actually did learn the concept as indicated by their response changing to correct in the post-test.
Positive Training Impact (PTI) = (4) 2.1.5 Negative Training Impact (NTI): the NTI, shown in formula (5), measures the proportion of participants who presumably knew the concept prior to training (answered correctly in the pre-training assessment) who answered incorrectly or IDK in the post-test assessment.
Negative Training Impact (NTI) = (5) 2.1.6 Learning Adjustment Coefficient (LAC): the LAC measures the necessity of the training by comparing the positive impacts of the training (PTI) to the prior knowledge (PK) of the participants, and it is calculated as shown in formula (6).
2.1.7 Net Training Impact Coefficient (NTIC): the NTIC measures the net impact of the training session by comparing the positive impacts of the training (PTI) to the negative impact of training (NTI) of the respondents, and it is calculated as shown in formula (7).
2.1.8 Training Effectiveness Matrix (TEM): The LAC and the NTIC can be summarized in a Training Effectiveness Matrix (TEM) that allows for visual identification of the training effectiveness for a concept/question, as shown in Figure 2. The quadrants of the matrix are described below. post-training assessments contained Multiple Choice Questions (MCQs) and True or False (T/F) items to determine the knowledge of the content for each participant. The pre-training assessment was completed just prior to the training session and collected on completion. The training session typically lasted from 1-3 hours and the same assessment was administered as the post-training assessment. The order of the options and the questions were not rearranged between the pre-test and post-test assessments. The number of MCQ and T/F questions for each of the utility sectors, based on the role of the participants, is summarized in Table 1. In the MCQ assessment, one question, in both the pre-and post-training assessment, was a question contextually similar to the content being trained but was not specifically covered in the training class. This is referred to as the Control Question (CQ) and is used to explain if the participants had prior knowledge of the concept or were guessing in the assessment. Further details of the CQ are provided in Samuel et al., (2019) and Caston, Cooper, and Campbell-Kyureghyan (2009). Additionally, for the pre-and post-training assessments for the Electric Transmission and Power Generation utility sectors an additional "I Don't Know" (IDK) option was added, as indicated in Table 1.
Training content and concepts were based on research that specifically targeted the areas of safety and ergonomics in non-repetitive work environments (Ahmed & Campbell-Kyureghyan, 2014). To define the ergonomic risks onsite visits were conducted, and data gathered from interviews with managers and employees and direct observations using videotaping methods. Due to the differences in the types of utilities and the work performed concepts were changed to best cater to each industry and combined with information from nationwide industry and fatality statistics for utility industries (Campbell-Kyureghyan & Cooper, 2012). Table 2 details the concepts trained and the number of questions in the assessments by concept for the various training groups in each utility sector. Both employees and mid-level management were trained as it has been reported that management's commitment to safety results in lowering injury rates and improving the company safety culture (Demirkesen, 2015).
The assessment metrics were calculated for each of the training groups and are compared and contrasted to identify the metrics that best help determine the performance of the participants and the direction of training improvements required.

Results
The pre-and post-training assessment results for the participants from the various utilities are calculated using the TPC, PPPC, and ATEAL measures to help identify the concepts which were best learned, the concepts for which the participants had the most prior knowledge, and the concepts for which the participants experienced higher negative impact. Additionally, the responses of the participants on the Control Question and its representation by the various metrics is examined. Ideally, in all cases, we would expect the CQs to be at (0.5, 0) in the TEM when using the ATEAL method, and zero when using the PPPC or the TPC as this would indicate jel.ccsenet.o zero learni than zero a the results on the other concepts that needed to be learned. Finally, the ATEAL method does an excellent job in identifying the CQ (numbered 1 in Figure 3) among the concepts taught. As indicated previously, the CQ is a concept that was not taught in the training, but was thematically similar to the rest of the content tested, and was used to estimate the amount of guessing by the participants. The results show that there was more negative training impact than positive training on the CQ and that the participants were having difficulty answering the question. This is the only question for which the NTIC is less than zero. By having the CQ and using it along with the other assessment results, we can clearly see that the ATEAL method helps provide considerably higher resolution in understanding the effectiveness of the training of each concept compared to the PPPC metric. Table 4 illustrates the training performance metrics calculated for the Tier 2 Employee training group in the Natural Gas Utility sector. A total of 347 participants answered each question/concept in this training group. Similar to the Tier 1 Employee group, the TPC metric does not indicate that Hearing Loss is the best learned concept as it includes the prior knowledge in the final assessment results reported. However, the PPPC identifies Hearing Loss as the best learned concept. In applying the ATEAL method, the Training Effectiveness Matrix for these 9 concepts, shown in Figure 4, clearly identifies Hearing Loss (numbered 5 in Figure 4) as the best-learned concept. The rest of the concepts have very similar results to those observed with the Tier 1 Employee training group, with the participants having higher prior knowledge for the Environment and Vibration concepts (numbered 3 & 9 in Figure 4). The CQ (numbered 1 in Figure 4) for Tier 2 trainees lands in Quad 1, whereas for the Tier 1 training group it was in Quad 3. This indicates that there was more positive learning on the CQ than both prior knowledge and negative training. However, its magnitude is very low (close to 0.5, 0) indicating that the net learning was almost zero. This could be explained by the fact that more participants in the Tier 2 Employee group guessed correctly on the CQ compared to the Tier 1 Employee group. jel.ccsenet.o Table 5 ill Utility sec Tier 1 and concept by however, t  Table 6 il group in t assessmen measure tr the best le large marg   Table 9 ill Generation the TPC to best learne by/caught Natural Gas Utility and that its content and method of delivery was highly effective due to its positive impact with such a large number of participants. However, because of the high level of prior knowledge, there should have only been a cursory overview of this concept and an argument can be made that it did not need to be tested in the post-test assessment.

Electri
The results of the PPPC in the scenario and simulation analysis in the companion paper show that it is better at compensating for prior knowledge than TPC. This benefit is further observed when looking at the results of actual training and assessments conducted on the participants from the various utility companies. For the Tier 1 Employees in the Natural Gas Utility, the PPPC identifies the concept of 'Hearing Loss' to be the best learned concept and 'Vibration' to the worst learned concept. In looking at the actual performance of the participants for 'Vibration' (CC = 80%; IC = 7%; CI = 9%) we observe that its negative PPPC value is due to the high prior knowledge among the participants and the small number of participants who experienced negative learning. The PPPC metric also does not isolate the CQ and, although it reports a low performance of the participants for the CQ, it is in line with the results for 'Vehicle Safety' which had a low score due to very high prior knowledge. The same trends for the concepts are observed for Natural Gas Utility Tier 2 Employees. For the Managers, the concept of 'Hearing Loss' is identified as the best learned concept and the CQ receives a negative score.

Electric Transmission Utility
Using the ATEAL method to analyze the data for the Electric Transmission Utility, for Tier 1 Employees there was positive learning on six of the nine concepts taught, with the 'General' concept being the best learned because the participants had the least prior knowledge and comparatively learned the most on this concept. For the Electric Transmission Utility Tier 2 Employees, the concept of 'Hearing Loss' was the best learned concept and the participants exhibited positive learning on four of the ten concepts tested. The CQ, as seen before, exhibited low learning and, although it is in Quad 1, it is the closest of all the concepts taught to (0.5, 0). When using the TPC to analyze the data in the Electric Transmission Utility, the concept of 'Vehicle Safety' again seems to be the best learned concept by the Tier 1 & 2 Employees due to the high level of prior knowledge (over 85%) among the participants. In using the PPPC to analyze the data of the Tier 1 Employees, the 'General' concept is identified as the best learned concept and 'Vehicle Safety' as the least learned concept. This is the exact opposite of the results from the TPC metric, and is a more accurate representation of participant knowledge levels as the participants had the highest amount of prior knowledge for 'Vehicle Safety'. Similarly, for the Tier 2 Employees in the Electric Transmission Utility, the 'General' concept is identified as the best learned concept. Due to high prior knowledge and a small number of participants experiencing negative learning, the metric identifies 'Vehicle Safety' and 'Employee Rights & Responsibilities' as the worst learned concept.

Power Generation Utility
Using the ATEAL method, we observe that there was positive learning on eight of the eleven concepts on which Managers were tested. It is extremely interesting to observe that the CQ was the best learned concept, as over 50% of the participants went from incorrect and IDK responses to the CQ in the pre-test assessment to correct responses in the post-test assessment. This could imply that the concept was inadvertently trained in the class by the trainer or that the participants were able to correctly guess the post-test answer. We observe that the concept of Root Cause Analysis had considerable negative training impact and very low prior knowledge. This is a critical issue as this concept is key for the Managers to diagnose safety issues correctly and implement countermeasures to improve the safety of the employees. In further researching the results, we observe that 58% of the participants exhibited zero learning; thus, it is important for the trainers to revisit this concept with this group to ensure that they understand and learn the concepts. It is not possible to quickly arrive at this conclusion when solely looking at TPC and PPPC metrics. Hence, this shows that using the ATEAL method is better and quicker at helping discern participant learning and helps trainers determine countermeasures in an expeditious manner. For the Power Generation Utility Employees, we observe that the CQ lies in Quad 3 and we observe that, for all the other concepts taught, the participants exhibited considerably higher prior knowledge than learning. This is concerning as it shows that a majority of the participants did not learn anything new and the effective use of their time comes into question.
Using the TPC the concept of 'Environment' is shown to be the best learned concept for both the Employees and the Managers due to high prior knowledge (over 84%). The Managers of the Power Generation Utility are also shown to have high learning for the concept of 'Confined Space' as reported by this metric. For this concept there was considerably less prior knowledge (66%) and 33% of the Managers learned the concept. In using the PPPC to analyze the results for the Managers in the Power Generation Utility, we observe that the CQ is reported as the best learned concept. Although this is counterintuitive, the results are due to the 0% prior knowledge and 50% of the participants who answered correctly in the post-test assessment. The other concepts ranked lower mainly due to the fact that participants had higher prior knowledge. Finally, for the Employees of the Power General Utility, the concept of 'Confined Space' is reported to be the best learned concept, although 48% of the participants had prior knowledge of this concept, and 'Environment' is the least learned concept due to 84% of the participants having prior knowledge of this concept.
A common observation through the results and discussion across all of the utilities is the level of prior knowledge that the participants possess for the various concepts trained. Using the ATEAL method we can clearly determine when there are more participants exhibiting prior knowledge than learning. This is impossible to determine when using the TPC metric as it does not compensate for prior knowledge and reports it as learning.
Using the PPPC, it takes more time to discern if the low (or) negative values are due to high prior knowledge or negative learning. The metric does not separate the elements, so it requires additional detailed review of the raw score that takes time and effort and may not always be conducted.
The limitation of ATEAL is that the method requires the presence of matched pre-and post-training assessment results, as the analysis is based on baseline knowledge and learning and cannot be used when there are only post-training assessment results. The application of the method may also require some basic training for trainers and organizations. This training, however, is minimal, as the calculations are simple and graphics are easily implemented by using widely available software packages such as MS Excel.
One of the generalizable benefits of the ATEAL method is that it can be used for any type of assessment situation where there is a pre-and post-test assessment. For example, suppose assembly workers were being trained to improve assembly practices, and the assessment was made by an assessor observing the assembler for performance in the categories of quality, speed, efficiency, following standard work, etc. If the assessment is made on the assembler prior to training, and a score obtained for the various categories, the training conducted and the assembler can then be reassessed on their performance post training and the ATEAL method can be used to measure the training effectiveness in this scenario. Thus, the method is more widely applicable than in just the case of MCQ assessments. This may have remarkable implications for the organizations and the participants as the training time can be reduced and the effectiveness improved simultaneously. Additionally, reduction in training time may have fiscal impacts that result in a higher return on investment (ROI) for the training with a higher focus on concepts for which the participants genuinely have knowledge gaps.

Conclusion/Future Direction
Metrics to quantify the amount of learning that training participants exhibit for a particular training course, or concepts within the course, are critical to understanding the effectiveness of the training, specifically in the context of workplace safety-related concepts. Using the ATEAL method to measure training effectiveness for training conducted with 1,466 participants from a variety of utility industries, and comparing the results to traditional measurement metrics, we observe that the ATEAL method proves very effective in quickly identifying the learning gaps that the participants experienced and in giving direction on the countermeasures that should be taken for each concept trained.
Some recommendations that can be derived from this study are: • Using only the TPC in the post-test assessment to evaluate training effectiveness (or) how much the participants learned is shown to be a highly inaccurate method and does not give clear guidance on areas of improvement.
• The PPPC is shown to be a better metric than the TPC to evaluate training effectiveness; however, it lacks the ability to quickly provide information on the changes needed in the training content or its delivery to improve training effectiveness.
• The ATEAL method uses metrics that are of greater accuracy, are easy to calculate, and provide intuitive output that allows for easy visualization of the training effectiveness results. It provides a great way to illustrate the training effectiveness of each concept taught to the participants and can be used to quickly determine the countermeasures that need to be taken by the trainer with regards to content delivery or development as part of the training program. This then provides information on how to improve training effectiveness in future training sessions on the topic. Organizations can also benefit considerably from this method as it helps them understand the concepts that the participants can be held accountable for as well as the specific concepts that need further reinforcement to ensure the employees have safe work practices in their work environment.
which the participants had considerable prior knowledge. This enables them to focus on concepts for which the participants truly have knowledge gaps and ensure the best return of investment on the training provided and the time used for the training.