Combining Simulation-based Training and Flipped Classroom in Project Management Learning

Every year, countless projects are finished late, go over budget or end up being cancelled, often because their project managers and project teams lack the necessary tools and techniques to support their decision-making. Students of project management courses around the world have difficulty integrating the different knowledge areas of project management, after studying each knowledge area separately. Students then struggle and even fail when it comes to applying these concepts in a real-life project. Simulation-based training contributes to the solution of these problems by linking the concepts learned during a project management course and providing the experience of managing a simulated project that serves as preparation for real life. The objective of this research is to study the impact of simulation-based training and flipped classroom methodology on students learning project management. The contribution of this research is twofold. First, from a theoretical perspective, simulation-based training and flipped classroom methodology literature is enriched and broadened by applying both teaching tools. Second, from a practical perspective, an improvement in results, satisfaction and lessons learned was found when using simulation-based training under flipped classroom methodology compared to simulation-based training in a traditional classroom.


Simulation-based Training
With the use of simulation-based training, students can take risks without any unwanted or unfortunate consequences they could face in the real world managing a real project.Salas, Wildman and Piccolo (2009) lists the advantages of simulation-based training for management learning as follows: Simulation-based training is superior to other training strategies for imparting complex applied competencies and can lead to learning in a reduced timeframe.Simulation-based training provides a more complex and realistic learning environment than other training strategies and more rapidly allows for reality to be simplified and manageable.It also provides a (relatively) risk-free environment for learning and experimentation.This is an ideal method for training infrequently engaged but critical skills and can be quite affordable, is (usually) simple to learn and operate; is a form of learner-controlled training and is inherently more engaging than other training methods.
In several studies, the successful use of a simulator for teaching project management has been recorded (Davidovitch, Parush, & Shtub, 2006, 2008, 2009;Perez, 2015).Working with a simulator provides the opportunity for reflective observation.Particularly in simulation-based training, the different scenarios and data items in the simulator enable the conceptualization and integration of ideas and models considered throughout the learning process.Moreover, the simulator provides a continuous and dynamic environment for active experimentation and is rapidly establishing itself as a critical factor influencing the transfer of learning (Alessi, 1988).

Flipped Classroom Methodology
Flipped classroom methodology is a type of blended learning, which reverses the traditional educational arrangement by delivering instructional content outside of the classroom.Fulton (2012) listed the following advantages of the flipped classroom methodology: Students move at their own pace while doing "homework" in class, giving teachers a better insight into student difficulties and learning styles.Teachers can more easily customize and update the curriculum and provide it to students 24/7; classroom time can be used more effectively and creatively and teachers using the method report seeing increased levels of student achievement, interest, and engagement.Learning theory supports the new approaches and the use of technology is flexible and appropriate for "21st century learning".
Simulation-based training is recognized as an efficient and effective way of teaching and learning complex and dynamic systems (Jacobson & Wilensky, 2006) while flipped classroom methodology is a successful learning methodology (Gaughan, 2014) which has been researched effectively in undergraduate courses (Findlay-Thompson & Mombourquette, 2014) as well as in graduate courses (Critz & Knight, 2013).However, the combination of these two has not yet been quantitatively tested in a well-designed research.Both methodologies are not completely autonomous-simulation-based training needs a teaching methodology that enhances its effect on the student while the flipped classroom methodology cannot be delivered at random and must be accompanied by technology that enhances its effectiveness through active learning.

The Simulator
The Project Team Builder training simulator was used in this research.Using the Project Team Builder, students can either rewind or fast-forward the development process to see what kind of challenges they may face and how one decision can affect their choices in the future.These challenges force the student to make higher-quality decisions and consider alternative methods for performing the project under uncertainty.To provide a higher level of functional fidelity, the Project Team Builder includes two functionalities that are not available in other project management simulators: (1) the ability to control the level of human resources, and (2) the ability to control the execution of the tasks.
The ability to control the level of human resources refers to the decision to assign or release resources in accordance with the changing demand over the course of the project; the student can manage the number of employees in the project in order to match availability to needs.The ability to control the execution of the tasks refers to the decision to split tasks during execution.A task can begin, be adjourned and continue later.Shtub (2004) outlined these principles of the Project Team Builder: "A simulation approach: the trainer simulates one or more projects.The simulation is controlled by a simple user interface and no knowledge of simulation or simulation languages is required.A case study approach: the trainer is based on simulation of case studies.Each case study is a project or a collection of projects performed under schedule, budget, and resource constraints, in a deterministic and dynamic stochastic environment.A dynamic approach: the case studies built into the trainer may be dynamic in the sense that the situation changes over time.A random effect may be introduced to simulate the uncertainty in the environment, and decisions made by the user cause changes in the state of the system simulated." This simulator developed at the Technion has been proved successful in many educational institutions and companies (Iluz, Moser, & Shtub, 2015).
Project planning in the Project Team Builder tool: In the Project Team Builder, the Gantt view, where tasks and durations are represented in a graph, is based on the most likely time of the planned task.Resources planning is performed by using the Resources Graph, which includes information about the maximum availability of each resource with information about the tasks using this resource.
Project budgeting is supported by the budget report, including the accumulated daily income/cost and the maximum available cash.
Running the simulation: While running the simulation in the Project Team Builder, the planned task time is changed to the actual task time.Due to uncertainty, project duration may differ from the planned during the execution.The student may manually change the task start time in order to comply with project constraints.

Design of the Experiment
Three universities were selected for this experiment, located on three continents, each one with a different instruction language, emphasizing in this way the intercultural approach of the research.
All the courses are project management courses, have the same number of academic hours per week (four hours) and cover fundamental aspects of project management, such as scheduling, budgeting requirements, planning, bottlenecks, resource allocation, cash flow, risks and decision-making.
The course at the Technion is taught to an average of 70 students per semester and is offered to students in their third out of four years of study.The course at the Universidad Tecnológica Nacional, is offered to students in their fourth out of five years of study, is attended by an average of 30 students per year.The course at the University of Nicosia is offered to students in the third year and attracts an average of 15 students each semester.

Flipping the Classroom
Outside the classroom: E-Learning courses (including videos and readings) were developed for the two flipped classroom methodology courses to support the simulation-based training experience the students performed in the classrooms.All the students had lectures throughout the semester and exercises to do at home, when using the flipped classroom methodology for the relevant topics; the students learned the theoretical material on their own before the lectures.
In the classroom: The lecturer reviewed and extended the learning content by means of slides and videos and by using the simulator.Lecturers and students discussed the material in depth, increasing interaction and collaboration between teachers and students.Students got quizzes (multiple-choice tests) at the beginning of some classes, as an "entry ticket" to the active learning environment where problem-solving, requiring higher-order thinking skills, took place (Miri, David, & Uri, 2007).These short quizzes were used primarily as an evaluation tool, but since the students knew they were going to be tested using a quiz, the preparation for the quizzes and the evaluations themselves were also part of the learning process.

Project Simulations
The simulation-based training included predefined projects.For each project, each task has one, two or three different modes and each mode has a different duration of the corresponding task.The predecessors of each task, resources, and costs were predefined parameters.The student was required to finish the project by the due date, maximize profit and value, and minimize waste.In the case of late completion of the project, a penalty was applied; in the case of early completion, a bonus (per period) is added to the project income.The duration and cost of the tasks were deterministic in the first scenario and stochastic in the second scenario.Each task takes an integer number of periods to be completed.The scenarios include four types of data: general information, task information, cash flow information and resource information.The objective function for both scenarios was to maximize the value (benefit in Project Team Builder terminology) and minimize the total cost, within the target period, target cost and cash constraints.
Scenario description: All students used the same Project Team Builder version and ran the same two scenarios.First scenario: A seven-task project under a deterministic environment (i.e., all the decisions were made with 100% certainty).Students had a target period of 12 periods and a single resource to manage.They had one academic hour to run and to complete as many scenarios as possible.
Second scenario: An eight-task project under a stochastic environment (under uncertainty).Students had a target period of 27 periods and three resources to manage.They had two academic hours to complete as many runs as possible.
Simulation procedure: The simulation took place in a computer classroom, in order to control performance and to avoid sharing information among students.During the simulations, students had several tasks: Manage time by planning and meeting a schedule, the resources involved in the project, and making sure that the project is performed at the minimum cost.They have to manage the costs and incomes to control the cash flow and cash position to prevent bankruptcy during the project, as well as manage the value so that the project's results are satisfactory, maximizing the objective function:   .

Experimental Conditions
Under the flipped classroom methodology, all students had access to the e-Learning course outside the classroom and to more advanced lectures inside the classroom; in a traditional classroom, all students had just traditional lectures.In Argentina and Israel, all the material as well as presentations and homework were assigned in Spanish and Hebrew, respectively.In Cyprus, all the material was provided in English since classes at the University of Nicosia are held in English.The simulator and all its functions are written in English.In all the courses, all the students had to fill out an initial questionnaire at the beginning of the experiment with personal details about themselves and a final questionnaire at the end about their experience and satisfaction.For all groups, the two Project Team Builder scenarios were used in the experiment, in consecutive teaching weeks in the middle of the semester (not necessarily consecutive calendar weeks).

Measurements
The results of the simulation were measured using the following performance indices, under the predefined objective functions: Project cost, project value or benefit, project time duration, and project cash.The results of the performance in class and knowledge in project management were measured using the following tests: Quizzes (in the flipped classroom methodology) and Knowledge self-evaluation (in a traditional classroom).
For all the courses, questionnaires were used to collect information about the characteristics of the students.

Pilot
A small pilot with five students was carried out in Israel during March 2016, to plan the logistics, check time-consumption of scenarios and verify the comprehension of the learning material, as well as the difficulty level of each scenario.Feedback reported by the participants about all the simulations was used to improve the real experiment.The pilot proved that an average student could complete several simulations (3 to 5 for the first scenario and 1 to 3 for the second scenario).However, it is possible that some students would fail to complete the simulation.

Hypotheses
Hypothesis 1: Mean results (ratio value/cost) in scenarios under the flipped classroom methodology combined with simulation-based training are higher than mean results (ratio value/cost) in scenarios in a traditional classroom combined with simulation-based training.
Hypothesis 2: There is a correlation between the flipped classroom methodology performance, and simulation-based training results.

Methods and Techniques of Analysis
Data were collected from three different sources: The simulator (the ratio value/cost of the simulations), quizzes and questionnaires.
The analyses were aimed at testing whether the observed differences between the means of data samples were significant, and at exploring the relationships between the observed variables.T-tests and ANOVA was used for analysis.Variability gauge charts allowed us to know more about the learning process of the students and the frequency distribution charts illustrated the characteristics of various outcomes.

Descriptive Statistics
In this section, the characteristics of the four different courses were quantitatively described with respect to participation and absenteeism of the students.
Participation: One hundred and ninety (190) students participated in the experiment: 89 students under the flipped classroom methodology (52 from the Technion and 37 from the Universidad Tecnológica Nacional) and 101 in a traditional classroom (92 from the Technion and 9 from the University of Nicosia).In total, 179 students participated in the first scenario and 170 in the second scenario.
Absenteeism: Students were motivated to maximize value, and minimize waste and project duration, within the given constraints.In other research, empirical evidence (Greene, Miller, Crowson, Duke, & Akey, 2004) suggests that the motivation, persistence and purpose promotes higher-order thinking among the students and therefore increases the chance of success.As such, in Argentina, although not mandatory, students had to participate in the simulations to avoid a midterm; in Israel, the simulations represented 10% of the grade; in Cyprus, they were part of the course.
In case of absenteeism, students were not allowed to complete missed simulations to avoid sharing information.The absenteeism for both courses at the Technion was low while at the Universidad Tecnológica Nacional and the University of Nicosia, the absenteeism was remarkably higher; the total absenteeism was 2.1%.
Seemingly, the motivation factor was part of these differences in the absenteeism level since in both courses at the Technion it was similar but not at the Universidad Tecnológica Nacional and at the University of Nicosia.In these two universities, the same percentage for the first scenario is discerned but a peak in the second scenario, especially in Argentina due to an extraordinary event (the day of the second scenario there was a manifestation on the street of the Universidad Tecnológica Nacional that made it impossible for some students to attend).

Initial Questionnaires
The scale in the questionnaires goes from 1 to 5 and was converted from 20 to 100 in order to use the same scale as the results obtained in the Project Team Builder simulations.
Regarding the teaching methodology at the university, both groups were almost equally satisfied with its effectiveness (68 for flipped classroom methodology and 68.86 for the traditional classroom).The traditional classroom group seems to be more satisfied with the current classrooms at their universities (63.30 vs. 61.88).Nevertheless, the flipped classroom methodology group was more satisfied with the material delivered by professors and teaching assistants (73.18 vs. 70.10).
The question that got the lowest grade was "how easy can you integrate all the concepts learned in each course?"(63.53 for flipped classroom methodology and 57.11 for the traditional classroom) coinciding with the statement that motivated us to do this research ("students of project management courses around the world have difficulty integrating the lessons learned").
The traditional classroom group is more satisfied with the learning methodology classrooms at the university (67.22 vs. 62.12).In parallel, they claim they have a considerably better English comprehension level (80.62 vs. 73.88).
With respect to the software approach, both groups have a similar evaluation about how helpful the tool can be (81.88 for flipped classroom methodology vs. 81.44 for the traditional classroom).The traditional classroom group claims they understand more easily how to use new software in comparison with the flipped classroom methodology group (77.11 vs. 74.12).Students from both groups find it very natural to work alone with a computer (81.16 for flipped classroom methodology vs. 80.94 for the traditional classroom).
Students from both groups prefer to study alone or with a partner, and consider profit and value generated to stakeholders the most important aspects in a project.It is clear that those students who prefer studying alone chose profit as the main aspect in a project but in contrast, those who prefer studying with a partner chose value to stake holders over profit.
In regards to experience in project management, project management software and Project Team Builder software, most of the students from both groups have no experience in project management or project management software (72% of them) and more than 90% of them have never used the Project Team Builder in the past.

Flipped Classroom Course Performance
The performance of the students in the different quizzes at the Technion was notably low: The average grade was 38.17 out of 100.At the Universidad Tecnológica Nacional, the performance was considerably higher than at the Technion.

Results of Simulations
The main hypothesis is that results using the Project Team Builder simulator are better when students are under the flipped classroom methodology than those in a traditional classroom.Two studies (Henderson & Trotta, 2016;Liu et al., 2017) proved that the flipped classroom methodology is a successful learning methodology that has been researched effectively in undergraduate courses as well as in this experiment when combining the flipped classroom methodology with simulation-based training; this is consistent with a recent research from 2017 in nursing education (Kim & Jang, 2017).
According to a t-test, there is a significant difference in the ratio value/cost (p<0.0014) when students are under the flipped classroom methodology for the first scenario: 87.68 vs. 79.20,comparing the Technion flipped classroom methodology vs. the Technion traditional classroom (p<0.0038):87.99 vs. 79.50.Furthermore, we identified very similar results for both the Technion flipped classroom methodology vs. the Universidad Tecnológica Nacional flipped classroom methodology (87.99 vs. 87.18)and the Technion traditional classroom vs. the University of Nicosia traditional classroom (79.50 vs. 76.00) for the first scenario.
Regarding the second scenario, there is no significant difference (p<0.2594)but indeed the mean is higher (76.99 vs. 74.60)and the standard deviation is lower (21.57vs 26.68) for the flipped classroom methodology group.The same phenomenon takes place comparing the Technion flipped classroom methodology to the Technion traditional classroom (81.61 vs. 77.53 and 6.42 vs. 22.26) and the Universidad Tecnológica Nacional flipped classroom methodology to the University of Nicosia traditional classroom (66.96 vs. 37.71 and 35.67 vs. 47.45).
A significant difference (p<0.0001) was found in the completed runs when using the flipped classroom methodology for the first scenario.It is higher also in the second scenario, although it is not significant (p<0.2722).
Regarding those who failed (i.e., the number of students who could not complete even one scenario) no flipped classroom methodology failures were found in the first scenario and more failures in a traditional classroom in the second scenario (11% vs 7%), although not significant.
The phenomenon is distinguishable: Below three completed runs, the traditional classroom group has more students, but there are more flipped classroom methodology students above four completed runs.Besides, not a single student from the traditional classroom group completed seven runs.The ratio, with the exception of those who completed two scenarios in the first scenario, was always higher under the flipped classroom methodology.
Analogically, if the mean represents the value and the completed scenarios the quantity, then there is higher value and more quantity when simulation-based training is combined with the flipped classroom methodology.In conclusion, it was proved that mean results for the flipped classroom methodology combined with simulation-based training were higher than for the traditional classroom combined with simulation-based training.

Correlation between Flipped Classroom Methodology Performance and Simulation-based Training Results
To prove this hypothesis, for the flipped classroom methodology group, flipped classroom methodology performance was compared, by means of quizzes, to the Project Team Builder results.For the traditional classroom group, self-perception knowledge in project management was compared to the Project Team Builder results.
For the first scenario, the performance in the flipped classroom methodology directly and proportionally affects the results in the Project Team Builder, but not significantly (p<0.2205).
Surprisingly, but not significantly (p<0.0529), for the second scenario, the performance in the flipped classroom methodology affects inversely proportionally the results in the Project Team Builder.Stone (1994) proved that positive expectations produce overconfidence, which may affect negatively effort or attention to strategy.It would be the main reason why flipped classroom methodology students who performed very well in the first scenario then did not show up trained and ready for the second scenario.
Stone also affirms that, in contrast, negative expectations increased effort and attention to strategy.Probably because of that, students who performed poorly then increased their effort and performed better.
In conclusion, although there are slight indications that there is a correlation between the flipped classroom methodology performance and the Project Team Builder results, the null hypothesis is rejected, as the p-value is 0.22.

Other Findings
Experience in project management: Project management experience refers to students who have worked in project management; it statistically affects the performance in the Project Team Builder in the first scenario (p<0.0211).Those who had some experience performed better, not only by methodology but also by observing each course.Some papers confirm that job experience in a particular area directly affects developing deeper knowledge in the area (Mikhail, Walther, & Willis, 1997;Schmidt, Hunter, & Outerbridge, 1986).
Learning process in simulation-based training: In analogy to the effect proved on Hospitality and Restaurant Management students (Mansker, Fulks, Peters, Curtner, & Ogbeide, 2016) and what Wolfe (1993) affirmed, each run tends to improve the learning experience, panning and decision-making of the students, who perform better as long as they use the simulator.

Final Questionnaires
Satisfaction of the students-Students were asked to complete a questionnaire after the experiment to find out whether they appreciated the experiment and to measure their overall learning satisfaction.
In agreement with the literature (Street, Gilliland, McNeil, & Royal, 2015;Peterson, 2016), for all the satisfaction questions, students from flipped classroom methodology were much quantitatively "happier", also qualitatively, with the experiment than those who studied in a traditional classroom.
Satisfaction in flipped classroom courses-It was clear that 3 out of 4 students would like to continue studying under the flipped classroom methodology.At the same time, 8 out of 10 in the case of the Technion and 9 out of 10 in the Universidad Tecnológica Nacional were extremely satisfied with the teaching staff during the classes.This is higher than the same question about the current situation asked in the initial questionnaires (7 out of 10).

Summary
Both methodologies-flipped classroom methodology and simulation-based training-proved to enhance learning.Flipped classroom methodology's success is evident when analyzing students' satisfaction over those who studied in a traditional classroom.The combination of both methodologies proved to be helpful for students, especially in the first scenario where it can be seen that the influence of the flipped classroom methodology on simulation-based training is high.
The contribution of this research is twofold.First, from a theoretical perspective, simulation-based training and flipped classroom methodology literature is enriched and broadened by applying both teaching tool and methodology in a unique multicultural context.Second, from a practical perspective, an improvement in results, satisfaction and lessons learned was found when using simulation-based training under the flipped classroom methodology compared to using simulation-based training in a traditional classroom.Although some recent research (Henderson & Trotta, 2016;Liu et al., 2017) addresses the issue of combining flipped classroom methodology and simulation-based training in the medical field, until now there has been no research integrating simulation-based training and flipped classroom methodology for individual students from a qualitative and quantitative point of view.