Using Academia-Industry Partnerships to Enhance Software Verification & Validation Education via Active Learning Tools

Imparting real world experiences in a software verification and validation (SV&V) course is often a challenge due to the lack of effective active learning tools. This pedagogical requirement is important because graduates are expected to develop software that meets rigorous quality standards in functional and application domains. Realizing the necessity the authors designed and developed 42 delivery hours of active learning tools consisting of Case Studies, Class Exercises, and Case Study Videos for use in courses that impart knowledge on SV&V topics viz. requirements engineering, software reviews, configuration management, and software testing. Four key skill areas sought after by employers, namely communication skills, applied knowledge of methods, applied knowledge of tools, and research exposure are used to drive the development funded by a National Science Foundation grant and perfected through an industry-academia partnership. In this paper, we discuss in detail the four project plans the researchers and their industry counterparts followed over the past two years in the development and eventual dissemination of the active learning tools. A course enhancement plan was used to drive activities related to reviewing, enhancing, and modularizing modules, identified by a gap analysis performed by focus groups comprised of industry and academic partners. The course delivery plan was used to drive activities related to developing content delivery strategies. An evaluation and assessment plan was used to drive activities related to periodically evaluating student learning and assessing the project. And finally a course dissemination plan is being used to drive activities related to distributing course modules and assessment reports. The tools have been shared through two workshops and other means with instructors in universities and industry partners.


Introduction
In 2015, major software glitches affected numerous companies: Bloomberg (inoperable trading terminals), Royal Bank of Scotland (payments failure), Nissan (airbag sensory detectors malfunction), Starbucks (register malfunction) and the F 35 Joint Strike Fighter (targets detection failure).Each of these resulted in financial, brand, and more important bodily damages (Jee, 2016).Software is now ubiquitous and software has also become critical.With mission critical and high-risk applications that have human lives and resources dependent on software applications, it is imperative to not only test for, but aim for zero defects.The irony is that even after decades of development, the software industry continues to spend considerable time and resources dealing with the quality problem.In the US alone in 2007, the cost of failed software was estimated to be upwards of $75 billion in re-work costs and abandoned systems (Michaels, 2014).
is not enough awareness of the Software Verification & Validation (SV&V) benefits, and secondly, there are a lack of practitioners who understand the SV&V topics and processes adequately.Both the lack of awareness and personnel shortage considerably hinder significant progress in software project success rates.Furthermore, Acharya et al. (2014) argue the root cause to be the lack of up-to-date SV&V courseware.To address this situation, a SV&V course curriculum has been improved at the author's institution through a project funded by a National Science Foundation-Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics (NSF-TUES) grant.
The goal of this project is to enhance and transform a SV&V course by incorporating academic research and industry best practices through an academia-industry partnership.This project achieved the following objectives: 1) Critically examined the existing SV&V course contents, 2) Identified areas where improvements could be made in pedagogy, 3) Developed 42 delivery hours of active learning tools, 4) Developed course delivery strategies, 5) Integrated and delivered new pedagogical tools in the course, 6) Performed assessments and evaluations of the effectiveness of these tools, and 7) Disseminated course modules and assessment reports.At this time dissemination to more institutions is being carried out.This project, scheduled for completion in August 2017, targets both undergraduate students and software practitioners and: 1) Improves SV&V knowledge and skills of students & practitioners, 2) Helps evolve a SV&V community, and 3) Improves SV&V teaching and learning opportunities.
Certain contents developed through the project can also be easily adopted to teaching SV&V in graduate programs.The end result is expected to improve product and process quality levels in the software development community, resulting in a larger and more skilled SV&V user community.This paper introduces this project, describes the academia-industry partnership, and provides an in-depth description of the project's execution and outcomes.

Project Partners
Academia-industry partnership is the key to achievement of the project outcomes.From the conceptual stage, it was believed that by working hand in hand with the industry, both the academia and industry would benefit.Students would gain knowledge that is currently practiced in the industry and which they would likely use in their professional career, and industry would be able to train/retrain their personnel using a tested delivery strategy.

Industry Partners-Formation and Role
Understanding project requirements, studying possible partnerships, and selling the project vision was instrumental in getting the industry on-board.Three industry partners-Eaton Electrical Corporation, ServiceLink, and JDA Software Group-have supported this project from the time it was conceived.Additional partners, PNC Bank and ANSYS, were invited as this project progressed and the need for diversification was realized.These companies are either large software companies or companies with large software development activities in the areas of electrical meters, mortgage, and intelligent pricing and revenue management.The project team is open to more industry joining the partnership but is selective in the industry domain.The role of the industry partner is well defined and involves the following:  Critically reviewing and identifying knowledge gaps in SV&V courseware Engineering.These partners also share a strong desire to strengthen their programs.

Development Academic Partner Activities
Distinguished faculty members from the Milwaukee School of Engineering and Virginia State University (a HBCU partner) have supported this project from the very beginning as Development Academic Partners.Mutual interest is instrumental in this longstanding partnership.The role of the academic development partner is well defined and involves the following:  Identifying at least one local industry partner involved in software development activities.
 Working with assigned focus groups to critically review current course.
 Developing six hours of course modules to address identified gaps in a content area familiar to the university program and its local industry partner(s). Assessing course contents through at least two delivery cycles.

Implementation Academic Partners and Activities
This project calls for delivery of the active learning tools, as well as dissemination of the tools and assessment reports.Six domestic and one international partner have supported this project including  Using all or some of the active learning tools developed by this project in at least one course through at least two delivery cycles, and  Evaluating the course(s) and assessing the instruction at several levels.
In addition, at the request of respective faculty members, the active learning tools have been shared with University of Alaska Southeast, Georgia Southern University, University of South Carolina-Upstate, Indiana University Southeast, Rose-Hulman Institute of Technology, and Eastern Mediterranean University (Cyprus).

Focus Groups
To ensure project plans were adequately executed, focus groups comprised of PI and/or co-PIs and academic development partners and industry partners were formed based on individual's interest and expertise in the projects four SV&V topic areas: requirements engineering, software reviews, configuration management, and software testing.Well-defined activities carried out by focus groups included the following:  Researching course contents, understanding industry requirements, identifying inadequacies in the projects SV&V topic areas.
 Searching for available SV&V teaching materials.
 Developing course contents to address inadequacies.
 Assisting in developing evaluation questionnaires and instruments.

Project Implementation
In the past three years, the focus groups have followed phase-wise project plans to accomplish project objectives.
A course enhancement plan was utilized to drive activities related to reviewing, enhancing, and modularizing.A course delivery plan was used to drive activities related to developing content delivery strategies.An evaluation and assessment plan was used to drive activities related to periodically evaluating student learning and assessing this project.Finally, a course dissemination plan was (is being) used to drive activities related to disseminating course modules and assessment reports.iv.Research Exposure: Students will learn research skills.Students will submit research assignments and participate in research discussions on SV&V related topics.These skills will enable students to learn how to remain current with industry best practices and to make educated decisions in using them.

Course
The groups identified a comprehensive list of Case Studies, Class Exercises, and Case Study Videos.

Course Enhancement Methodology
An iterative development methodology depicted in Figure 1 was used to ensure the modules reflected both academic research and industry best practices.The content development process began with a meeting of the focus groups at the author's institution.The groups drafted a list of active learning content topics and delivery formats.The list was reviewed by the PI and co-PIs and shared with the partners for further review.The finalized list was then used to guide the development process.In this methodology, an industry partner or academic partner led the development effort through a collaborative effort.Once the contents were ready for review, they were shared with focus group members and subsequently with all partners.The finalized contents were then transferred to a shareable media where they became available for delivery, further reviews, and dissemination.For ease of adaptation, each active learning tool is of 25 minutes duration, with some active learning tools having multiple parts delivered in multiple sessions.

Project Results
Several course enhancement tools were developed in this work (Acharya, Manohar, Wu, Schilling, & Ansari, 2016) by the collaborative focus groups that brought together the best practices both in industry and in academia.
The domain knowledge was then packaged in active learning tools and modules that encourages significant student engagement in their learning process.Such active learning tools created in this work are described in the following sections.

Active Learning Tools
Active learning tools engage students in their learning process and lead to excitement about learning amongst the students.Active learning instructional strategies include a wide range of activities that share the common element of-involving students in doing things and thinking about the things they are doing (Bonwell & Eison, 1991).Active learning instructional strategies can be created and used to engage students in (a) thinking critically or creatively, (b) speaking with a partner, in a small group, or with the entire class, (c) expressing ideas through writing, (d) exploring personal attitudes and values, (e) giving and receiving feedback, and (f) reflecting upon the learning process (Eison, 2010).It should also be noted that active learning instructional strategies can (a) be completed by students either in-class or out-of-class, (b) be done by students working either as individuals or in group, and (c) be done either with or without the use of technology tools.This project has developed three categories of active learning tools as described in the following sections.

Case Studies
Case Studies serve as useful tools to teach software methods & processes.Raju and Sankar (1999) define Case Study education as providing students with a record of a technical and/or business issue that actually has been faced by managers, together with surrounding facts, opinions, and prejudices upon which management decisions have to depend.In this project, Case Studies are drawn from a wide range of industry SV&V practices and reported in detail elsewhere (Manohar et al., 2015).Students are presented industry standard documents for review to prepare for the tasks.This project has disseminated eleven Case Studies.Examples of some Case Studies are Understanding User Requirements, A Software Test Plan, and Importance of Peer Reviews.A complete set of case studies has been published by Acharya and Manohar (2016).

Class Exercises
Class Exercises have been designed for the class time to explicitly raise questions to invite student participation.These exercises may be questions to think further into the concepts for a deeper understanding, or practice using student knowledge with hands-on practice for problem solving.The authors have disseminated sixteen exercises (Wu et al., 2016).Examples of some Class Exercises are Requirements Ambiguity, Defect Lifecycle, and IEEE Standards.

Case Study Videos
Videos are viewed as an effective method of presenting standard material while addressing students of different learning styles.In this project, the Case Study Videos provide a realistic picture for the audience to appreciate many SV&V processes being practiced.The video scripts were first drafted by the industry partners and subsequently confirmed by the testimonies shared in focus group discussions.The process has been described in details in a previous publication (Acharya, Manohar, & Wu, 2016).Four videos, ranging from 15 to 24 minutes, have been disseminated.Some Video Case Study titles are Formal Inspection Scenes and Requirement Analysis Scenes.

Building a Collaborative SV&V Community
As mentioned in Section 2, the project has brought together 5 industrial partners, 2 academic development partners and 7 national and international academic implementation partners.In addition, over 14 different academic and industrial organizations have taken advantage of the summer workshops.Thus about 40-50 people working full time or teaching in SV&V area have been working together and collaborating in developing and implementing the active learning tools and thus building a thriving SV&V professional community (Acharya, Manohar, Wu, Schilling, & Ansari, 2014).This is a really significant result of this project that is most likely to continue in future well beyond the time frame of the project.

Course Delivery
The following narrative explains the instructor kit, and delivery strategy of the enhanced SV&V course that deployed the active learning tools in the classroom.The course was delivered at the author's host institution as well as other academic institutions and industrial organizations.The academic implementation partners have started implementation at various levels of adaptation in their respective institutions in their 2015 and/or 2016 terms.The industry partners are considering taking selected tools to use in their on-the-job training for practitioners.

Instructor Kit
To

Course Delivery Strategy
As student learning retention and eventual application of the knowledge gained depends on knowledge retention activities delivered in different settings it is important to identify and incorporate delivery strategies that meet course and/or module learning outcomes.Researchers in both the academia and the industry have published literature on delivery strategies.Some of the more prominent strategies used in higher education are lecture, case method, discussion, active learning, cooperative learning, integrating technology and distance learning (Teaching Strategies, http://www.gmu.edu/resources/facstaff/part-time/strategy.html).Mishra et al. (2014) report that prior knowledge about software quality concepts and good programming skills can help students to achieve success in a software verification & validation course.Furthermore they also suggest that team work can be chosen as a strategy, since it facilitates students' understanding and motivates them to study.
With the active learning tools designed to impart practical knowledge into theoretical understanding, we encourage a flipped classroom model (Bonwell & Eison, 1991) in which we can maximize class time for active learning tools so as to engage the students for further digestion of the knowledge in the context of industry practices.Students are expected to be prepared outside of the classroom beforehand, with assigned textbook readings or reviewing of online materials.For effective delivery it is also recommended that students work in small teams.
The active learning tools are built on basic knowledge and engage the students in different ways to study SV&V in practice.The Case Studies are explicit in this approach: each Case Study makes the point to consider issues in realistic practices.Instructors can present the Case Study while guiding students into further study and discussion of the practical issues in SV&V.The Class Exercises are designed for interaction in the classroom during group discussions.The instructor brings the question(s) as the moderator and guides the discussion session.The instructor may also use the Class Exercise to lead students into subsequent group mini projects or individual mini projects.The Instructor Notes component of the Class Exercise discusses these possibilities.Students are likely to find the Case Study Videos by their nature as multimedia, as highly engaging.These videos share real-life perspectives of actions and their consequences.The videos are by design in sequences of scenes.For instructional purposes, it is highly beneficial to "pause" the video at appropriate moments so as to engage the students in discussion.Suggested discussion questions accompany the videos.
As mentioned before, the active learning tools are modularized into flexible modules.Instructors may consider their various needs such as curriculum design, class time, and class size to adapt the active learning tools to the situations in their institutions.Furthermore, though we recommend it, instructors do not need to practice the flipped classroom model or may do so partially.
To summarize, the course delivery plan encourages the following: The recommended flipped classroom model is used to deliver the contents at the host institution.A week-by-week course delivery plan, developed by the focus groups, was used to deliver a course on SV&V in spring 2015.The feedback from the students and the instructor was used to revise the tools and the delivery plan.Table 1 depicts the revised delivery schedule being used at the author's institute in spring 2016 delivery of the same course.At the author's institute this 3-credit course meets 4 hours a week.

Course Delivery at other Academic Institutions
It is recommended that the Academic Institutions follow a similar delivery plan.However, they may choose to offer the enhanced course materials as a new course offering or as modules that fit into their existing course offerings.Partners may also supplement contents with locally developed materials.These supplemental materials will be part of the course repository if the PI and the partners agree that such materials add to student learning and teaching of such materials can be easily replicated by implementing instructors.

Course Delivery at Industry Partner Facilities
Industry partners local to the author's institute are considering delivering specific modules to their employees at their facilities or at the author's institution.During assessment periods, industry partners will deliver selected modules and provide assessment results.Using the provided assessment tools, both academic and industry implementation partners are expected to have their students and instructor(s) complete surveys pertaining to the tools delivered and the delivery strategy.

Project Assessment and Evaluation
Project assessment and evaluation are important activities in this project.The following narrative explains the overarching project goals, followed by a description of the assessment and evaluation plan by both internal and external evaluators.The narrative also provides samples of actual assessments conducted by the different stakeholders.Various stakeholders in this project include the students (SIR II surveys and in-class feedback surveys), faculty (faculty course assessment reports), industrial and academic partners (surveys) and external evaluators (surveys and assessment reports).Each of the stakeholders have been completing work diligently.
There is an abundance of information available at this stage to identify the strengths and weaknesses of this project, and then chart out the future course of action as this project moves rapidly in the dissemination and conclusion phase.

Project Assessment and Evaluation Plan
The expected project outcomes that are needed to be assessed by the project evaluators and the suggested evaluation tools are summarized as follows:  Outcome 1: Project improves knowledge and skills of students  Assessment tools (in-use and suggested)-outcomes assessment in class (faculty), SIR II surveys (students), industry training feedback (industrial partners/trainers). Outcome 2: Project helps evolve SV&V community-focus groups work together in developing appropriate pedagogical tools for enhancing SV&V education:  Assessment tools (in-use and suggested)-Organization of meetings, workshops and seminars, e-mail based communication, surveys, LinkedIn group membership, revisions, YouTube videos, new curriculum or revision developed and delivered at other institutions, curriculum adaptations (PI, Co-PI to implement, external evaluators to assess this activity). Outcome 3: Project improves SV&V teaching and learning opportunities  Assessment tools (in-use and suggested)-expanded availability of new teaching and learning tools and materials-books, class notes, Case Study Videos, Case Studies, Class Exercises (PI, Co-PI to implement, external evaluators to assess this activity).
The focus groups have worked with the external evaluators to develop the evaluation questionnaires and instruments for this plan.

Examples of Project Assessment and Evaluation from External Evaluator
The external evaluator was provided with all the student feedback forms, industrial and academic partner feedback forms, and also online surveys completed by the developers and users.Based on all of these diverse data sources, the external evaluator submitted a Year-End Program Evaluation reports for years 2014-2015, and 2015-2016.The specific questions that were examined in this Year-End Program Evaluation were as follows: 1) To what degree are industry and academic partners involved and satisfied with their involvement in NSF TUES Project # 1245036?
Assessment Result: The data were collected from the industry partners, using a short questionnaire that consisted of closed-and open-ended items.Results show a strong, on-going, frequent communication and collaboration between the academic and industrial partners to identify contemporary and emerging domain-specific issues and topics and then working together to discuss appropriate pedagogical tools to be developed to deliver enhanced learning experiences for the students.
2) To what degree were best-practices in industry and research utilized during development of the instructional materials?
Assessment Result: The evaluation showed that all of the Class Exercises and Case Studies followed the intended template, and all materials required for each class activity have been developed and will be provided to participating instructors for future class sections.Each activity required communication skills and application of course content.The topics and issues delivered in the new active learning tools were based on the brain-storming sessions between the academic and industrial partners to maintain a strong currency of knowledge.
3) To what extent did students learn from and find the newly developed materials to be useful?
Assessment Result: Basic psychometric analysis of the items on each pre-and post-test was conducted, and a paired samples t test was conducted to determine significant differences in the pre-and post-scores.The data show some improvement from pre-to post-test.There was a mean difference of approximately one point, with the post-test scores being higher.It should be noted that the sample size is small (9 students) for utilizing t tests and item analysis, and the Principal Investigators should use the results from these analyses keeping the limitation of sample size in mind.In future semesters, similar analysis will be run based on data collected from a variety of classes at several different institutions, and larger sample sizes are likely to provide more usable information.

Examples of Project Assessment and Evaluation from Academic Partners
Several academic partners adapted the newly developed curriculum to different degrees and also delivered some of the active learning tools.The effectiveness of the curriculum and the tools was assessed based on the data collected from the students, including pre-and post-tests, a general evaluation of course instructional activities, and detailed evaluations of specific pedagogical tools.The data sources collected from students included the following: Assessment Results: Basic psychometric analysis of the items on each pre-and post-test was conducted, and a paired sample t test was conducted to determine significant differences in the pre-and post-scores.It was found that there was significant improvement in students' post-test scores in the areas of IEEE standards and configuration management.For other areas, there was some improvement, but it is determined to be statistically significant at this time in the data collection process.Some of the assessment questions may need to be rewritten to improve clarity, while for other questions, an increase in the degree of difficulty would help in discerning differences in student learning.

Student Feedback
The student feedback about specific pedagogical tools was also obtained in the form of open ended questions.These data are currently being analyzed quantitatively and will be reported at a later date.Some of the representative comments from the student feedback are summarized below:  Security Inspection Video: This video did facilitate small and large group discussions.I thought the acting in the video was actually pretty decent it portrayed events that could have actually happened.It's always good to have students think about subject and then explain the right thing to them because it shows if they were right or not.It was nice to see examples of security vulnerability and some ways to look to prevent this.I understand to always cover security thoroughly and often during development.
 Cost Effective Testing Exercise: Fun and informative!This was good activity to think critically about how to test a system. Software Requirements Specification (SRS) Review Exercise: Useful.Very long and detailed.Does look like a real world document but if made shorter could get through entire document.
 Test Case Development: It helped to show the importance of working through case studies.I thoroughly enjoyed the activity.I was able to analyze proper methods of creating test cases.Practical and straight to the point.I enjoyed comping up with individual solutions before discussing as a group.

Examples of Project Assessment and Evaluation from the Instructor
One of the authors has been delivering a SV&V course since 2005 and is required to perform an ABET Criterion 3 outcomes assessment.In the spring 2013 term, when the new pedagogical tools were not yet available, there was a weakness associated with the ABET learning outcome "e" (an ability to identify, formulate, and solve engineering problems), where less than 60% of the students scored better than 80% on the assessment tasks.When appropriate Case Studies were developed in 2014 and implemented in the Spring 2105 and 2016 terms, the student performance related to outcome "e" rose to the excellent range (>=90%).This presents clear evidence that the Case Study based teaching method is more effective in supporting student learning and increasing the ability to identify, formulate, and solve engineering problems.Therefore, Case Study based educational tools will be progressively developed, adopted, and delivered in several other aspects of the SV&V area such as legal issues in software, software consumer protection, and requirements from the customers' perspectives.The results of those implementations will be reported later as more data become available.

Students' Perceptions of the Course Instructional Materials
Table 2 provides a summary of students' perceptions of the course instructional materials.Twelve students completed the questionnaire.Students were asked to rate each type of assignment or activity according to the scales shown in Table 2.The results show that 90% or more of the students who just completed the class found the exercises, case studies, video case studies, full class discussions, and projects to be moderately or extremely useful.Students found less utility in the written homework assignments and textbook readings.
The end of course questionnaire also asked students about how often they do several tasks in a typical course in their major and in the current class.Results shown in Table 3 suggest that in typical classes in their major, the majority of students pay attention in class, but only engage in small group and large class discussions to a small or moderate degree.
When asked the same questions about the current class, results related to many items were similar, but more students reported that they complete real world exercises and feel accountable to classmates in full class discussions (Table 4).These results support the inclusion of the hands-on, real-world application activities as well as the integration of communication skills into the activities.

Course Dissemination
In consonance with NSF's broad-dissemination requirement, the project materials and assessment reports have been disseminated through multiple channels.The following narrative explains the execution of the course dissemination plan.

SV&V Workshops
The project team hosted two, two days Software V&V workshop.In the August 2015 workshop eleven academic implementation partners, including academic development partners, travelled to Pittsburgh, PA.In addition 6 representatives from industry who had participated in the development of materials also attended.In the August 2016 workshop nine new academic implementation partners, one academic development partners, and 1 new industry partner and 1 representatives from the industry participated.The objective of this workshop was to share the course modules and to take the participants through delivery strategies.During this workshop, representatives of focus groups led the participants through a subset of the materials and also explained in detail the delivery strategies.Half a day was allocated for each active learning tool category.It is felt that through this workshop a strong interaction between facilitators and peer learners was established.Both the academic and industrial participants were provided with a set of the larger materials on a flash drive and a link to the instructor kit.

Centralized Course Repository
In this project, initially Dropbox was used as the centralized course repository.Forty-two hours of active learning tools have been stored in Dropbox.Now the active learning tools are available through the project website www.rmu.edu/nsfvv(depicted in Figure 2) and ENSEMBLE, a Computing Portal connecting Computing Educators, accessible through www.computingportal.org(depicted in Figure 3).The tools and supporting documents are organized based upon SV&V topics.Folders are provided for activities related to Configuration Management, Requirements Management, Software Reviews, and Software Testing.Underneath each of these folders are folders for the active learning tools: Case Studies, Class Exercises, Case Study Videos, and Topical Assessments.For greater availability, the videos have been uploaded to YouTube. Figure 4 depicts a scene of a Case Study Video as seen in YouTube.

Conferences and Publications
Research papers on project activities and assessments have been or are in the process of being presented at multiple conferences.

LinkedIn Group
A LinkedIn group called "Software V&V Community" has been established to enable SV&V implementers to avail themselves of peer-to-peer learning opportunities.This moderated group provides a venue to educate and to consult.Implementers have shared ideas and best practices, asked and answered questions, and collaborated.
This group is open to all interested parties.

Figure 1 .
Figure 1.Development methodology ensure instructors have the resources they need to start a new course or strengthen existing courses an instructor kit has been created.The kit consists of: active learning tools, sample syllabus, pre-tests/post-tests and answer key, and sample of midterm/final exams.Each active learning tool module consists of the following components: a) Active learning tool description b) Instruction notes c) Student handout d) Assessment instrument Using the flipped classroom model (if applicable)  Have students work in small teams (2-3 students per team)  Delivering tools in one or multiple 25-minute sessions  Using pre-test and post-test instruments to tailor course delivery  Evaluating student learning of the module immediately after delivery 5.2.1 Course Delivery at the Author's Institution

Figure 2 .
Figure 2. Active learning tools in project website : Embry-Riddle Aeronautical University, Montana Tech, University of Michigan (Dearborn), Virginia State University, Fairfield University, Milwaukee School of Engineering, and ORT Braude College, Israel.Partners were identified through networking at the American Society for Engineering Education conferences, and sharing of the author's research.
A workshop held at the author's institute in August 2015 led to the inclusion of five other implementation academic partners namely: Auburn University, East Carolina University, Kennesaw State University, Bowie State University, and Clarion University.Likewise a second workshop held at the author's institute in August 2016 led to the inclusion of nine other implementation academic partners namely: Rocky Mountain College, Kentucky State University, Mount Mercy University, University of Maryland, Baldwin Wallace University, Whitworth University, SUNY Oneonta, George Mason University, & Keynon College.The role of the academic implementation partner is well defined and involves the following: Enhancement PlanProject objectives, target population, SV&V topics, employer-sought key skill areas, and a sound development methodology were used by the focus groups to develop 42 hours of active learning tools consisting of Case Studies, Class Exercises, and Case Study Videos.The following narrative explains the execution of the course enhancement plan.

Table 1 .
Course delivery schedule  CLASS EXERCISE: Implied and Stated Requirements  CASE STUDY -Requirements from a Customer Perspective -Ambiguity  CLASS EXERCISE: Developing Use Case from Needs Statement Quiz Related to IEEE Standards  Pre-Post Quiz Related to Configuration Management  Pre-Post Quiz Related to Requirements, Requirements Engineering, Requirements Management  Pre-Post Quiz Related to Software Peer Reviews  Pre-Post Survey on Software V Focus Areas  Post-Only General Student Evaluation of Couse Instructional Activities  Post-Only Student Evaluation of Specific Course Instructional Activities

Table 2 .
Descriptive results from the general evaluation of course instructional activities part 1

Table 3 .
Descriptive results from the general evaluation of course instructional activities part 2

Table 4 .
Descriptive results from the general evaluation of course instructional activities part 3

Address how often you do each of the following in a TYPICAL DAY IN THIS CLASS:
To date, conference papers have been presented at ASEE 2014, ASEE 2015, EDSIG 2015, ASEE 2016, and WMSCI 2016 annual conferences.A keynote address was also delivered in WMSCI 2016.In 2016 this project was presented at the NSF Showcase at SIGCSE 2016 and in the Envisioning the Future of Undergraduate STEM Education: Research and Practice symposium organized by AAAS.A workshop was conducted in EDSIG 2016.Research work related to this project has been published in multiple journals.A book on SV&V Case Studies has been published by the Alexandria Street Press (online).