Procedures for Developing Evidence-Informed Performance Checklists for Improving Early Childhood Intervention Practices

A conceptualization-operationalization-measurement framework is described for developing evidence-informed early childhood intervention performance checklists. Performance checklists include lists of practice indicators where the indicators, taken together, operationally define particular types of intervention practices that, when used as intended, can be expected to have desired outcomes and benefits. The use of the framework for operationalizing recommended practices is described to illustrate its applicability for developing tools for improving early childhood intervention.


Introduction
Early childhood intervention involves the provision of different types of supports and resources to young children birth to 6-8 years of age and their families to improve child, parent, and family functioning (Shonkoff & Meisels, 2000).The term early childhood intervention encompasses early intervention for infants and toddlers with identified disabilities or developmental delays (Dunst & Espe-Sherwindt, in press), early childhood special education for preschoolers with identified disabilities (Reichow, Boyd, Barton, & Odom, 2016), early years education for infants, toddlers, and preschoolers who are at-risk for poor developmental outcomes due to family circumstances (Burger, 2010), and early childhood education for all young children birth through eight years of age (Copple & Bredekamp, 2009).
The knowledge base for early childhood intervention practices has expanded rapidly in recent years to the point that it is almost impossible for any early childhood intervention practitioner to keep pace with advances in theory, research, and practice (e.g., Guralnick, 2005;Reichow et al., 2016;Shonkoff & Meisels, 2000;Sukkar, Dunst, & Kirkby, 2017).A PsychINFO search for "early childhood intervention" AND "models OR research OR practice" yielded over 12,600 results, and a Google Scholar search using the same search terms resulted in 18,700 hits.A Google Scholar search of "early intervention" and "model OR research OR practice" resulted in 1,120,000 hits.These search results, taken together, are an indication that the knowledge base far exceeds one's capacity to keep up with the contemporary knowledge explosion in early childhood intervention.exceeds an individual's ability to deliver its benefits correctly, safely, or reliably" (p.13).More than 50 years of research and practice in early childhood intervention has resulted in a knowledge base that challenges even the most highly trained and hardworking practitioner's ability to remember and internalize the key characteristics of multiple sets of interventions as a matter of routine practice (Druckman & Bjork, 1994;Jaber, 2011).This is at least one factor that most likely contributes to poor fidelity of use of intervention practices and less than optimal intervention outcomes and benefits (e.g., Vancel, Missall, & Bruhn, 2016).As noted by Gawande (2009), the solution "seems almost ridiculous in its simplicity… It is a checklist" (p.13).Checklists include the minimal number of clearly stated actions or steps for a practitioner to achieve outcomes of interest.

Early Childhood Intervention Checklists
Different types of checklists have been developed for different purposes (e.g., Oxman, 1994;Rowlands, 2007;Stufflebeam, 2000;Taylor, Grey, & Satterthwaiter, 2013).Wilson (2013) describes the intended purposes of eight different types of checklists, one of which "provides a list of tasks or steps required to complete a procedure successfully… and serve as concrete reminders of what tasks need to be performed" to ensure the procedure is implemented correctly (p.4).This type of checklist is described in this paper as a performance checklist whereas Wilson (2013) describes it as a procedure checklist.
The ability to use an early childhood intervention practice competently is dependent, in part, on a clear understanding of key characteristics or active ingredients of the practice.This is often facilitated by using performance checklists that include lists of practice indicators or key characteristics (Westgaard, 2001).Performance checklists include sets of indicators that, taken together, define a particular skill set or practice that is intended to have desired benefits or outcomes (e.g., Greenwood, Luze, Cline, Kuntz, & Leitschuh, 2002;Roggman et al., 2013).These types of checklists include the behavior, skills, or actions that define desired performance or the ability to implement a practice as intended.Checklists are especially useful for a practitioner to understand the key characteristics of a practice or expected performance because they include indicators for knowing if a procedure, process, or activity has been implemented in a competent manner (C.Wilson, 2013).
Performance checklists have been used widely in a number of different fields and professions (e.g., Gwynne, Blick, & Hughes, 1996;Lockyer et al., 2006;McSwain, Mahan, & Herrin, 1979;Perry, Potter, & Ostendorf, 2014) including education (Rowlands, 2007;Sullivan, 1998) and early childhood intervention (e.g., Casey & McWilliam, 2011;Greenwood et al., 2002;Roggman et al., 2013).Karges-Bone (2000), for example, developed more than 50 performance checklists to facilitate student knowledge acquisition, teacher classroom management, and parent-teacher communication.Similarly, Herbert-Jackson et al. (1977) and O'Brien et al. (1979) developed performance checklists of procedures for administering child care centers; carrying-out feeding, diapering, and sleeping routines; classroom organization; transitions between classroom activities; preparing children's snacks and lunch; facilitating child play; and child health and safety procedures among other practices.More recently, McWilliam's (2010) book, Working with Families of Young Children with Special Needs, includes nine chapters each of which includes checklists that practitioners can use to improve their interventions with young children with developmental delays or disabilities and their families.
Despite the fact that checklists are widely used in a number of professions to improve practitioner performance, very little has been explicitly written about procedures for developing checklists.Results from literature reviews by Frank (2006) and Winters et al. (2009), for example, found that articles describing the development of checklists rarely included descriptions of the procedures that guided checklist development (see C. Wilson, 2013, for an exception).Most often, expert panels or consensus procedures have been used to develop checklist items (Gorter et al., 2000).Winters et al. (2009), based on their review of checklists, concluded that checklist development could be improved considerably if scientific evidence was used to inform checklist indicator development.
The purpose of this paper is to describe an evidence-based approach to performance checklist development that combines key features of practice-based research syntheses for identifying evidence-informed checklist indicators (Dunst, 2016) and a framework for systematically operationalizing intervention practices (Babbie, 2009).The approach has been used to develop performance checklists for research purposes (Dunst, Trivette, & Raab, 2015).This paper uses the same approach with several extensions to illustrate its applicability for developing performance checklists for improving early childhood intervention practices.

Framework for Developing Evidence-Informed Performance Checklists
Figure 1 shows the framework used to develop evidence-informed performance checklists.The three overlapping circles depict the relationship between different types of intervention practices (conceptualization), procedures for identifying key practice characteristics (operationalization), and the use or nonuse of the key practice characteristics (measurement) where findings from practice-based research syntheses are used to inform the foundations for internally consistent sets of intervention practices.The indicators in turn are used to develop checklist items that serve as reminders or cognitive tools that help practitioners implement a practice as intended where fidelity of use of the practice is hypothesized to be associated with outcomes of interest (Dunst, Trivette, & Raab, 2013;Wolery, 2011).
Figure 1.Framework for developing evidence-informed and operationally defined performance checklist indicators

Practice-Based Research Syntheses
A practice-based research synthesis focuses on unpacking and disentangling an intervention practice to identify which practice characteristics "go together" and matter most in terms of explaining outcomes of interest (Dunst, 2016).This ensures practice indicators are evidence-informed where the relationship(s) between the indicators and the intended outcomes of the indicators have been empirically replicated in studies in a practice-based research synthesis (Dunst & Trivette, 2009b;Trivette & Dunst, 2013).Odom (2008), for example, noted that this type of research synthesis can help tease apart early childhood practices in ways that can inform how best to implement the core components, active ingredients, or key characteristics of the practices.
An example from adult-child interaction research is used to illustrate how the key characteristics of an intervention practice are identified and become the foundations for checklist indicators.Richter (2004), as part of a review of caregiver-child interactions, concluded that particular features of caregiver behavior "stand out" as being particularly important for optimal child growth and development.de Wolff and van IJzendoorn (1997) and Kassow and Dunst (2007), in reviews of 75 studies including more than 4500 caregiver-child dyads, found that 10 different caregiver behaviors have been hypothesized to be important determinants of child outcomes.Dunst and Kassow (2008), in a practice-based secondary reanalysis of the 10 caregiver behaviors, found that four of the behaviors proved most important in terms of explaining variations in child outcomes.The four behaviors included both sensitivity to child behavior initiations and caregiver contingent responsiveness to child behavior.The results were used to develop interaction practices indicators on checklists to improve adult-child interactions (e.g., Raab & Dunst, 2006).

From Research Evidence to Performance Checklists
Babbie's ( 2009) framework for differentiating between conceptualization, operationalization, and measurement has proven especially useful for developing different kinds of early childhood intervention performance checklists (Dunst et al., 2015).Conceptualization refers to the process of identifying or defining as specifically as possible the meaning of a construct or practice (e.g., responsive parenting).Operationalization refers to the development or identification of the indicators that "make up" the key characteristics of a construct or practice (e.g., positive affect, following a child's lead, sensitivity to child initiations, and contingent responsiveness to child behavior as indicators of responsive parenting).Measurement refers to the method used to determine the presence or absence, or degree to which, the indicators were used as part of a practitioner or parent using a practice (e.g., using a Likert scale).
The adaptation of Babbie's (2009) conceptualization-operationalization-measurement framework had been especially useful for developing performance checklists.In terms of checklist development, conceptualization is the process used to describe the meaning of a particular type of practice (e.g., responsive teaching) and to distinguish the practice from other related practices (e.g., incidental teaching, milieu teaching).Dunst et al. (2011), for example, content analyzed six different naturalistic teaching methods and found that despite similarities in the different practices, there were procedural constructs unique to each practice.The analyses led to a more precise meaning of each type of instructional practice.
The process of identifying indicators of a practice is what is meant by operationalization.Results from practice-based research syntheses are especially informative for identifying evidence-informed indicators.Raab et al. (2013), for example, identified six evidence-based indicators of responsive teaching in a practice-based research syntheses of 46 studies including more than 5000 study participants.The findings in turn were used to develop a performance checklist for practitioners to promote parents' use of responsive teaching with their children as part of everyday child learning activities (Raab, Trivette, Dunst, & Adkins, 2013).
Performance checklist measurement is intended to be used by a practitioner to do a self-assessment of how many and how well checklist indicators were able to be used for a particular checklist practice.As noted by both Gawande (2009) and Wilson (2013), performance checklists are cognitive tools that serve as reminders of intended practices and as benchmarks for determining if actual performance mirrored expected performance.Findings from several practice-based research syntheses of adult learning studies indicate that coach-facilitated practitioner self-assessment is associated with optimal learner benefits (Dunst & Hamby, 2015;Dunst, Trivette, & Hamby, 2010).

Checklists for the DEC Recommended Practices
The experiences, lessons learned, and findings from both research and field-test evaluation studies have proven valuable in terms of informing the development of the framework and procedures for developing evidence-informed performance checklists described in this paper.The framework and procedures were recently used to develop performance checklists for early childhood practitioners to understand, adopt, and use different kinds of recommended early childhood intervention practices (Division for Early Childhood, 2014).The performance checklists were developed at the Early Childhood Technical Assistance (ECTA) Center at the Frank Porter Graham Institute at the University of North Carolina-Chapel Hill.

DEC Recommended Practices
The DEC Recommended Practices include eight topic areas and 66 individual practices.The topic areas include assessment, environment, family, instruction, interaction, leadership, teaming and collaboration, and transition practices.The number of practices for each topic area varies from 2 (transitions) to 14 (leadership).The types of practices in each topic area vary from broad categories of practices (e.g., environment, transitions) but with no practice indicators to collections of interrelated practice indicators (e.g., interaction, instruction).
Four of the eight topic areas include child and family intervention practices (environment, family, instruction, and interactions), three topic areas include practitioner practices (assessment, teaming and collaboration, and transitions), and the other topic area includes program leadership practices.Close inspection of the individual topic area finds that the practices (broadly construed) differ considerably in their formatting, specificity, and internal coherence (i.e., how well the practices "go together" and operationally define particular types of practices).For example, the environment topic area includes basically a list of different types of practices (e.g., universal design, assistive technology, adaptations, etc.) with no descriptions of the practice indicators that make up each type of practice.In contrast, the instruction topic area includes different types of instruction (embedded and systematic), practice characteristics (e.g., identifying child strengths, preferences, and interests; practitioners use of explicit feedback), and other recommended practices (e.g., duration of instruction needed to address child learning).The other topic area practices fall somewhere in between these extremes.
The lack of specificity of the DEC Recommended Practices both in terms of the interventions and outcomes of the practices, and the fact that individual practices are formatted so differently, makes their usefulness to early childhood intervention practices questionable.These among other factors were the foundations for developing performance checklists where each checklist includes operationally defined evidence-informed checklist indicators developed in the manner described above.Twenty-nine performance checklists were developed for the 66 DEC Recommended Practices (www.ectacenter.org/decrp/type-checklists.asp).

Performance Checklists
The process of developing performance checklists for the DEC Recommended Practices began by a content analysis of each topic area to identify either specific types of practices (e.g., environmental adaptations, assistive technology) or internally consistent sets of practice characteristics that are considered the active ingredients or key characteristics of a particular intervention practice (Dunst, Raab, & Trivette, 2013b).The content analysis was informed primarily by findings from research syntheses where different sets of practice characteristics were found to be related to the intended outcomes of the practice (see e.g., Dunst, 2016;Dunst & Espe-Sherwindt, 2016).This process resulted in the identification of different subsets of practices for each topic area where each subset described or defined a specific kind of practice (e.g., family capacity-building practices).The process mirrors what Babbie (2009) describes as conceptualization.
The evidence-informed subsets of practices for each topic area were then used to identify or develop a set of key characteristics or active ingredients of a particular kind of practice which were used as the performance checklists' practice indicators.The sources of the indicators were the different DEC Recommended Practices topic area practices to the extent that they were evidence-based.For example, a content analysis of the 10 family practices was used to develop four performance checklists (family-centered practices, informed family decision-making, family engagement, family capacity-building) where the practice indicators were informed by the descriptors in the 10 family practices.This process is akin to what Babbie (2009) describes as operationalization.
The ability to use a checklist to plan interventions that include all or most of the checklist indicators or to assess whether the indicators were used as part of implementing the checklist practices is facilitated by a 4-point Likert scale ranging from Seldom or Never (0-25%), Some of the Time (25-50%), As Often As I Can (50-75%), or Most of the Time (75-100%) was able to incorporate each checklist indicator in an intervention plan or use the indicators as part of an intervention with a child or family.The use of the Likert scale to assess adherence to the practice indicators is what Babbie (2009) describes as measurement.This type of scale is often used to measure the fidelity of use of an intervention practice (Dunst et al., 2013;Rudnick, Freeman, & Century, 2012;Wolery, 2011).
Each performance checklist is formatted in the same way to facilitate practitioner understanding and use of the different sets of operationalized practice indicators.Each checklist includes: (1) a brief description of the purpose of the checklist and how it can be used to plan or evaluate a specific type of a practice, (2) a set of internally consistent evidence-informed operationalized practice characteristics, (3) a rating scale for assessing how much or how well the practice characteristics were able to be used by a practitioner, and (4) space for a practitioner to record notes.The goal was to have similarly formatted checklists in order for practitioners to have a set of tools informed by the DEC Recommended Practices to facilitate understanding and use of the checklist practice indicators.
Figure 2 shows one of the four checklists for the family practices.A practitioner can use the checklist to promote and support parents' or other primary caregivers' abilities to provide their children development-enhancing learning opportunities as part of everyday activities and routines.The research foundations for the capacity-building practices include findings from research reviews of family-centered participatory help giving practices (Dempsey & Keen, 2008;Dunst, Trivette, & Hamby, 2007, 2008;Rosenbaum, King, Law, King, & Evans, 1998) and adult learning studies of authentic learning opportunities (Dunst & Hamby, 2015;Dunst et al., 2010).Findings reported in these research reviews as well as results from individual studies (e.g., Dunst & Dempsey, 2007;Swanson et al., 2011;Woods & Brown, 2011) indicate that active parent participation in using everyday familiar activities as sources of child learning opportunities together with practitioner suggestions, supportive feedback, and encouragement, are associated with changes and improvements in parenting confidence and competence (Dunst et al., 2008(Dunst et al., , 2010)).
The indicators on the family capacity-building checklist were developed by unpacking two DEC family recommended practices that include parents' active participation in acquiring an understanding and use of everyday activities as sources of child learning and the family strengthening consequences associated with the use of the practices (Dunst et al., 2007(Dunst et al., , 2008)).The nine checklist indicators incorporate this information into the checklist content in ways that constitute an internally consistent set of practices that, taken together, operationalize the purposes and outcomes of family capacity-building practices.The 4-point Likert scale on the checklist provides a practitioner one way of determining if he or she was able to use the practice indicators to plan or evaluate his or her performance.

Discussion
As noted by both Gawande (2009) and Wilson (2013), performance checklists serve two important purposes: (1) they specify the actions or steps that define expected or desired performance and (2) they reduce or eliminate the likelihood that a practitioner will miss or fail to enact one or more steps or actions.Both experts note, however, that checklists are not the answer to all problems or meet all needs.When, then, are checklists warranted as mnemonic devices or cognitive tools for improving practitioner competence?Gawande (2009) describes work by Glouberman and Zimmerman (2003) on the science of complexity that indicates different problems and tasks can be categorized as simple, complicated, or complex.Simple tasks such as cooking, household chores, and raking leaves do not warrant the use of checklists because once they are learned they are carried out as a matter of routine.In contrast, checklists can be especially useful for implementing both complicated and complex tasks.Complicated tasks include such things as flying an airplane, building a rocket, and heart surgery.These are complicated tasks but with enough experience and guidance, "there is a high degree of certainty of the outcomes" (Glouberman & Zimmerman, 2003, p. 22).
Interestingly enough, Glouberman and Zimmerman (2003) describe raising a child as a complex task.This is the case because no two children are alike, child rearing practices are likely to vary for individual children, and perhaps most important, "their outcomes remain highly uncertain" (Gawande, 2009, p. 49).The latter is an operational characteristic of complex tasks (Glouberman & Zimmerman, 2003).This child rearing example of a complex problem is easily extended to early childhood practitioners who in the course of their work interact not only with many different children who are "not alike" but also families who differ considerably in their complexities as well.Herein lies the value of checklists.The more checklists a practitioner has at his or her disposal, the more likely a tool is available to carry out particular tasks tailored to individual situations and contexts.This in turn increases the likelihood that tasks are completed as intended and have intended outcomes and benefits.

Strengths and Limitations
The procedure for developing checklists described in this paper resulted in similarly formatted tools so that as one moves from one checklist to another checklist, no relearning is necessary (Jaber, 2011).This is in marked contrast to the rather haphazard formatting of the DEC Recommended Practices (Division for Early Childhood, 2014) where the practices differ considerably in their formatting, specificity, and comprehensiveness.The procedure also resulted in individual checklists for a particular practice and a set of internally consistent indicators that, taken together, operationally define the key characteristics of the practice (Dunst, 2016).The fact that the procedure itself results in evidence-informed indicators based on findings from practice-based research syntheses increases the likelihood that the use of the practice indicators will have intended benefits and outcomes (e.g., Dunst, 2016).This is also in marked contrast to the DEC Recommended Practices (Division for Early Childhood, 2014) that include no evidence for the practices in each topic area.The shortcomings and limitations of the DEC Recommended Practices were, to a large degree, the reason the ECTA performance checklists were developed in order to have evidence-informed tools that practitioners could use to implement practices that mirror the intent of the DEC Recommended Practices, where the checklists themselves provide a sense of coherence and order in terms of the "make up" of different early childhood intervention practices.
Checklists in general, and the types described in this paper in particular, are not for everyone or every purpose.Checklists, for example, are not able to account for every possible contextual or situational variation that an early childhood intervention practitioner will likely encounter in day-to-day practice (C.Wilson, 2013).Similarly, checklists may not fit well with a practitioners' learning style and stifle rather than facilitate skill development (Brady, 2013).These, as well as other factors, need to be considered when deciding if a checklist is an appropriate tool to promote practitioner understanding and competence.

Figure 2 .
Figure 2. Example of an ECTA Center performance checklist for practitioner-implemented family capacity-building practices