Parents’ Social Validity Appraisals of Early Childhood Intervention Practice Guides

Findings from three field-tests of parents’ ratings of early childhood intervention practice guides are reported. Results from the first field-test were used to inform changes to the practice guides in the second field-test, and results from the second field-test were used to inform changes to the practice guides in the third field-test. Parents’ judgments of the practice guide designs and their social validity appraisals of the practice guide intervention activities and child outcomes were correlated with parent-informed improvements in the intervention materials. The results add to the knowledge base in terms of how parent-informed improvements to the practice guides are related to product design judgments and the social validity appraisals of the importance and acceptability of early childhood intervention materials.


Introduction
Early childhood intervention includes the learning experiences and opportunities used with infants, toddlers, and preschoolers to promote and enhance the children's development (Dunst & Espe-Sherwindt, 2017;Groark, Eidelman, Maude, & Kaczmarek, 2011;Guralnick, 2016;McWilliam, 2015).Early childhood intervention practitioners often support and strengthen parents' and other primary caregivers' abilities to use these experiences and opportunities to promote the learning and development of their young children (e.g., Acar & Akamoğlu, 2014;Bernheimer & Keogh, 1995;Friedman, Woods, & Salisbury, 2012).There is, however, considerable variability in the extent to which parents use the experiences and opportunities as early childhood intervention practices with their children (Halgunseth, 2009;Korfmacher et al., 2008;Roggman et al., 2016).
Among the many reasons parents use or do not use early childhood intervention practices with their children are their beliefs about the importance and acceptability of the practices and the intended child outcomes of the practices (e.g., Dunst, Trivette, Prior, Hamby, & Embler, 2013b;Reimers & Wacker, 1988).These types of beliefs or subjective judgments have been described as social validity appraisals (Foster & Mash, 1999).As noted by Strain, Barton, and Dunlap (2012), an intervention practice is not likely to be used by a parent (or a practitioner) if it is not considered worth the time and effort to use.This was demonstrated in a study by Dunst, Raab, and Hamby (2016) where parents' social validity appraisals of interest-based child language learning practices were related to the fidelity of use of the practices, where fidelity of use of the practices in turn was related to improvements in the children's language development.
The study described in this paper is part of a line of research and practice investigating parents' and practitioners' judgments and appraisals of early childhood intervention practice guides where end-user suggestions and feedback were used to improve the design, organization, and content of the intervention materials (e.g., Dunst, 2017;Dunst & Hamby, 2017;Dunst, Trivette, Prior, Hamby, & Embler, 2013a;Trivette, Dunst, Masiello, Gorman, & Hamby, 2009).This brief report includes findings from three field-tests, where results from each field-test were used to inform changes and improvements in next generation practice guides.
Parents in all three field-tests selected from among a list of practice guides to review and make social validity appraisals of the importance and acceptability of both the practice guide intervention activities and child outcomes.The parents also made judgments of the attractiveness and appeal of the practice guide design.The progressive changes and improvements in the practice guides made in response to parent feedback were expected to be related to increases in parents' judgments of the importance, acceptability, and design features of the intervention materials.Exploratory structural equation modeling was used to trace the pathways of influence from improvements to the practice guides to the judgments of the attractiveness and appeal of the interventions materials to parents' social validity appraisals of the practice guide intervention activities and then to the social validity appraisals of the practice guide child outcomes.

Participants
The participants were 173 parents and other primary caregivers of infants, toddlers, and preschoolers involved in early childhood intervention programs throughout the United States.The parents were recruited through parent organizations and early childhood programs and by announcements on parent organization websites and listserves.
The participants' children were receiving early childhood intervention because of identified disabilities, developmental delays, or because they were at-risk for poor developmental outcomes for medical (e.g., low birth weight) or socio-environmental (lower family socioeconomic status) factors.Forty-two percent of the children were involved in birth to age three early intervention or Early Head Start programs, and 58% of the children were involved in age three to five year old early childhood-special education or Head Start programs.

Practice Guides
The practice guides are formatted in similar ways.Each practice guide includes a description of an intervention practice and the intended benefits or outcomes of the practice, a description or list of activities (methods, strategies, etc.) for a parent to use a practice with his or her child, a vignette of parents using a practice with their children, and a list of three outcome indicators for determining if the practice guide activities had intended child benefits.Lessons learned from the first field-test informed changes and improvements in the practice guides in the second field-test, and lessons learned from both the first and second field-tests informed changes and improvements in the practice guides in the third field-test.The latter involved captioned videos of parents using the practices with their children and an external link to other resources for using the practice to promote child learning and development.Figure 1 shows one of the practice guides evaluated by parents in the third field-test.

Procedure
Invitations to participate in the field-tests were posted on parent and early child intervention organization websites, on parent organization listserves, and parent organization web-based newsletters.Requests were also sent to early childhood program directors who were asked to distribute the invitations to interested parents.
The invitations included a description of the purpose of the field-tests and instructions for choosing and reviewing a practice guide.The respondents were asked to read the practice guide with a specific focus on the practice guide activities and intended child outcomes.The invitation also included a web-based link to the field-test survey for respondents to evaluate the practice guides.
The survey included eight parent appraisal items, space for making comments or suggestions for improving the practice guides, a question about the age of the parent's child, and a question about their child's early childhood intervention program.The survey included items measuring parents' appraisals of the practice guide design, practice guide intervention activities, and practice guide child outcomes.The parent appraisal items included four items measuring the social validity of the practice guide intervention activities (e.g., "The practice would easily fit into my everyday schedule"; "The practice would be worth my time and effort to use"), three social validity items measuring the intended child outcomes of the practice (e.g., "The practice would help my child learn"; "The practice guide activities would be interesting to my child"), and one program design item ("The practice guide format is both attractive and appealing").The eight items were each rated on a 5-point scale ranging from Do Not Agree at All to Agree a Great Deal with the survey statements.
Figure 1.Example of a practice guide that was the focus of parents' evaluative judgments and appraisals

Methods of Analysis
Three between field-test ANOVAs with a priori tests for linear trends were used to determine if there were progressive increases in both the parents' practice guide design ratings and social validity appraisals of the practice guide intervention activities and child outcomes.The dependent variables were the average respondent ratings for the three sets of items so that the scores for all three measures ranged between 1 and 5. Cohen's d effect sizes for the linear trends were the primary metrics for substantive interpretation of the results since effect sizes rather than p-values provide the best estimate of the magnitude of improvements to the practice guide design and content (Coe, 2002).
The fit of the hypothesized model to the pattern of relationships among the field-test variables in the exploratory structural equation model were evaluated by the Root Mean Square Error of Approximation (RMSEA), Root Mean Square Residual (RMSR), Comparative Fit Index (CFI), and Incremental Fit Index (IFI).The closer RMSEA and RMSR are to zero, and the closer CFI and IFI are to one, the better the fit of the model to the data.The standardized structural (path) coefficients were used to evaluate the direct and indirect effects of the variables in the model.These can range from -1.00 to 1.00 when the size of effect is a measure of the strength of relationships among the variables in the model.

Between Field-Test Comparisons
The mean ratings for the practice guide design judgments and social validity appraisals are shown in Figure 2.There were between field-test differences in the mean scores for the practice guide design, F(2, 171) = 8.16, p = .0004.There was also a linear increase in the parents' judgment of the practice guide design, F(1, 171), p = .0000,Cohen's d = 1.71.Follow-up effect size calculations showed that the linear increases were primarily for the Field-Test 1 vs. Field-Test 2 and the Field-Test 1 vs. Field-Test 3 differences.The Cohen's d effect sizes for these two field-test comparisons were d = .64and d = .80,respectively, for the parents' judgments of the practice guide design, whereas the effect size for Field-Test 2 vs. Field-Test 3 was d = .21.There were no between field-test differences for the parents' social validity appraisals of the practice guide child outcomes, F(2, 171) = 0.68, p = .5090.There was, however, a small effect size for the linear change in the mean scores, F(1, 170) = 1.31, p = .2540,d = .48.

Correlational Analyses
Table 1 shows the correlations between the field-test study measures.The patterns of relationships were as expected.Improvements to the practice guides were correlated with both the practice guide design judgments and social validity appraisals of the practice guide intervention activities but not the child outcomes.Practice guide design judgments were correlated with both social validity appraisals, and the social validity appraisals of the practice guide intervention activities were correlated with social validity appraisals of the child outcomes.

Structural Equation Modeling Results
Figure 3 shows the results from the structural equation modeling analysis.RMSEA was .03,RMSR was .02,CFI was .99,and IFI was .99.These results indicate a good fit of the model to the relationships among the variables in the model (Table 1).
Progressive changes to the practice guides were directly related to parents' judgment of the practice guide designs, and indirectly related to parents' social validity appraisals of the practice guide intervention activities, mediated by practice guide design judgments, β = .27x .57= .15,p = .000.The progressive changes to the practice guides were also indirectly related to the social validity appraisals of the practice guide child outcomes mediated by both practice guide design judgments and practice guide intervention activity social validity appraisals, β = .14,p = .0035.Parents' practice guide design judgments were directly related to parents' social validity appraisals of both the practice guide intervention activities and child outcomes, although the size of effect for the relationship with the intervention activities was four times larger than those for the child outcomes.The parents' practice guide design judgments were also indirectly related to the parents' social validity appraisals of the practice guide child outcomes mediated by the social validity appraisals of the practice guide intervention activities, β = .57x .60 = .34,p = .0000.The parents' social validity appraisals of the practice guide intervention activities were directly related to their social validity appraisals of the practice guide child outcomes.

Discussion
Findings from the between field-test comparisons showed that parents' product design judgments and social validity appraisals of the practice guide intervention activities and child outcomes increased as a function of improvements to the intervention materials.Results from the structural equation modeling analyses showed that there was an adequate fit of the hypothesized model to the pattern of relationships among the field-test variables.
The two sets of findings, taken together, indicate that parent-informed improvements in early childhood intervention materials in general, and the parent practice guides specifically, can enhance the usability, acceptability, and importance of intervention products, materials, and practices.
As noted earlier, the field-tests described in this paper are part of a line of research and practice investigating (a) improvements in both parent and practitioner early childhood intervention practices (Dunst, 2017;Dunst, Pace, & Hamby, 2007;Dunst, Trivette, Gorman, & Hamby, 2010), (b) the relationship between end-user appraisals and judgments of the practices and fidelity of use of the practices (e.g., Dunst & Hamby, 2015;Dunst, Trivette, & Raab, 2014), and (c) the effects of fidelity of use of early childhood intervention practices on child outcomes (Dunst et al., 2016).This study adds to this knowledge base by demonstrating that end-user appraisals and judgments of intervention practices are related in discernible ways.The structural equation modeling results, together with those found in other studies (e.g., Dunst et al., 2016;Dunst et al., 2014;Trivette, Raab, & Dunst, 2014), also add to the knowledge base in terms of an understanding of the manner in which different types of personal beliefs influence judgments of materials and products (e.g., Dunst & Hamby, 2017;Seva et al., 2010).
Enhancing the usability of intervention materials (Santos, Fowler, Corso, & Bruns, 2000) and the acceptance and importance of intervention practices (Strain et al., 2012) have been "called for" to improve parents' and practitioners' adoption and use of early childhood intervention practices with infants, toddlers, and preschoolers.Santos et al. (2000), for example, noted that "we need to find ways to enhance the usability of [intervention] materials" (p.20) so as to be acceptable to a wide range of end-users.Similarly, Strain et al. (2012) noted that improving judgments of the acceptability and importance of intervention practices can facilitate increased use of the practices.Studies like the one described in this paper contribute to these two goals by involving end-users in evaluating early childhood intervention practices and materials and making end-user-informed improvements in the practices and materials.The interested reader is referred to Rice and Valdivia (1991), Santos et al. (2000), and Springston and Champion (2004) for suggestions and guidelines for designing user friendly materials.

Figure 2 .
Figure 2. Parents' mean practice guide design judgments and social validity appraisals of the practice guide intervention activities and child outcomes

Figure 3 .
Figure 3. Pathways of relationships among the four variables in the structural equation modeling analysis Note.The main pathways in the model are in bold.

Table 1 .
Correlations among the field-test variables