The Arbitrary Choice of the Predictor in Meritocratic Selection to Higher Education Affects the Selection Outcomes

Predictive validity considerations in selection dictate choice of the predictor with the highest predictive validity. Implementation of this principle in any specific selection process inevitably entails choice between imperfectly correlated alternative predictors, real or hypothetical, which are equivalent in terms of predictive validity. We show that use of different predictors necessarily results in the admission of partially overlapping groups of candidates. The amount of non-overlap is inversely related to the correlation between predictors and to the selection ratio. For typical values of these factors, the amount of non-overlap is considerable (50%-80%). Thus, the predictor’s identity affects the identity and characteristics of the admitted candidates independently of its validity and can be intentionally used to bias the selection outcomes in a desired direction.


Introduction
Selection to institutions of higher education is frequently motivated by the fact that the number of otherwise acceptable candidates N exceeds the number of available places n.Hence, only n candidates can be admitted (fixed-n selection; Cahan & Gamliel, 2006).The meritocratic principle dictates top-down admission of the 'best' n candidates in terms of performance on the criterion of merit Y (e.g., grade point average at the time of graduation or job performance).However, because Y is unknown at the time of selection, top-down selection is actually performed on an imperfect predictor X (where 0 < Corr(X, Y) < 1) (e.g., achievement or ability test scores such as the Scholastic Aptitude Test (SAT) or the General Aptitude Test Battery (GATB)). 1 Choosing between alternative predictors is, therefore, a necessary and possibly consequential component of predictor-based meritocratic selection.
The treatment of this issue in the literature has been dominated by the paramount interest in maximizing predictive validity.Assuming equal availability and cost, when choosing between alternative predictors, predictive validity considerations unambiguously dictate choice of the predictor that shows the largest correlation with the criterion.In contrast, the arbitrary choice between alternative, imperfectly correlated predictors which are equivalent in terms of predictive validity has been by and large ignored (De Corte, Sacket, & Levens, 2010) 2 , despite its far-reaching consequences.This paper focuses on the examination of these consequences.The first section presents the connection between the choice of the predictor and the identity of the admitted candidates, with special emphasis on the resulting partial overlap between the candidates admitted on the basis of alternative predictors.The second section estimates, by means of empirical simulations, the extent of overlap between the admitted candidates on the basis of functionally equivalent alternative predictors, as a function of (1) the correlation between predictors and (2) the selection ratio.The third and last section discusses the implications of our results as well as ways to reduce the undesirable consequences of the inevitable arbitrary choice of the predictor in selection to educational institutions.

The Arbitrary Choice of the Predictor and the Identity of the Selected Candidates
A major implication of imperfect correlations between alternative predictors, equivalent in terms of predictive validity, is the partial overlap between the n candidates admitted on the basis of each predictor: some of the candidates admitted on the basis of one predictor are rejected if selection is based on another predictor, and vice-versa.Hence, the choice of a particular predictor in these cases is consequential, since this choice determines to a significant degree the identity of the accepted candidates.For example, suppose that there are only two alternative predictors, which have equal correlations with the criterion in a specific selection situation: a mathematics test (X1) and a vocabulary test (X2).Selection based on the mathematics test scores will lead to the admission of the n highest math scoring candidates, whereas selection on the basis of the vocabulary test scores will lead to the selection of the n highest vocabulary scoring candidates.The overlap between the two groups for any given selection ratio depends on the extent to which the best mathematicians in the candidate population also have the richest vocabulary, that is, on the correlation between scores on the two tests (Corr(X1, X2)).
Thus, the predictor's identityand not only its predictive powermatters, because it independently affects the outcome of the selection process.The effect of the arbitrary choice of a particular predictor out of a set of functionally equivalent predictors on the identity of the admitted candidates is therefore of major interest in selection situations.Particularly, in light of the fact that this effect can be intentionally used to manipulate the selection outcomes and bias them in a desired direction (e.g., favouring 'mathematicians' vs. candidates with the richest vocabularies or vice versa in the above example).
Yet this implication of the arbitrary choice of the predictor in selection to educational institutions has received little if any attention in the selection research literature (De Corte et al., 2010).In particular, there is no information regarding the extent of overlapand non-overlapbetween the admitted candidates on the basis of functionally equivalent alternative predictors.Relying on empirical simulations, the next section fills this lacuna.

Estimating the Extent of Overlap between the Candidates Accepted on the Basis of Imperfectly Correlated Alternative Predictors
For the sake of simplicity, and without loss of generality, we assume that the decision maker has to choose between only two predictors, X1 and X2, identical in terms of predictive validity, that can be alternatively used for selection of candidates (e.g., two cognitive tests) with Corr(X1, X2)= ρ .By the definition of fixed-n selection, a certain (fixed) number n of the highest scoring candidates are selected, whether selection is based on X1 or X2.Nevertheless, according to our argument, unless the correlation between predictors is perfect (i.e., 1), the group of n candidates accepted on the basis of X1 (group A1) does not entirely overlap the group of n candidates accepted on the basis of X2 (group A2).Some of the candidates in A1 are not included in A2 and vice-versa.One measure of the amount of overlap between A1 and A2 is the share of those accepted by the second predictor among those accepted by the first predictor (or vice-versa), that is, the conditional probability P of those accepted by X2 given that they were accepted by X1.The probability P depends on the correlation between the two predictors in the candidate population and on the selection ratio (SR=n/N as seen in the formulas in Appendix A). Figure 1 presents P for different values of the correlation between the two predictors and several selection ratios, assuming that X1, X2 have a bivariate normal distribution.Appendix B gives the corresponding exact conditional probability values.As illustrated by Figure 1, the amount of overlap between the candidates accepted on the basis of each of the two predictors is larger the higher the correlation between predictors and the higher the selection ratio.For typical values of the correlation between predictors (0.30-0.50) and selection ratios (0.10-0.30),P ranges from about 20% to 50%.In other words, given our assumptions, 50% -80% of the candidates accepted on the basis of one predictor are rejected on the basis of the other predictor.Needless to say, the amount of non-overlap increases the greater the number of alternative predictors.Thus, for example, it is seen in Figure 2 that if there are three predictors, the share of those accepted by two predictors among those accepted by the third predictor is 7-34% for the above ranges of ρ and SR (see exact probabilities in Appendix C).
Figure 2. Share of candidates accepted by three predictors among those accepted by a single predictor (conditional probability) for different values of the correlation between each pair of predictors and chosen selection ratios (SRs)

Discussion
The moral of this paper is unambiguously clear: Despite its apparent objectivity, fixed-n selection based on the meritocratic principle of selecting the highest-scoring candidates on the 'best' predictor (in terms of predictive validity) inevitably involves --explicitly or implicitly--arbitrary choice between alternative, functionally equivalent predictors, real or hypothetical.As demonstrated by this work, this choice results in the admission of partially overlapping groups of candidates on the basis of different predictors: Some of the candidates admitted on the basis of one predictor are rejected on the basis of other predictors and vice-versa.The lower the correlation between predictors and the smaller the selection ratio, the greater the amount of non-overlap.As we have shown, for typical values of these factors, the amount of non-overlap is considerable (50%-80%), despite the equivalence between predictors in terms of predictive validity and the associated equality between the respective candidate groups admitted in terms of mean criterion score.Even if the correlation between predictors is as high as 0.70, for selection ratios lower than 20%, the majority of the candidates admitted on the basis of one predictor will be rejected on the basis of the other predictor and vice-versa (Fig. 1).
Thus, the predictor identity matters: The identity of the admitted candidates in predictor-based meritocratic selection is not only dictated by objective predictive validity considerations but rather significantly affected also by the arbitrary choice among objectively equivalent predictors (real or hypothetical).This arbitrary element constitutes an integral and consequential component of predictor-based meritocratic selection.The resultant variability of the selection outcomes corresponding to the use of different predictors introduces an element of chance or luck in the apparently objective selection process and affects its fairness or equity.Recognition of this fact calls for significant modification of our conceptualization of predictor-based, meritocratic selection as a fully objective, 'deductive' decision making process and focuses attention on its arbitrary aspects and their implications for the selection outcomes.
At the same time, recognition of the arbitrary component of predictor-based meritocratic selection should also affect our conception of the admitted candidates on the basis of a given predictor.In view of the dependence of the identity of these candidates on the identity of the particular predictor, they cannot be unconditionally and unqualifiedly considered anymore as the 'best' available candidates.Rather, they can only be considered as one of many possible, partially overlapping 'criterion-equivalent' sets of 'best' candidates.Basing selection on an alternative, equally valid, predictor would have led to the admission of a markedly different, however criterion-equivalent candidate set.The very existence of this arbitrary aspect of predictor-based meritocratic selection as well as its implications should be publicly acknowledged and explained to all the parties involved: Institutional decision makers, candidates and the general public.
The dependence of the identity of the admitted candidates on the identity of the arbitrarily chosen predictor is particularly important in view of its likely social and political consequences.Because criterion-equivalent predictors can still have different correlations with other candidate characteristics (e.g., demographic or psychological characteristics), admission of different candidates also means admission of candidates who differ in terms of a multitude of other characteristics, in addition to their different rankings on different predictors.The well-known between-predictor variability in the amount of adverse impactthat is, selection of different proportions of minority and majority candidatesis a case in point.Awareness to this variability led to recognition of the fact that the amount of adverse impact induced by a predictor is an important normative considerationin addition to predictive validityin predictor selection (e.g., De Corte et al., 2010;Druart & De Corte, 2012;Outtz, 2010;Pyburn, Ployhart, & Kravitz, 2008;Schmitt, Rogers, Chan, Sheppard, & Jennings, 1997;Wee, Newman, & Joseph, 2014).As conventional cognitive tests (e.g., SAT and GATB) have revealed relatively high adverse impact, attempts were made to devise alternative selection measures that would result in similar predictive validity and smaller adverse impact, such as situational tests (Lievens & Coetsier, 2002) and work samples (Bobko, Roth, & Buster, 2005).More recent developments in the field (Druart & De Corte, 2012) allow for between-predictor variability in both predictive validity and adverse impact and include analysis of the associated trade-offs.The resulting validity versus adverse impact dilemma is now widely recognized as one of the most perplexing problem facing the practice of personnel selection and selection to higher educational institutions today (e.g., Campion et al., 2001;Outtz, 2010;Wee et al., 2014).
Of course, adverse impact is but a particular instance of a much broader phenomenon, namely the implications for selection outcomes if alternative predictors have different correlations with other characteristics of the candidates (Z) (e.g., cooperativeness).Suppose, again, that there are only two alternative predictors, which have equal correlations with the criterion in a specific selection situation: a mathematics test and a vocabulary test.If the correlation between Z and the mathematics test ( 1  ) is higher than the correlation between Z and the vocabulary test ( ρ 2 ), basing selection on the former will result in a higher probability of admission among high Z scoring candidates than basing selection on the latter.Consequently, in this case the accepted candidates will be more cooperative, on the average, than if selection was based on the vocabulary test.This effect becomes more dramatic, the higher the difference between the two correlation values and will be particularly strong if the correlations of Z with the two predictors have opposite signs; for example, if cooperativeness correlates positively with vocabulary and negatively with math.
Thus, the arbitrary choice of a particular predictor out of a set of alternative predictors, equivalent in terms of predictive power, is consequential, since this choice unintentionally determines to a significant degree the demographic and psychological characteristics of the accepted candidates.Adverse impact is but a particular instance of this issue, characterized by the fact that Z is a salient and socially and politically sensitive candidate characteristic, such as race or gender (i.e., a protected status).Particularly noteworthy in this context is the possibility, in fact plausibility, of purposeful choice of a particular predictor, out of a set of functionally-equivalent predictors, in order to bias selection decisions in the desired direction.Depending on the underlying motivations and one's value system, such purposeful manipulation may be considered to be desirable or harmful to the selection process's equity.
The arbitrary choice of the predictor is inherent to predictor-based, meritocratic selection and cannot be eliminated from the selection process.By definition, each predictor-based selection must rely on a specific predictorbe it a single characteristic or a complex combination of several characteristicsand not rely on all the other alternative predictors, some of which may be equivalent in terms of predictive validity.While changing the predictor's identity will result in the admission of mostly different candidates, it will not eliminate the arbitrary component from the selection process and its effect on the selection outcomes.Thus, the dependence of the identity of the admitted candidates on the particular predictor's identity is an unavoidable, 'chronic' and endemic feature of any particular instance of predictor-based selection (e.g., the selection of candidates to a given medical school).
The biasing effect of this feature, whether intended or unintended, is particularly strong and consequential if: (1) Candidates apply to multiple departments in the same educational institution or to the same department in different institutions (both of which are fairly common); (2) The same predictor (e.g., SAT scores) is used for selection to multiple departments in the same educational institution or for selection to the same department (e.g., medical school) across institutions, state-or nation-wide, both of which are frequently practiced.Common to these cases is the carrying-over effect of the bias associated with the use of the particular predictor across departments in the same institution or across institutions for the same department.Specifically, the use of the same predictor for selection to different departments within the same institution inevitably results in an artificially inflated correlation between the candidates' chances to be admitted to essentially different departments (e.g., literature and physics or technology): Candidates who are rejected by one department, due to their relatively low predictor scores, are likely to be rejected by most of the other departments to which they apply in the same institution (and vice-versa for high scoring candidates on the predictor).Similarly, candidates that have been rejected by the department to which they have applied in one institution are likely to be also rejected by the same department in other institutions using the same predictor.
Fortunately, however, adoption of a macro, cross-departmental and cross-institutional selection policy enables deliberate use of the bias effect of predictor choice to remove bias from the overall selection process and improve its equity without harming its validity.This can be accomplished via planned manipulation of the chance component inherent in selection processes, through the use of different, functionally equivalent predictors for selection to different departments or programs in the same institution or to the same department in different institutions.For each department in the same institution, as well as for all institutions with respect to the same department, implementation of this strategy ensures the acceptance of one of the 'best' available subgroups of candidates.
At the same time, the implementation of this strategy helps counterbalancing, from a macro, overall, cross-departmental and cross-institutional perspective, the biasing effect of the particular predictor used in each department or institution.This effect is not carried over across departments or institutions, thereby attenuating the predictor-induced interdependency between the outcomes of selection to different departments or institutions.On the contrary, the biasing effect of one predictor is compensated for by the differently biased effects of alternative predictors.Using our familiar example, implementation of this approach may allow a high math-scoring candidate that has been rejected by the medical school in one institution on the basis of her relatively low vocabulary score to be admitted by the medical school of another institution, where selection is based on the mathematics test.Similarly, use of different, functionally equivalent predictors for selection to different programs in the same institution may allow a candidate that has been rejected by one department in a given institution on the basis of her math score to be admitted by another department of the same institution, where selection is based on the vocabulary score.
In both cases, from the macro perspective, a fringe benefit of the adoption of this policy is the diversification of the population of the candidates admitted to different departments or institutions, while maintaining the average 'quality' of the accepted candidates.Furthermore, in view of the expected variability of the predictive validity of the same predictor between the departments of the same institution, as well as between-institutions, with respect to the same program, use of different predictors may even improve predictive validityas well as the associated average quality of the accepted candidatesin each particular selection context, by allowing for the use of the department/institution 'best' predictor.

Figure 1 .
Figure 1.Share of candidates accepted by both predictors among those accepted by a single predictor (conditional probability) for different values of the correlation between the two predictors and chosen selection ratios (SRs)