4.4.2 The substantive aspect

The substantive aspect of validity refers to the theoretical rationale behind tacit knowledge and its relationship to task (test) performance. A step toward the goal of substantive validity is provided by our cognitive model and the characterization of tacit knowledge presented in the sixth section. The model illustrates how tacit, procedural knowledge is acquired and how it comes to be applied in solving everyday problems. The model also helps to illustrate how tacit knowledge confers a performance advantage (relative to that conferred by nontacit, procedural knowledge) in people's skill to respond to contextualized problems of realistic complexity. The characteristic features of tacit knowledge (i.e., acquisition on one's own, procedural nature, and instrumental value) further highlight its potential contribution to successful performance. The cognitive model of tacit knowledge, on which the identification and measurement of tacit knowledge is based, provides a theoretical rationale for tacit-knowledge test performance and, as such, directly serves the goal of substantive validity. Substantive validity also may be supported by showing, through empirical research, the extent to which participants draw on personally-experienced rather than received knowledge in performing everyday, real-world tasks.

4.4.3 The structural aspect

The structural aspect of validity refers to the level of fit between the internal structure of the test and the internal structure of the construct domain. It is related to the issue of construct representativeness we discussed earlier. A first step toward the goal of structural validity is taken by interviewing and eliciting knowledge from job incumbents in all areas that represent the performance domain. For example, in our study with military leaders, we interviewed officers in all three of the major branch categories within the Army (i.e., combat arms, combat support, combat service support). The goal of structural validity also is served by administering measurement instruments (e.g., the Tacit Knowledge Survey) to a wide variety of job incumbents. By using broad samples of job incumbents, we are able to avoid basing our analyses and test development on a restricted subset of the tacit-knowledge domain. Of course, the structural aspect of validity is addressed most directly through statistical techniques like cluster analysis and multidimensional scaling that identify the internal structure of the sample of items. By examining the internal structure we cast a wider net in our selection of tacit-knowledge items, and in so doing, we have improved our prospects for developing tacit-knowledge tests that mirror the structure of the construct domain (e.g., the domain of practical, action-oriented knowledge that individuals acquire from personal experience).

4.4.4 The generalizability aspect

The generalizability aspect of validity refers to the extent to which score properties and interpretations generalize across groups, settings, and tasks. The generalizability aspect includes concerns that traditionally fall under the heading of "reliability." In the context of tacit-knowledge test development, the goal of generalizability calls for tacit-knowledge test scores that generalize across (1) roles within the organization, (2) repeated administrations, and (3) alternate forms of the test. Test development efforts relevant to the content, substantive, and structural aspects of validity also are relevant to the generalizability aspect. In general, by seeking to specify and measure the construct, rather than merely pursuing correlation with an external criterion, we presumably increase the generalizability of score interpretations for our tacit-knowledge tests.