4.4.5 The external aspect

The external aspect of validity refers to the issue of criterion-related validity. That is, we seek to establish that the test relates to an external criterion. More specifically, the goal is to obtain evidence of convergent and discriminant validity. Establishing criterion-related validity entails showing that tacit-knowledge test scores correlate more highly (i.e., converge) with theoretically related constructs (e.g., performance) and correlate less highly (i.e., diverge) with theoretically distinct constructs (e.g., general cognition, formal job knowledge).

Test-development efforts to specify and measure the tacit-knowledge construct also support the goal of criterion validity. For example, job incumbents are asked to provide examples of important lessons they learned in the course of performing their job rather than knowledge they gained in school. These instructions increase the likelihood that the tacit-knowledge items obtained will be related to performance criteria and be distinct from formally-acquired knowledge. Research during the item-selection phase involves assessing more directly the relation of these items to external criteria. This step helps to identify tacit-knowledge items that are indicative of successful performance.

Beyond these efforts during test development, additional steps should be taken to provide evidence of convergent and discriminant validity. For tacit-knowledge tests, possible discriminant evidence would be that which discounts the effects of general cognition, reading comprehension, and formally-acquired knowledge on tacit-knowledge test scores. Evidence of convergent validity would include a correlation between tacit-knowledge test scores and variables such as perceived job effectiveness, degree and rate of career advancement, and performance on job-relevant tasks. To obtain such evidence requires conducting a validation study in which measures of these variables are administered to or obtained from individuals. For example, in our research with managers and military leaders, we administered the tacit-knowledge inventory along with a measure of general cognition and related constructs, and obtained various performance criteria, such as supervisor ratings, salary and productivity. Correlational and hierarchical regression analyses can be used to assess convergent and discriminant validity. Convergent validity is supported by a significant relationship between tacit-knowledge test scores and the performance criterion (e.g., supervisor ratings). Discriminant validity is supported by zero to moderate correlations with measures such as general cognition and general job knowledge, as well as the incremental validity of tacit-knowledge test scores beyond these measures.

4.4.6 The consequential aspect

The consequential aspect of validity refers to the value implications of the intended use of score interpretation as a basis for action. Because tacit-knowledge tests may be used for employee assessment and development, or even selection, it is important to consider how the knowledge included in those tests fits into the culture and rules of the organization. For example, if an item of tacit knowledge meets all the criteria discussed above (e.g., satisfies the definition of tacit, exhibits a strong positive correlation with effective performance), but it conflicts with the organizational culture (e.g., suggesting that females should be given less responsibility than males) or it involves disobeying a regulation (e.g., suggesting that financial figures should be fudged when information is unavailable), then it may be inappropriate to include the item in a tacit-knowledge test. Relying on experts to review the tacit-knowledge items throughout the test-development process helps to ensure that issues related to the consequential aspect of validity are addressed.