4.4.1 The content aspect

The content aspect of validity refers to evidence that test content is relevant to and representative of the focal construct. It addresses the concerns that fall under the traditional heading of content validity. In the context of tacit-knowledge test development, the goal of construct relevance calls for tacit-knowledge test questions that are sensitive to knowledge of the type specified by the focal construct and insensitive to knowledge that falls outside the focal construct. A first step toward this goal is taken during the identification phase of test development, in interviews with job incumbents, when we orient participants toward personal experiences and away from formal principles or theory within their performance domains. A second step is taken in the item-selection phase when incumbents are asked to rate the quality of tacit-knowledge items. These ratings (i.e., item means and variances) may provide evidence regarding the relevance of tacit-knowledge items to the underlying construct. For example, tacit-knowledge items with low mean ratings (i.e., when respondents, on average, consider the knowledge represented in the item to be bad advice) may not be relevant to successful performance. And items with low variances (i.e., when respondents agree highly about the quality— good or bad—of the knowledge reflected in the item) may not reflect knowledge gained through personal experience if the knowledge is generally agreed upon as good. In addition to these steps, the goal of establishing construct relevance also is supported by asking domain experts, at various stages in the test development process, to judge the relevance of the items to the tacit-knowledge construct.

The goal of construct representativeness calls for tacit-knowledge items that are typical rather than atypical of knowledge-based items specified by the focal construct. An initial step toward this goal is taken in the identification phase by interviewing job incumbents that are representative of the range of specialty areas within the domain. For example, military leaders in the same position (e.g., platoon leader) may serve in one of many branches (e.g., infantry, engineering). Therefore, in our research, we sought to interview officers from these various branches to increase the representativeness of the knowledge that was elicited. A second step is taken during the item-selection phase, when participants are asked to rate how "often" a situation presented in a tacit-knowledge item occurs. Items that receive both a low mean and small variance, for example, are ones that most incumbents agree occur almost never, and therefore may not be representative of the knowledge domain. The categories derived from cluster analyses of the tacit-knowledge items also provide a source for ensuring construct representativeness. Items can be chosen to represent each of the major categories of tacit knowledge, thus providing better coverage of the construct domain. Finally, at several points during test development, expert judgments are sought regarding the construct representativeness of the items. After an initial pool of potential tacit-knowledge items is obtained from the interviews, an expert panel is asked to judge the representativeness of each item. The experts are asked to eliminate items that are too narrow or technical in focus (e.g., how to safely store chemical weapons) and knowledge that is relevant to a small proportion of job incumbents (e.g., how to manage stress at work if you are a single mom). Experts again are asked to evaluate the representativeness of the items after preliminary test questions have been developed.