4.3 Developing everyday situational-judgment tests to measure tacit knowledge

We have developed tests to assess tacit knowledge for academic psychology, elementary-school teaching, business management, sales, entry-level jobs in organizations, college education, and military leadership. In this section we present a framework for developing tacit-knowledge tests of the format described above, a framework that is based on the techniques we have used to measure tacit knowledge in the various domains we have studied.

The development of tacit-knowledge inventories readily may be understood as a production process, beginning with the "raw materials" of experience-based tacit knowledge elicited from successful practitioners in a given domain and culminating in a revised and validated inventory. At each step in the development process, "value" is added through the conduct of research and analysis.

All of the phases are designed to support the development of assessment instruments based on (a) the theory and methods of tacit-knowledge research, and (b) the substantive knowledge in the domain of interest. Specifically, the steps are intended to aid in selecting the content that is most promising with respect to the goals of the assessment phase, that is, in measuring an individual's possession of tacit knowledge. The term promising is used here to refer to that subset of tacit knowledge with the highest probability of yielding or contributing to tacit-knowledge test questions that, taken together, constitute a valid measure of the underlying, domain-relevant tacit knowledge of respondents. This process was developed over the course of several research projects, and is applicable to the identification and assessment in tacit knowledge in any performance domain. We describe each stage in the process below, from the identification of exemplars of tacit knowledge to the construction of the final inventory.

4.3.1 Knowledge identification

We generally begin with a review of job-relevant literature (e.g., sales manuals, Army trade publications) to identify on a preliminary basis the experience-based, tacit knowledge for the relevant profession (e.g., salespersons, Army leaders). This review may suggest some of the content for use in a tacit-knowledge inventory, and may provide a preliminary taxonomy, or category framework, for organizing the knowledge. For example, in research with managers, Wagner and Sternberg (1986) proposed a framework of practically-intelligent behavior consisting of tacit knowledge about managing oneself, managing others, and managing one's career.

Typically, a review of the literature does not provide a sufficient number of examples of knowledge that meet our criteria or include enough detail from which to create tacit-knowledge questions of the format described above. We have found that the practical advice presented in the professional literature tends to be decontextualized and already converted to semantic knowledge. We also surmise that the politics of professional print may keep some truly tacit knowledge—knowledge that contradicts doctrine, for example—out of print altogether. Therefore, the next step is to conduct interviews with successful practitioners in the domain to generate a larger body of knowledge from which to draw in developing the tacit-knowledge inventories. We described here a method for conducting these interviews.