We are interested in items that are (1) rated as better advice by those considered to be successful in their domain; (2) not viewed as common knowledge by individuals in the domain; (3) representative of the situations faced by most individuals in the domain; and (4) a good fit to the concept of performance held by successful individuals in the domain. In order to identify items that are endorsed by individuals who are successful in a domain, we obtain data on a relevant performance criterion. In our research with military leaders, we obtained two criterion measures—experience and performance ratings. Experience was expressed in terms of expert-novice differences and performance was assessed using ratings of leadership effectiveness by other leaders. Responses to the TKS are analyzed along with the criterion measure to identify items that have promise for inclusion in the tacit-knowledge inventory. This analysis generates a number of item statistics that can be used in the selection process.

In our research, we used discriminant analysis to identify items that distinguish individuals with more from those with less experience (see Hedlund et al., 1999). In the discriminant analysis, a linear combination of the discriminating variables (e.g., TKS items) is derived that maximizes the divergence between groups (e.g., experienced/ novice). The linear combination of the discriminating variables (the canonical discriminant function or CDF) can be tested for significance to determine if the set of variables distinguishes between groups. In addition, the correlations between discriminating variables and the CDF can be computed to assess the discriminating power of individual variables (e.g., TKS items).

We used point-biserial correlations between ratings on the items and ratings of effective performance to identify items that reflected the responses of effective performers. Item statistics such as these can be used, along with the category framework developed in the interview phase, to select items that have the most potential to explain successful performance and provide the best "coverage" of the tacit-knowledge domain.

4.3.3 Instrument construction

The "knowledge identification" and "item selection" phases generate several outputs that serve as materials for the final phase of "instrument construction." These outputs include: (a) interview transcripts and interview summaries, (b) the category framework derived from expert sortings and cluster analyses, (c) a set of item statistics for use in the selection of content for the inventories, and (d) the knowledge items retained on the basis of the category framework and item statistics from the questionnaire study. In the next phase of test development, preliminary inventory questions are constructed, using both selected knowledge items and the interview summaries from which they were drawn. A tacit-knowledge question consists of a situation description followed by several potential responses to that situation. Although the condensed tacit-knowledge item may serve to describe the situation, it is preferable to include the details from the original story to provide a richer, more in-depth problem description. Including more contextual and situation-specific information in the question provides the respondent with a clearer basis on which to evaluate the appropriateness of potential responses to the situation. The original story also provides a source for developing the response options to a question.