4.3.2 Item selection

Although one may proceed to develop test questions directly from the tacit-knowledge items generated from the interviews, a further selection process may be necessary for a number of reasons. First, the interview study may yield too many items of tacit knowledge to include in a tacit-knowledge inventory of reasonable length, depending on the context in which the test might be used. Second, we cannot determine on the basis of the interviews alone what tacit knowledge is diagnostic of experience or predictive of effective performance in a given domain, or alternatively, what tacit knowledge is not related to these criteria. A manager, for example, may have learned that subordinates are more likely to come to her with problems if she leaves her door open. But the extent to which this practice contributes to her success is unclear. By leaving her door open she may become the repository for problems that are the responsibility of other managers, which may create a distraction for her from her job. Third, the results of the preliminary sorting of interview data may not be sufficient for determining the internal structure of the tacit-knowledge construct domain. That is, for the purposes of test construction, we would want further evidence of the structure of the performance domain to ensure the representativeness of our items. For the reasons above, we take an additional step to narrow down the pool of items from which test questions will be constructed.

The next step in the process of selecting items for instrument development is more quantitative than qualitative. It entails surveying job incumbents to assess the "quality" of each tacit-knowledge item. In order to develop a questionnaire that can be administered to job incumbents, the tacit-knowledge items may need to be condensed. For example, if we want professionals to evaluate 100 examples of tacit knowledge, it would be unreasonable to ask them to read 100 items in a very long and complex format. Therefore, it may become necessary to condense the items into briefer descriptions. Condensing the items involves extracting only the most important information and deleting unnecessary information. Attempts should be made to increase the comprehensibility of the items for the intended audience and to preserve the intent of the interviewee who provided the knowledge. The procedural structure that we consider to be characteristic of tacit knowledge is maintained in the rewriting of items.

The condensed items are compiled into a survey, which we refer to as a Tacit Knowledge Survey (TKS), which is in turn a situational-judgment test. A TKS differs from a tacit-knowledge inventory in that respondents are asked to rate the perceived quality of the tacit knowledge in the former, whereas they are asked to rate the quality of responses to the problem in the latter. Job incumbents can be asked to rate each item on a number of dimensions. We have used four seven-point scales that ask for the following judgments: (1) how good does the respondent think the advice is, (2) how commonly known does the respondent think the advice is, (3) how often, in the judgment of the respondent, do incumbents at the specified level face situations such as the one described, and (4) to what extent does the advice match the respondent's personal concept of job performance? Each of the scales is intended to provide a different sort of information about the tacit-knowledge item being rated. The "good" scale is intended to assess the overall quality of the knowledge being rated. The "known" scale is intended to assess one possible index of tacitness (i.e., on the theory that knowledge whose acquisition is not well supported by the environment may be less commonly known than other knowledge). The "often" scale is intended to assess the generalizability or applicability of knowledge items across job settings within the domain. Finally, the "concept" scale is intended to assess respondents' implicit theories of performance. Together, the four rating scales are intended to provide a comprehensive but non-redundant picture of each tacit-knowledge item for the purpose of evaluating each item's potential for development into tacit-knowledge test questions.