5.5.3 Findings and conclusions

Figure 5
Item difficulty parameters for the four classes of items

Figure 5 shows the distribution of item difficulty parameters within the four classes of items. For each level, the range from lowest to highest item difficulty is shown. Half of the item difficulty parameters are within the shaded area. The straight line at the center of the box indicates the median of item difficulties for the respective level. As expected, item difficulty shows a continuous increase from level 1 to level 4. Also, despite some overlap, the four ranges of item difficulty can be clearly discriminated. In fact, 77 percent of between-item variance can be attributed to the classification scheme. From these findings we can conclude that the ALL problem-solving scale can in fact be interpreted as a scale that measures the quality of analytical reasoning and decision making. According to the complexity of the required reasoning processes, the items can be classified into four levels as described above. The pilot results support this four-level model of item complexity, because the theoretically defined levels show a clear pattern of increasing empirical item difficulty.

When it comes to assessing the problem-solving competency of the respondents, it is possible to distinguish four levels of problem-solving proficiency analogous to the four levels of item complexity/difficulty. Respondents on level 1 can be expected to cope successfully only with content-related, concrete problems. At level 2, respondents are also able to evaluate actions and make decisions with regard to well-defined criteria. At level 3, respondents can apply formal operations which are needed to order and integrate multiple or ill-defined criteria. At the highest level, level 4, respondents are also able to evaluate a system of goals or criteria as a whole, applying critical thinking and meta-cognition.