Figure 4
Item difficulty distributions for selected vs. eliminated items

Based on the results from Canada, Italy, Norway, and Switzerland, it can be shown that the selected items perform in a similar way in all countries. Some very few country by item interactions were identified, but in all cases the reasons could be traced back to layout, translation or adaptation mistakes. Also, the selected items make up short versions of projects with a realistic and suitable scenario, i.e. the items fit together well and the projects are plausible as a whole. Thus, it can be expected that the final version of the ALL problem-solving scale will show excellent psychometric properties in the main study and allow for a well-founded discrimination between the levels of analytical problem-solving proficiency. An attempt to identify those levels based on the selected items is presented in the next section.

5.5 Proficiency levels

5.5.1 Criteria and expectations

In chapter 2, four levels of analytical problem solving were defined. From a theoretical point of view, a scale covering the following competency levels was expected:

1. Content related reasoning
2. Evaluating
3. Ordering/Integrating
4. Critical Thinking