The construction of the problem-solving assessment instrument involved developing a pool of projects with different real-life contexts and a variety of item formats. Experts from several countries provided input for the development of different projects. Based on this, eight projects were initially developed and the best four were selected after a feasibility study and modified for the final instrument used in the pilot study. The different phases that directed the development of the problem-solving instrument and some of the criteria that are relevant to this process are illustrated in figure 2.
Figure 2
Five phases of test construction
- is suitable for adults ages 16 to 65 and with varying educational backgrounds
- is relevant to everyday life for the greatest possible number of persons within the target group, and can generate an adequate problem context
- requires a level of general knowledge that an entry-level worker would typically have (and not more)
- is domain-independent
- is politically acceptable
- can be transferred to other cultures
- has no group-related bias (e.g. no gender bias)
- allows at least six mutually independent items to be created
- covers most of steps in the process model
After drafting the problem situation, sketch out a sequence of actions that could form a part of solving the problem. This sequence of actions should approximately follow the five steps of the process model and yield a realistic sequence that would make sense in everyday life.
- Develop a rudimentary task structure
- Aim at tapping certain processes and a certain level of difficulty
- Formulate the general problem and question
- Select adequate information to be processed (relevant, distracting, and irrelevant but contributing to the coherence of the problem context)
- Select an item format
- Develop response alternatives (correct answers and distractors)
- Create a solution key and scoring guides