Since assessing problem solving skills in large-scale assessments is a relatively new endeavour, it might be helpful to provide a detailed account of the construction process. Table A1 provides an overview of the problem solving steps as they correspond to the action steps identified above. Different components and aspects of each of the problem solving steps are listed.

TABLE B1

Problem-solving steps and instructions
Define the goals
  • Set goals.
  • Recognize which goals are to be reached and specify the essential reasons for the decision.
  • Recognize which goals/wishes are contradictory and which are compatible.
  • Assign priorities to goals/wishes.
Analyze the situation
  • Select, obtain and evaluate information.
    • What information is required, what is already available, what is still missing, and what is superfluous?
    • Where and how can you obtain the information?
    • How should you interpret the information?
  • Identify the people (e.g. with what knowledge and skills) who are to be involved in solving the problem.
  • Select the tools to be used.
  • Recognize conditions (e.g. time restrictions) that need to be taken into account.
Plan the solution
  • Recognize which steps need to be taken.
  • Decide on the sequence of steps (e.g. items on the agenda).
  • Coordinate work and deadlines.
  • Make a comparative analysis of alternative plans (recognize which plan is suitable for reaching the goals).
  • Adapt the plan to changed conditions.
  • Opt for a plan.
Execute the plan
  • Carry out the individual steps (e.g., write a letter, fill in a form, make calculations).
Evaluate the results
  • Assess whether and to what extent the target has been reached.
  • Recognize mistakes.
  • Identify reasons for mistakes.
  • Assess consequences of mistakes.

The construction of a pool of assessment tasks that could be mapped back to these five action steps involved several phases of activities. First was the identification of appropriate projects that would be suitable for adults with varying educational backgrounds and relevant to the greatest number of people in the target group. Next, developers had to identify and sketch out the problem situation and the sequence of action steps that relate back to the model. Third, they had to develop a pool of items that were consistent with the action steps and that tapped into particular processes including the development of correct responses and appropriate distractors for multiple choice items and solution keys and scoring guides for open-ended tasks.