ALL included a total of 4 projects involving 20 tasks in the assessment of problem solving. These resulted in 19 scorable items than ranged from 199 to 394 along the scale and, like the literacy and numeracy tasks, their placement was determined by the patterns of right and wrong responses among adults in participating countries. Rather than release one of the four projects that were used in ALL, we will characterize the hypothesized proficiency scale for analytical problem solving that was tested using pilot data and present an example from the pilot data that was not used in the main assessment3. Similar models have been described within the frameworks of other large-scale assessments of problem-solving competencies such as the project test for Hamburg/Germany (Ebach, Klieme and Hensgen, 2000) and the PISA 2003 assessment of cross-curricular problem solving (OECD, in press).
In ALL, four levels of problem-solving proficiency are postulated:
At a very elementary level, concrete, limited tasks can be mastered by applying content-related, practical reasoning. At this level, people will use specific contentrelated schemata to solve problems.
The second level requires at least rudimentary systematical reasoning. Problems at this level are characterized by well-defined, one-dimensional goals; they ask for the evaluation of certain alternatives with regard to transparent, explicitly stated constraints. At this level, people use concrete logical operations.
At the third level of problem-solving proficiency, people will be able to use formal operations (e.g., ordering) to integrate multi-dimensional or ill-defined goals, and to cope with non-transparent or multiple dependent constraints.
At the final and highest level of competency, people are capable of grasping a system of problem states and possible solutions as a whole. Thus, the consistency of certain criteria, the dependency among multiple sequences of actions and other “meta-features” of a problem situation may be considered systematically. Also, at this stage people are able to explain how and why they arrived at a certain solution. This level of problem-solving competency requires a kind of critical thinking and a certain amount of meta-cognition.
The following example illustrates a concrete realization of a project. For this purpose a project that is not included in the final ALL instrument is introduced and one typical problem-solving task is shown. The project is about “Planning a trip and a family reunion”.
In the introductory part of the project, the respondent is given the following summary describing the scenario and overall problem:
“Imagine that you live in City A. Your relatives are scattered throughout the country and you would like to organize a family reunion. The reunion will last 1 day. You decide to meet in City B, which is centrally located and accessible to all. Since you and your relatives love hiking, you decide to plan a long hike in a state park close to City B. You have agreed to be responsible for most of the organization.”