Problem solving is dependent on knowledge of concepts and facts (declarative knowledge) and knowledge of rules and strategies (procedural knowledge) in a given subject domain. Although it is evident from past research that declarative knowledge in the problem domain can substantially contribute to successful problem-solving strategies, procedural knowledge is crucial as well. The amount of relevant previous knowledge available could also account for the relation between intelligence and problem-solving performance, as shown in the work of Raaheim (1988) and Leutner (1999). People with no relevant previous knowledge at all are unable to explore the problem situation or plan a solution in a systematic manner and are forced to rely on trial and error instead. Those who are already very familiar with the task are able to deal with it as a matter of routine. General intellectual ability, as measured by reasoning tasks, plays no role in either of these cases. When problem solvers are moderately familiar with the task, analytical reasoning strategies can be successfully implemented.
The approach taken for the assessment of problem solving in ALL relies on the notion of (moderately) familiar tasks. Within a somewhat familiar context the problems to be solved are inexplicit enough so as not to be perceived as pure routine tasks. On the other hand, the domain-specific knowledge prerequisites are sufficiently limited as to make analytical reasoning techniques the main cognitive tool for solving the problems.
How can contextualized, real-life problems be defined and transformed into a set of assessment tasks? After reviewing the various approaches that have been taken in previous research to measure problem solving, a decision was made to use a project approach in ALL. The project approach has the potential to be a powerful means for assessing analytical problem solving skills in real world, everyday contexts for several reasons. Solving problems in project-like settings is important and relevant for adults in both their professional and their private life. In addition, the project approach has been successfully implemented in other large-scale assessments, and it can be realized as a paper-and-pencil-instrument, which is of crucial importance for contemporary large-scale surveys. Furthermore, the project approach uses different problem-solving stages as a dimension along which to generate the actual test items. Following Pólya (1945, 1980), the process of problem solving has been frequently described in terms of the following stages:
The different action steps define the course of action for an “everyday” project. One or more tasks or items are generated to correspond to each of these action steps. Respondents are expected to work on individual tasks that have been identified as steps that need to be carried out as a part of their project (a sample project, for example, might involve “planning a reunion” or “renovating a clubhouse”). Embedding the individual tasks in a project is believed to yield a high degree of context authenticity. Although they are part of a comprehensive and coherent project, the individual tasks are designed so that they can be solved independently of one another and are expected to vary in complexity and overall difficulty for adults.