5.6 Relation to other domains

To what extent does analytical problem solving in real world contexts, when measured with the project approach, overlap or go beyond literacy and numeracy skills? Reading skills, particularly document literacy, and some rudimentary numeracy skills, are necessary pre-requisites for successfully responding to the problem-solving tasks. We therefore expect the problem-solving score to co-vary with the scores of literacy and to a certain extent the numeracy score. Furthermore, we expect that analytical intelligence, in the psychometric sense (which is not measured in the ALL study), will influence the score.

The project tasks present several special aspects that go beyond literacy tasks and abstract intelligence tasks:

1. They are embedded in a complex, realistic, multi-step action context. Whereas the understanding of textual and document information is central for the literacy items, in problem-solving tasks the focus is on using this information in order to solve everyday-type tasks.
2. They demonstrate a high complexity made possible, for example, by integrating various representational formats (verbal, numerical, pictorial, diagrammatic), by setting multi-dimensional or ill-defined goals and by some intransparency in defining the problem situation. They therefore require more regulation activity than the more "transparent" literacy and numeracy tasks.
3. They require various kinds of reasoning activity, including analogical, inductive, deductive, and critical reasoning, involving content-related as well as logical and formal operations. As explained in section 2.5, the kind of reasoning required determines the complexity and difficulty of a problem.

These distinctive features of cross-curricular problem solving competency have also been highlighted in other assessment frameworks such as PISA 2003 (OECD, in press), PISA/Germany (Klieme et al. 2001) or the large-scale study in Hamburg/ Germany (Ebach, Klieme and Hensgen 2000). The empirical results of the two German projects clearly support the hypothesis that problem-solving tasks form a new dimension within the structure of competencies that is usually addressed by large-scale assessments. Based on structural equation modeling, it has been shown that problem-solving tasks of the "project" type do constitute a unique factor, which can be discriminated from reading literacy, numeracy/mathematical literacy, and also from psychometric intelligence.

When the ALL problem-solving items were developed, there was a clear intent to keep the language structure simple and straightforward. However, the frequently higher complexity of project tasks clearly requires a sufficient understanding of the instructions by the respondents, which itself pre-supposes a minimal level of reading literacy. This can result in respondents with very low literacy skills not understanding the project tasks, and, therefore, being unable to solve them. Thus, literacy skills are likely to have a threshold function for the analytical problem-solving scale. On the other hand, each of the literacy and numeracy tasks presented in ALL requires a certain amount of reasoning. Therefore, it could also be expected that a minimal level of analytical problem-solving proficiency is a prerequisite for prose and document literacy and numeracy. These competing hypotheses will be tested within the ALL main survey.