5. Defining the ALL proficiency scale for problem solving: Empirical results from the field trial

5.1 Design of the ALL pilot study, and scaling and validity issues

Based on the development work outlined in the previous chapter and the subsequent results of a feasibility study in Germany and the United States, four projects were selected for an extensive pilot study in a number of countries. For each of these projects, a long and a short version were created in order to provide for more flexibility and to check if a shorter and more economical version of the projects yields data of equivalent quality and comparable measures for problem-solving competencies.

Thus four projects, each with a long and a short version, were tested in the ALL international large-scale pilot study. Two of the projects revolve around planning some kind of public event. The respondent is instructed to imagine she is a volunteer in a local group who has decided to organize a major event. The respondent is then led through a sequence of 3 to 7 tasks that correspond to action steps. The exact number of tasks or steps depends on the project and the version (short vs. long) she is working on. As explained in the last chapter, these action steps break up the general aim of the project (in this case "organizing an event") into setting a goal, analyzing, planning, executing, and evaluating. Within each of the action steps, the respondent is provided with some pieces of information — e.g. program of the event, time tables, places, people involved, budgets, and so on. The respondent needs to put a given set of actions into the right order, to draw conclusions and to make decisions based on the information given. In a similar manner, the two other projects implemented in the pilot study address complex decision making processes in a private context, e.g. making an important purchase.

The pilot study took place in five countries and in six languages: Belgium (Flemish), Canada (French and English), Italy, Norway and Switzerland (French, German, and Italian). The preliminary analyses discussed here are based on data from four countries (excluding Belgium), covering seven test versions in five languages. The total number of respondents was N=2102.

A problem-solving test booklet consisted either of two 30-minute problem-solving projects (long versions), or of four 15-minute projects (short versions). Therefore, respondents worked either on two long versions of the projects, or all four short ones. The long versions were labeled E, F, G and H, and the respective short versions were labeled I, J, K and L. Approximately equal numbers of respondents worked on each of the combinations E + F, F + G, G + H, and H + E (so-called matrix design), while twice this number of respondents worked on I + J + K + L. Because of design issues, missing values and some deviations from the standard implementation, case numbers varied from analysis to analysis. The preliminary analyses reported here are based on data sets of 530-710 respondents per item with a mean of 668 respondents per item.

The general aims of this pilot study were to:

1. test whether there is a common scale across projects.
2. evaluate empirical and other arguments pro and contra the short and long versions of the problem-solving projects, and make a recommendation for one of the versions.
3. select and revise items for the main survey.
4. validate the proficiency scale, especially the four levels of problem-solving competency that were introduced in chapter 2.