Building an interpretative scheme

Identifying and validating a set of variables that predict performance along each of the literacy scales provides a basis for building an interpretative scheme. This scheme provides a useful means for exploring the progression of information-processing demands across each of the scales and what scores along a particular scale mean. Thus, it contributes to the construct validity of inferences based on scores from a measure (Messick, 1989). This section summarizes an interpretative scheme that was adopted by IALS. The procedure builds on Beaton's anchored proficiency procedures (Beaton and Allen, 1992; Messick, Beaton, and Lord, 1983), but it is more flexible and inclusive than the one originally developed and used in the 1980s by NAEP. It has been used in various large-scale surveys of literacy in North America (Kirsch and Jungeblut, 1992; Kirsch et al., 1993).

As shown in the previous section of this paper, there is empirical evidence that a set of variables can be identified that summarize some of the skills and strategies that are involved in accomplishing various kinds of prose, document, and quantitative literacy tasks. More difficult tasks tend to feature more varied and complex information-processing demands than are required by easier tasks. This suggests that literacy is neither a single skill suited to all types of tasks nor an infinite number of skills each associated with a particular type of task.

In the North American literacy surveys, when researchers coded each literacy task in terms of the process variables described in this paper they noted that the values for these variables tended to "shift" at various places along each of the literacy scales. These places seemed to be around 50-point intervals, beginning around 225 on each scale (Kirsch et al., 1998). While most of the tasks at the lower end of the scales had code values of 1 on each of the process variables, tasks with scores around 225 were more likely to have code values of 2. Among tasks with scores around 275, many of the codes were 2s and an increasing number were 3s. Among tasks with response probability values of 325, at least one of the three variables had a code value of 4. Code values of 4 or higher predominated tasks at around 375 or higher on the literacy scales.

Although there were some variations across the literacy scales at the points where the coding shifts occurred, the patterns were remarkably consistent. Further, as was shown in this paper with the IALS tasks, this system of coding tasks accounts for much (although not all) of the variance associated with tasks along the literacy scales. Based on these findings, researchers defined five levels of proficiency having the following score ranges:

Level 1: 0—225

Level 2: 226—275

Level 3: 276—325

Level 4: 326—375

Level 5: 376—500

Once the literacy levels were identified based on the noted shifts in code values for the three process variables, criteria were identified that would describe the placement of tasks within these levels. These criteria are summarized along with the data to which they were applied in a chapter appearing in the IALS technical report (Kirsch et al., 1998). Based on evidence resulting from this work, the five literacy levels were used for reporting results from literacy surveys in both national and international surveys using these literacy scales.