Evaluating the contribution of the variables to task difficulty

As with the prose scale, IRT was used to establish the document literacy scale as well as to characterize tasks along it. Again, a response probability of 80% was used as an indicator that someone at a specified point on the document literacy scale has mastered or is proficient with tasks at that place on the scale. It does not mean that they cannot perform tasks above their estimated proficiency; rather, they may do so, but with less consistency. Their expected consistency on tasks above their level of proficiency depends on how far the task is from their estimated proficiency.

Once the document literacy tasks are placed along each of the scales using the criterion of 80% (RP80), it is possible to determine to what extent the variables associated with the task characteristics explain the placement of tasks along the scales. A multiple regression was run using RP80 as the dependent variable (see note number 3). The independent variables were the three process variables (TOM, TOI, and POD) used to characterize the prose and document literacy tasks, plus a newly developed measure of document readability (READ) (Mosenthal and Kirsch, 1998).⁵ The results are shown here in Table 2.

Multiple R = .89
Adjusted R² = .76

Table 2 shows the zero order correlation between each of the predictor variables and RP80, along with the results from the regression analysis. These data reveal that each of the predictor variables is significantly correlated with RP80, yet only two process variables received significant beta weights. It should be noted that while each of these variables may not be significant in terms of this regression analysis, each was taken into consideration when constructing the literacy tasks and, therefore, each is important as to how well the domain is represented. Together the set of variables accounted for 79% of the variance in RP80 values. Type of match received the largest standardized regression weight, followed by plausibility of distractors.

Easy tasks on the document literacy scale tended to require readers to make a literal match on the basis of a single piece of information. Tasks further along the document scale become somewhat more varied. While some may still require a single feature match, more distracting information may be present in the document or the match may require a low text-based inference. Some tasks may require the reader to cycle through information to arrive at a correct response. Tasks that are more difficult can take on a variety of characteristics. They may still require the reader to make a match, but usually the reader has to match on multiple features or take conditional information into account. Tasks may also require the reader to integrate information from one or more documents, or cycle through a document to provide multiple responses. The most difficult tasks typically require the reader to match on multiple features, to cycle through documents, and to integrate information. Frequently, these tasks require the reader to make higher-level inferences, process conditional information, and deal with highly plausible distractors. These tasks also tend to be associated with more complex displays of information.