4. Factors affecting complexity of Numeracy Items

Because of the scarcity of research on adults' use and application of numeracy, there is insufficient empirical knowledge to determine what factors make a numeracy activity or task more difficult or complex. One of the more exciting, and challenging, aspects of the project was the development of a scheme to account for the difficulty of different numeracy assessment tasks. We sought such a scheme to inform item development, i.e., help in the creation of items that spread over a range of difficulty levels. However, if the scheme could be shown to correlate with actual difficulty levels of items as measured in actual testing of a sample of individuals, it could also be used to help explain observed performance. Given its importance for both item development and interpretation of results, the complexity scheme is described in detail in this section.

4.1 Previous research on task complexity

In IALS, three factors were found to be the principal components of task difficulty regarding literacy or text-based tasks: plausibility of distractors, type of match required, and type of information required. The difficulty of the Quantitative Literacy tasks appeared to be a function of several other factors:

1. The particular arithmetic operation required to complete the task
2. The number of operations needed to perform the task
3. The extent to which the numbers are embedded in printed materials
4. The extent to which an inference must be made to identify the type of operation to be performed (i.e. problem transparency; see below)

The IALS QL difficulty factors overall fit those used in large-scale assessments of mathematical skills (with children), which often make use of three or four factors:

1. The mathematical concepts involved: number systems and number sense, spatial and geometrical topics, functions and algebra, chance/statistics topics, etc. Concepts that are related to topics taught in lower grades are considered easier.
2. The complexity of operations: addition, subtraction, multiplication, and division, as well as dealing with whole numbers, with decimals, and with percents. Operations that are related to topics taught in lower grades are considered easier.
3. The number of operations: one-step problems are considered easier than multistep problems.
4. Problem transparency: This factor is sometimes relevant; it refers to the extent to which the problem situation includes clearly identified numbers or entities and the extent to which it is clear what operations or actions to perform. To the extent that these are not clear or transparent, respondents have to extract needed information by applying comprehension and inference strategies, making the task more complex.

There are other adult-related assessment projects on which to draw to develop the levels of complexity. Both the Essential Skills Research Project and the Applied Numeracy sub-test of the Work Keys test battery (American College Testing, 1997) use a two-factor model of complexity in their description of numeracy levels. The first factor "operations required;" is seemingly straightforward and refers to the difficulty of operations called for. However, this is complicated by the level of difficulty of the numbers being manipulated: computations that include fractions and decimals are usually more difficult than those with whole numbers.