Ideally a survey of adult numeracy skills would ask respondents to complete tasks that are couched in real-life situations and that encompass all components of the facets of numerate behavior described earlier. Further, respondents would ideally respond to in-depth and qualitative interviews in order to understand their answers and reactions as well as their thought processes. Since the ALL is a paper-based household survey, however, there are restrictions on the 'real' nature of the situations that could be included. In addition, the available options for presenting questions and for eliciting, recording, and reliably scoring responses are limited when large samples across wide ranging populations have to be surveyed. Finally, given that numeracy involves a sizable number of facets and subcomponents, practical limitations on the number of items that can be administered to respondents to assess each of the domains covered in ALL mean that not all aspects of the facets of numerate behavior can necessarily be covered at an equal depth.

Operating within these constraints, the goal for the item development process described in the next part was to create Numeracy items that are based on realistic stimuli and include tasks that serve a purpose for most people within their culture. A scheme of factors affecting the complexity of numeracy tasks was developed as a tool to support item development and to help explain performance on Numeracy items. Information derived from performance on the Numeracy scale was designed to be supplemented by information gathered via the Background Questionnaire about prior practices and attitudes related to numeracy, in addition to other important explanatory variables such as those related to demographic data and literacy practices. The process of developing and evaluating items along with their scoring rubrics is described in Part B.

Closing this summary of challenges in converting the conceptual framework of Numeracy into an assessment tool, it must be noted that Numeracy cannot be fully separated from Document Literacy and Problem Solving, two other important domains assessed by ALL. Numeracy is somewhat related to Document Literacy, due to the role that text and graphical or tabular displays play in both constructs. Likewise, Numeracy is not fully distinguishable from Problem Solving, as certain numeracy tasks require the implementation of problem solving processes, albeit brief at times, while in contrast certain "problems" (e.g., designing a budget) involve numerical content.

Hence, a final challenge in designing items for the Numeracy scale was to achieve a sensible "division of labor". On one hand, it is important to prevent excessive overlap between what is measured by the Numeracy scale and the type of skills assessed by scales in other domains. On the other hand it is also important to maintain a reasonable coverage of the core issues encompassed by the construct of numeracy as envisioned in this report, even if this creates some acceptable degree of overlap with other scales. Such overlap, we argue, is not a limitation of the conceptual framework developed to describe Numeracy and numerate behavior, but simply reflects the complex nature of real-world situations and tasks. While it is convenient for educators and designers of assessment tools to think of skill areas as distinct and having clear-cut boundaries, real-life situations may require that people who have to manage and respond to them rely on interrelated knowledge and skills.