Part C
Pilot Study: Summary of Numeracyrelated results and decisions
8. Overview and purposes of the pilot stage
The ALL Pilot study was conducted in 2002 in six countries (and five languages):
Belgium (French), Canada (French and English), Italy, Norway, Switzerland (German,
French, Italian), and the United States. Large samples responded to test booklets
assessing the key skill domains targeted by ALL: Document literacy, Prose literacy,
Numeracy, and Problem-solving. The Background Questionnaire (BQ) provided
additional information about respondents' experiences with Information Technology
and about numerous variables of importance such as employment status, consumption
of health services, participation in learning activities, and perceptions of quality of life.
The BQ also collected additional personal information that could help to explain or
interpret results pertaining to the key variables.
8.1 General goals
The Pilot study was designed to accomplish two goals:
- Enable the ALL management and the development teams to evaluate the
psychometric characteristics of the item pools created to assess the different
skill domains, and provide sufficient data from which to select the items
to be included in the test booklets and the BQ that will be used in the Main
ALL survey.
- Enable participating countries to field-test and evaluate all survey
materials and administrative procedures planned for the Main ALL survey,
such as
contacting respondents, setting home visits, test administration, scoring
of responses, data capture, and quality assurance mechanisms.
8.2 Numeracy-specific goals
In addition to the two general goals listed above, the Pilot study aimed to provide
information regarding specific issues relevant to Numeracy assessment in ALL:
- Enable selection of 40 items out of the 81 Numeracy items tested in the
Pilot, whose development was described in Part B. The target number of
40 items was determined on the basis of knowing average response times
per item in the feasibility studies, and the overall time expected to be
available for Numeracy assessment in the Main study.
- Analyze response patterns and verify that errors are not caused by unclear
or problematic stimuli, questions, or instructions, or by inconsistent
translation. Also, determine whether the multiple scoring codes created
for some items should be maintained for the Main study
- Collect data on the extent to which respondents used the calculator
provided as part of the assessment.
- Evaluate the relationships between Numeracy scale scores and variables
measured by the BQ, including scores on scales related to numeracy
practices and to beliefs and attitudes.
|