Several precautions were taken to ensure that response rates would be adequate. Low response rates are of concern in any survey because non-response might result in biased estimates. Interviewers were instructed to return several times to non-responding households in order to obtain as many responses as possible. In addition, all sample designs included some over-sampling. This refers to the inclusion in a sample of more randomly selected households than are necessary for the required number of completed interviews, to ensure a sufficient number of responses. Finally, the IALS sampling guidelines included an adjustment during the weighting procedure to help correct for non-response bias. This correction, known as post-stratification, adjusts the population weights so that they match known population counts, e.g. by gender, age group or education level. All countries post-stratified their data to such counts. Table B.2 presents the response rates achieved by the participating countries.

Response rates by country

* The response rate for Poland includes only the first wave of sampled persons, before interviewer follow-up. The response rate for Italy is low but the achieved sample matches known population counts. Portugal conducted its literacy survey as part of an European Union sponsored research study undertaken independently of the IALS project but using a similar methodology and equivalent test instruments. Care must be taken when performing more complex data analyses because the number of completed cases is comparatively low.

The response rates realized in IALS are generally lower than those obtained in international surveys of student achievement, in which data are collected from samples of schools, classrooms and students.¹⁷ In several countries with low, response rates, follow-up surveys were conducted in order to determine the presence of bias. No evidence of serious bias was found in the countries investigated.

Subsequent to the data collection, the responses were scored and codes entered onto a highly structured international record layout file. Persons charged with scoring in each country received intense training in scoring responses to the open-ended items using the IALS Scoring Manual. To further ensure accuracy, countries were monitored as to the quality of their scoring in two ways. First, within a country, at least 20 percent of the tests had to be re-scored. Second, each country had 10 percent of its sample re-scored by scorers from another country. Further, as a condition for their participation in the IALS, countries were required to capture and process their files using procedures that ensured logical consistency and acceptable levels of data capture error. Specifically, countries were advised to conduct complete verification of the captured scores (i.e., enter each record twice) in order to minimize error rates. Because the process of accurately capturing the test scores is essential to high data quality, 100 percent keystroke validation was needed.

17. The Third International Mathematics and Science Study (TIMSS), for example, that was conducted at three grade levels in 41 countries under the auspices of the International Association for the Evaluation of Educational Achievement (IEA) during the 1995 school year.