The analysis and interpretation of data were carried out by two test developers in collaboration with an external assessment consultant. The objectives of the analysis were to determine the following:
All of the pilot forms were constructed to include more items than would be needed for the operational assessment. This was done so that any items revealed by the data analysis to be flawed, redundant or problematic could be removed from the final versions, leaving only the most informative and technically sound material for the operational test. As a result of the data analysis, a number of items were revised or relocated, and in some cases task order was changed to create a smoother progression of difficulty.
Following the data analysis, a second round of pilot testing was carried out to ensure that revised or relocated items would perform as expected. Pilot subjects wrote both forms of the test, each form written on a different day. Order of administration of forms was counterbalanced to mitigate the possibility of a practice effect. The assessments were scored by a single assessor, and task and item data were again analyzed and reviewed.
The second pilot procedure confirmed that the refined test forms were sufficiently equivalent and that they provided a balanced snapshot of a learner’s ability in the four target skill areas. Therefore, no further changes were made to the instruments.
When the content of CAMERA Stages 1, 2, and 3 had been finalized, materials were selected for the operational version of the CAMERA Placement. The selection process began with a review of data gathered on the original placement test, which revealed that many of the original tasks and items were technically sound. Those original tasks which proved to distinguish most clearly between levels were retained and streamlined through the removal of unnecessary items. In cases where a task to be retained was among those that had been revised for inclusion in CAMERA Stage 1, 2 or 3, the revised task was used for the CAMERA Placement.