5.3 Long vs. short version of projects

5.3.1 Criteria and expectations

From an operational and financial point of view, the short versions of the projects were considered more desirable than the long versions. Therefore the critical question was whether the short versions are as informative as the long versions:

  • Do they measure the same dimension, i.e. can the item parameters of the short versions be linearly transformed into the parameters of the long versions?
  • Is the reliability of the combined short versions comparable to that of the long versions?
  • Do the combined short versions cover a similar range of difficulties and proficiency levels?

5.3.2 Findings and conclusions

The following diagram visualizes the item difficulty1 parameters of the 18 items that were included both in the short and in the long versions of the problem-solving projects. All points in the diagram (each representing a single item) are close to a straight line, indicating that there is an almost perfect linear dependency between the difficulty estimated from the short version and the corresponding difficulty index estimated from the long version. The correlation across 18 items is .90. Thus, both versions represent the same underlying construct.

Figure 3
Comparison of item difficulties for long and short versions

graphic of Item Difficulty Comparison Graph with increasing trend: X-axis: Item difficulty - long version, Y-axis: Item difficulty - short version
1. Within ALL, the so-called P80 index is used to operationalize item difficulty. It indicates the point on the latent competency scale (with logits as units of measurement) where the probability of solving the item correctly equals 80 percent.