Vocabulary-frequency lists

During the 1920s, two major trends stimulated a new interest in readability:

1. A changing school population, especially an increase in "first generation" secondary school students, the children of immigrants. Teachers reported that these students found textbooks too difficult.
2. The growing use of scientific tools for studying and objectively measuring educational problems.

One such tool, Thorndike's Teacher's Word Book (1921), was the first extensive listing of words in English by frequency. It provided teachers with an objective means for measuring the difficulty of words and texts. It laid the foundation for almost all the research on readability that would follow.

Its author, psychologist Edward. L. Thorndike of Columbia University, noticed that teachers of languages in Germany and Russia were using word counts to match texts with students. The more frequent a word is used, they found, the more familiar it is and the easier to use. As we learn and grow, our vocabulary grows as does our ability to master longer and more complex sentences. How much that continues to grow depends on how much reading is done throughout life.

A vocabulary test on the meaning of words is the strongest predictor of verbal and abstract intellectual development. The knowledge of words has always been a strong measure of a reader's development, reading comprehension, and verbal intelligence. Chall and Dale (1995, p. 84) write, "It is no accident that vocabulary is also a strong predictor of text difficulty."

It happens that the first words we learn are the simplest and shortest. These first, easy words are also the words we use most frequently. Most people do not realize the extent of this frequency. Twenty-five percent of the 67,200 words used in the 24 life stories written by university freshmen consisted of these ten words: the, I, and, to, was, my, in, of, a, and it (Johnson, 1946). The first 100 most frequent words make up almost half of all written material. The first 300 words make up about 65 percent of it (Fry et al, 1993).

Around 1911, Thorndike began to count the frequency of words in English texts. In 1921, he published The Teacher's Word Book, which listed 10,000 words by frequency of use. In 1932, he followed up with A Teacher's Word Book of 20,000 Words, and in 1944 with Irving Lorge, A Teacher's Word Book of 30,000 Words.

Until computers came along, educators, publishers, and teachers commonly used word-frequency lists to evaluate reading materials for their classes. Thorndike's work also was the basis for the first readability formulas for children's books.

After Thorndike, there was extensive research on vocabulary. The high mark came in Human Behavior and The Principle of Least Effort by Harvard's George Kingsley Zipf (1949).

Zipf used a statistical analysis of language to show how the principle of least effort works in human speech. Zipf showed that, in many languages, there is a mathematical relationship between the hard and easy words, now called Zipf's curve. This notion of saving energy is a central feature of language and is one of the principle bases of research on the frequency of words.

Klare (1968), reviewing the research on word frequency, concludes: "Not only do humans tend to used some words much more often than others, they recognize more frequent words more rapidly than less frequent, prefer them, and understand and learn them more readily. It is not surprising, therefore, that this variable has such a central role in the measurement of readability."

Dale and O'Rourke: the words Americans know In 1981, publishers of the World Book Encyclopedia published The Living Word Vocabulary: A National Vocabulary Inventory by Edgar Dale and Joseph O'Rourke. The authors based this work on the earlier work of Thorndike and others as well as on a 25-year study of their own. It contained the grade-level scores of the familiarity of 44,000 words. For the first time, it gave scores for each of the meanings a word can have and the percentage of readers in the specified grade who are familiar with the word.