Critics of the formulas (e.g., Manzo 1970, Bruce et al. 1981, Selzer 1981, Redish and Selzer 1985, Schriver 2000) rightly claim that the formulas use only "surface features" of text and ignore other features like content and organization. The research shows, however, that these surface features - the readability variables - with all their limitations have remained the best predictors of text difficulty as measured by comprehension tests (Hunt 1965, Bormuth 1966, Maxwell 1978, Coupland 1978, Kintsch and Miller 1981, Chall 1984, Klare 1984, Davison 1984 and 1986, Carver 1990, Chall and Conard 1991, Chall and Dale 1995).

Text leveling

An important byproduct of the cognitive and linguistic emphasis was the renewed interest in text leveling. This involves a subjective analysis of reading level that examines vocabulary, format, content, length, illustrations, repetition of words, and curriculum. Text leveling is perhaps the oldest method of grading a text. The McGuffey readers were graded by leveling, and their success is an indication of its validity.

Leveling recently became popular largely due to the work of the New Zealand Department of Education. In the U.S., Marie Clay's (1991) Reading Recovery system uses leveling in tutoring of children with reading problems. In this system, teachers use leveling to find books with closely spaced difficulty levels, particularly at the first-and second-grade levels. Most traditional readability formulas are not particularly sensitive at those levels (Fountas and Pinnell, 1999).

For that same reason, readability experts have long encouraged the use of subjective leveling along with the readability formulas. Leveling can spot the items that the formulas do not measure (Klare 1963, pp. 137-144; Chall et al. 1996; Fry 2002).

R. P. Carver (1975-1976) introduced a method of using qualified raters to assess the difficulty of texts. Raters become qualified when accurately judging the difficulty of five passages using his "Rauding Scale," consisting of six passages representing grades 2, 5, 8, 11, 14, and 17. Carver claimed his method was slightly more accurate than the Dale and Chall and Flesch Reading Ease formulas and provides grade-level scores through grade 18.

H. Singer (1975) created a method called SEER, "Singer Eyeball Estimate of Readability." It involves the use of one or two accurate SEER judges matching a sample of text against one of two scales, each consisting of eight rated passages. Singer claims his method is as accurate as the Fry graph.

The problem, of course, is that it takes some effort to learn how to do leveling accurately. Advanced readers often fail to recognize how difficult texts can be for others. Leveling also becomes more effective and accurate as the number of experienced judges increases (Klare, 1984).

Jeanne Chall and her associates (1996) published Qualitative Assessment of Text Difficulty, A Practical Guide for Teachers and Writers. It uses graded passages, called "scales," from published works along with layouts and illustrations for leveling of texts. You can assess the readability of your own documents by comparing them to these passages and using the worksheet in the book. The 52 passages are arranged by grade level and by the following types of text:

• Literature
• Popular fiction
• Life sciences
• Physical sciences
• Narrative social studies
• Expository social studies