Directions

A READABILITY SURVEY OF SOCIAL SCIENCE, SCIENCE
AND INDUSTRIAL ARTS UNDP MATERIAL USING THE
CLOZE PROCEDURE
Daniel Stamp and fourteen Diploma students*
School of Education, USP.
As teachers, we all realise the importance of choosing material which our
pupils can read and understand. If the material is too difficult, pupils may
become frustrated and turn away from the topic. The teacher, therefore, in
his new role as organiser, director, stimulator and evaluator of individual
learning, must be aware of the problems associated with printed materials.
In its generic sense, the term 'readability' refers to the relative ease with which
people can respond in various ways to passages of written prose. Historically,
researchers have measured a variety of responses to index the readability of
prose passages. The types of. responses have included measures of reader
interest, reading speed, recall of the passage, proportion of correctly
answered questions about the passage and proportion of cloze items
answered correctly.
1 The objective of readability research is to find out the suitability of language
used in thestudents' instructional materials.
There are two major approaches to this objective. The first approach consists
of adapting materials to the students, either by selecting materials which are
understandable to him or by adjusting the language in materials to suit his
language comprehension ability. The main direction of research into readability
has been to develop accurate prediction formulas and testing methods for
estimating the suitability of materials for students, but increasing attention is
now being given to developing the technology necessary for adjusting
materials to suit the abilities of the students.
The second major approach consists of adapting the student to the materials
by giving him appropriate instruction in language comprehension skills.
Readability research has contributed to the theory of language comprehension
instruction by analysing and identifying linguistic features which serve as the
stimuli for the comprehension process.
*This study was undertaken as a class exercise by the following students, and
was co-ordinated by Daniel Stamp: Takui Ma'afu, Sesilia Fifita, Ofa Uepi
Aoete Apelu, Michael Liliu, Salote Loanakadavu, Tara Kenatale, Sateki
Fiisipeau, Leela Narayan, Hari Chand, Maya Lai, Akilesh Kumar, Teaukura
Eliaba, and Indar Deo.
23

There are two methods by which readibilty is measured. They are the
readability formula and cloze procedure method. The former are the most
frequently used and most widely accepted methods for measuring readability.
Their advantage is that they can be used to predict how difficult a passage
would be for a group of readers. They are usually based upon the analysis of
easily identifiable aspects of the text such as sentence length and word
complexity or familiarity.
Until the cloze readability procedure was developed by Taylor1, investigators
usually measured the comprehension difficulty of materials by making the
conventional type of completion or multiple-choice test for each of a set of
passages, administering the passages and their corresponding tests to a single
group of students, and then calculating the mean percentage of questions
the students answered correctly on the test for each passage.
The second procedure, the cloze readability procedure, has provided a more
adequate method of measuring passage difficulty. Cloze readability tests are
made by deleting every fifth word in a passage and replacing the deleted
words with underlined blanks of a standard length. Students are instructed to
write. in each blank the word they think was deleted, and they are given
whatever time they need in order to complete the task. Responses are
scored correct when they exactly match the words deleted. Because cloze
readability tests are made entirely from a set of standard mechanical
operations, they are not subject to the biases that test writers may impose
during the writing of conventional tests.
There has now accumulated quite a number of investigations into the validity
of various aspects of the cloze readability procedure. Rankin, who reviewed
the research on the general cloze procedure, and Bormuth2, who reviewed the
research pertaining specifically to the cloze readability procedure, each
concluded that the evidence seems to support strongly Taylor's contention
that responses to items on cloze tests seem to be psychologically indis-
tinguishable from responses to items normally used to measure comprehension
and also his contention that the tests provide valid measures of the
comprehension difficulties of passages.
Researches and arguments about cloze readability have therefore led many of
us to conclude that the cloze method is the most effective method of measuring
readability.
As to its applicability in the South Pacific, Professor J . Anderson of the
University of Adelaide in South Australia. carried out research in Papua New
Guinea based on the cloze method3: The results that he obtained were similar
to the results of a similar test conducted in America and Australia. Below is a
table illustrating this point:
24

TABLE 1
Cloze Test Percentage Scores Corresponding to Multiple-Choice Test
Scores of 7 5 % and 9 0 %
Comparable Cloze Scores
Multiple-Choice
Bormuth
Bormuth
Rankin &
Anderson &
Criterion Levels
(1967)
(1968)
Culhane
Hunt (PNG)
USA
USA
USA
75%
38
44
41
44
90%
50
57
61
53
Table 1 shows therefore that the cloze method works in a similar way with
second language learners, as it does in USA.
If results from independent investigations conducted in different countries and
within different educational systems seem remarkably close, then this is
evidence that points to the reliability and effectiveness of the cloze procedure
method. Therefore, there is no reason why this procedure cannot be applied in
the South Pacific.
Meanwhile, Warwick Elley of the University of the South Pacific conducted a
similar survey in Fiji. He gave the t w o tests to 100 Class 7 students. The
correlation he obtained between the multiple-choice type and the cloze test
type was .79. Again, this shows that the cloze procedure method is reliable.
Today, in many schools of the South Pacific, the students are given a lot of
reading to do from the pupils' booklets prepared by the UNDP. In view of the
above point, the Curriculum Development Unit research group in the Study of
Contemporary Issues at USP felt it appropriate to test the readability of the
prepared pupils' booklets.
METHODOLOGY OF SURVEY
When designing the three cloze tests used in the survey the researchers took
into account several criteria considered relevant, such as age and the level of
study of the students. The three cloze tests were prepared for Forms 2 and 3
students in the Suva-Nausori area with ages ranging from 13-15 years old.
Other criteria are discussed below.
Three discipline areas were tested. They were Social Science, Science, and
Industrial Arts. The three samples tested in each discipline were taken from
UNDP Form 3 Social Science and Science Form 2 Industrial Arts pupils'
pamphlets.
25

" A Close Look at the Cloze Test in Assessing Readability" by Warwick Elley
suggested the following methodology:
1. Select 2 or 3 typical passages from the material to be rated, each at least
100 words in length.
2. Delete every fifth word after the first line of the passage.
3. Prepare the mutilated passage as a test, with blanks of uniform length in
the gaps.
4. Administer the test to the students for whom the material is intended,
and ask them to fill the gaps by writing in the words intended by the
author.
5. Score the students' responses, accepting only the exact word deleted
from the original text. Mis-spellings are not penalised.
6. Where students score approximately 40-45% correct, or more, it can
be concluded that the material is within their comprehension.
The researchers followed the same method as above. Out of the total words
deleted in each passage, one fifth were nouns; for example, if there were 20
words deleted, 4 were nouns. The time taken for the three tests was not
specified but varied, to allow all pupils to finish.
One sample test from Social Science is shown below along with the standard
cover that was used to introduce each test. In five of the nine selections,
accompanying diagrams were included alongside the text.
Cloze Test — Directions
Name: Race:
School: Sex:
Form: Date:
The Tests
The tests are to find out how well we read. Some words have been
left out the passages, and we have to write missing words,
in the blank
Each blank stands for one word only.
Now that the teacher has explained what to do, turn over the page and
complete the tests.
Social Science - Test 2
The conflict that I am going to describe is one that I myself was personally
involved in. It took place between parents and myself over
Muslim boy.
I became friendly with this boy spent long hours in
26

with him. Anyway, things going fine until my parents
thought we were getting involved in our relationship
personally did not like boy at all because was not of our
, but he was a Muslim just couldn't reason with
Anyway, after several attempts discourage me from talking
this boy, things came a head one particular I
was talking to boy when my inquisitive interfered. She
told the boy and chased me inside house.
SUMMARY OF FINDINGS.
The sample is not large enough to come to hard and fast conclusions.
Nevertheless, a few comments can be made from the data obtained.
The mean percentage scores for all three disciplines are well below the
expected mean of 40% plus.
TABLE 2
Mean Scores on Readability Tests
No. of Schools
No. of Pupils
Mean %
Science
6
164
22
Social Science
7
206
21
Industrial Arts
3
80
26
1. All the materials proved very difficult for these Form 2-3 pupils.
2. Industrial Arts materials showed a slightly higher mean.
3. No significant differences were found between Fijians and Indians,
although the number of Fijians tested was much smaller in each case.
4. No significant differences were found between the sexes.
5. Statistical analysis of the tests showed that each of the 3 tests in the
different disciplines was of equivalent difficulty.
6. Initial indications show that urban school children scored slightly higher
than rural school children. A factor which may account for this is that
urban students are more accustomed to the English language.
IMPUCATONS FOR SOUTH PACIFIC EDUCATORS
The majority of activities in which pupils work, either individually or in small
groups, depends on the use of written directions, explanations, comments
27

and questions which the pupils must read. All practising teachers realise
that an inability by the pupils to read and comprehend these instructions all
too often results in failure and frustration.
This survey indicates that the readability of some of the pupils' materials in
the South Pacific is far too difficult and that the vast majority of school
students cannot comprehend the written word at this level. The results show
that very few students are capable of reaching the 40% platform which
indicates comprehension.
If the innovatory curriculum with its emphasis on pupils' materials is to
succeed in the South Pacific, then teachers must be made aware of the
language difficulties of their students. The readability of curriculum materials
currently in use should be critically examined, not only by educators but by
secondary school teachers themselves.
We make the following recommendations:
* This study should provide the starting point for further research into the
readability of pupils' materials. Teachers can design their own tests
based upon the cloze procedure to gauge the readability of materials
they are using.
* If a teacher suspects that written material is too difficult for his students,
stories or activities should be rewritten, and the passage simplified. A
comparison of the teacher's text with the published one would be a
worthwhile follow-up.
* In choosing text-books for courses or for the school library, use reada-
bility as a major factor in deciding which book to buy. Most publishers
will provide sample pages on request, which the teacher could use to
make up a 'cloze test'.
* Always be aware of the comprehension problems faced by the students.
Read through the text, simplifying where necessary. Use other materials
to supplement the text, e.g., tapes (if available), diagrams and charts.
REFERENCES
1. Dieghton, Lee C. The Encyclopaedia of Education, Crowell-Collier
Educational Corporation, 7, 361-67.
2. Bormuth, John R. (1963), Cloze as a Measure of Readability, International
Reading Association Proceedings, 8, 131-4.
3. Anderson, Jonathan, Cloze Procedure as a Measure of Readability,
Papua New Guinea Journal of Education, 6(3), 21-8.
4. Elley, Warwick B. (1977), A Close Look at the Cloze Test, Set 7 7 Number
One NZCER.
28