Literacy data: more, but not always better
Elizabeth Buckner, Research Associate, EPDC; Rachel Hatch, Research Associate, EPDC
This post is adapted from EPDC's latest polic brief, Literacy data: more, but not always better. In part 1 of 2 posts, we explain the main sources of literacy assessment data and difficulties arising from trying to compare them, from factors such as sampled populations to language to the instruments used for assessment.
Sources of external literacy assessment data
External literacy assessments can be classified into three major types: cross-national, standardized exams administered in schools (both regional and international); community-based assessments administered in homes; and, nation-specific assessments typically administered in schools. In addition, we examine data from the Early Grade Reading Assessments (EGRA), which measures students’ early reading skills such as word recognition and phonemic awareness.
Community-based Household Assessments
UWEZO, ASER and Beekungo are the three major community-implemented household assessments that assess literacy. They test basic skills, including letter name knowledge, simple word reading, sentence reading, and basic comprehension. Because they assess students in their homes, they are able to assess literacy skills of children who are both in and out of school, which is something that no other assessment does. They also cover a broad age range, testing children aged 5-16, rather than those in specific school grades. Nonetheless, the levels of literacy tested are quite low – the highest level of reading tested in any of the exams aligns to the ability to answer two questions on a two to three sentence story related to daily life. This low benchmark emerges from the exams’ focus on assessing functional literacy; however, it is likely too low to be a meaningful indicator of reading in an academic setting, which is an important factor in students’ ultimate educational attainment. Additionally, although UWEZO efforts are coordinated across multiple East African nations, literacy assessments are pegged to the 2nd grade curriculum in their national context. This means that the results of these exams are not strictly comparable to those in other nations, given the differences in national curricula.
Table 1 - Key dimensions of selected literacy assessments
.png)
Regional and international student assessments
One of the major advantages of regional and international student assessments, which distinguishes them from other assessments, is that they are standardized cross-nationally, thereby allowing for meaningful comparisons of student skills and competencies. The major regional exams with available literacy data are the: Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ), the Programme d’Analyse des Systèmes Educatifs de la CONFEMEN (PASEC) and the Segundo Estudio Regional Comparativo y Explicativo (SERCE). Regional exams such as these offer standardization across many countries in the same language (English, French and Spanish, respectively). Despite the obvious advantage of comparison between countries participating in regional studies, there are also limitations. These assessments only test students who are in school, and exam content is pegged to a certain grade. For example, SACMEQ does not provide insight into the literacy abilities of children who dropped out of school before 6th grade, which means that its indicators significantly overestimate literacy in the population as a whole.
At the international level, a growing number of countries also participate in the Progress for International Reading and Literacy Study (PIRLS), an internationally-standardized student assessment conducted at 4th grade, which benchmarks students’ literacy to four levels.However, very few low and middle-income nations participate in PIRLS, simply because even the lowest literacy benchmark in PIRLS is above the literacy abilities of the vast majority of their students. The lowest benchmark for fourth graders in PIRLS is the ability to “recognize, locate, and reproduce explicitly stated details from the texts, particularly if the details were close to the beginning of the text,” as well as making straightforward inferences. The overwhelming majority of students in many low-income countries would fall into this level; as a result, PIRLS offers little insight into to students’ spectrum of literacy abilities or the specific needs of students in many low- and middle-income nations.
Nation-specific assessments
In addition to standardized regional and international assessments, there are two other widely cited types of nation-specific data on literacy. The first is national exams – many countries engage in standardized exams of samples of their students with the goal of assessing students’ educational achievement and examining their progress over time. Because of their close linkages to national curriculum, these exams are not intended to be comparable cross-nationally. The data from such exams is also hard to access, often only provided in national reports that do not give much detail into the definitions of literacy used or the specific test items. As a result, making meaning of reported statistics can be quite difficult. Also, literacy levels may be linked narrowly to the national curriculum, which can mean distinguishing generalized literacy abilities as distinct from mastery of content can be difficult. Finally, the Early Grade Reading Assessment (EGRA) is a rapid-diagnostic tool widely adopted in the international development community. It takes between 15 and 20 minutes per student and is administered individually and orally. EGRA is a fast and relatively cost-effective method for assessing students’ literacy sub-skills, including phonemic awareness, letter name knowledge, and fluency, as well as basic comprehension. However, EGRA makes it clear that comparisons should not be made across languages or countries, which limits its ability to produce generalized data on the state of literacy in a country. Additionally, one of the most commonly cited indicators produced by EGRA is the average number of words read per minute. Rather than viewing literacy as a relatively static indicator of ability (i.e., can or cannot read) EGRA’s indicators emphasize the rate of reading as a more meaningful definition of literacy (i.e., the speed at which one can read). Although this ultimately may be a more meaningful indicator of overall literacy abilities, its findings are difficult to compare to other exams, where there is little documentation concerning how long students are given to read passages and complete comprehension questions. In fact, until now, the only meaningful indicator of literacy that has been considered comparable across countries and contexts is a zero score vs. non-zero score.
Key challenges in understanding literacy data
The growing number of cross-national assessments, with their varying definitions of literacy, mean that consumers of literacy data face real difficulties when seeking answers to even basic questions about literacy, such as: what percent of children at a given age can read? A major problem is simply that each assessment samples from a distinct population – each is targeted to children of different ages, either students in school or all children in and out of school, tested in different languages, and sometimes, in only some regions of the country. For example, EGRA data is not always available nationwide, as it tends to be implemented in conjunction with donor-funded projects. UWEZO assesses students both in and out of school, while PIRLS and SACMEQ only test students in school. This means that we cannot compare indicators of literacy rates across these assessments because they are drawn from different sample populations. As a result, it is very difficult to compare or validate data across sources. Another issue is that each assessment uses slightly different definitions of literacy – for example, each assessment may focus on only a set of literacy sub-skills, or base its definition of reading competency on substantially easier texts than another assessment. Moreover, it is important to note that even while most definitions of literacy include writing, few external assessments actually assess or report writing abilities. Each assessment also classifies students into varying levels of literacy (four, five, six, up to eight levels of literacy) and it is simply not clear how a given level of literacy on one assessment maps onto levels from another assessment.This is made more complicated by the fact that in some tests, a literacy level maps onto a particular skill (e.g., ability to identify letters, ability to read words) while in other tests, students are given a composite score (e.g., 500) based on multiple test items, and this score refers to generalized abilities.
Moreover, even within the same assessment, results may not be comparable across samples. For example, EGRA assessments cannot be compared across languages even within the same country. Most EGRA tests are targeted to a high 1st grade or low 2nd grade level, while in reality, word difficulty is very context-specific (e.g. students may be familiar with British English terms, rather than American English). Until now, no scientific process exists for standardizing EGRA assessments across languages, and country contexts. This means that literacy data from different assessments cannot be reliably compared. Linguistic diversity also poses distinct challenges for garnering comparable indicators of literacy cross-nationally. Ideally, literacy assessments should be able to capture students’ linguistic abilities in multiple languages to best assess their literacy levels. However, regional and international literacy assessments rarely test more than one language, and when they do, their findings are not always consistent across languages either because students’ abilities differ across languages or tests are not designed similarly across languages.
Finally, despite the growing number of assessments, availability of data remains an issue. Although substantial data on literacy exists for some countries, there are still major gaps in coverage of literacy rates for some countries, populations, and sub-groups. This means that UNESCO self-reported literacy data is still the only indicator of literacy rates available in many nations, despite the move towards external assessments in the international community. Similarly, even when comparisons across tests are possible because they assess similar samples of students (similar ages, grades, languages, same country), getting comparable data is not easy. Compiling findings from different exams requires mining through various report documents, posted on different websites and downloading and filtering Excel data. In response, EPDC’s Learning Outcomes Data has standardized and compiled findings from various reports and sources to make data on learning outcomes comparable as possible. Ultimately, we argue that we must take all literacy data with a healthy dose of skepticism – the specific definition of literacy used in assessments influences the conclusions we draw about the state of literacy in a given context. Moreover, given the variety of sources of data on literacy, obtaining reliable and consistent indicators of literacy rates worldwide is still extremely difficult. In the next section, we examine how difficult it is to get consistent data on literacy by examining the case of Uganda.
Add new comment