THE MISSISSIPPI LEGISLATURE
The Joint Committee on
Performance Evaluation and Expenditure Review
Report # 319
Executive Summary for
An Evaluation of Mississippi's Assistant Reading Instructor Program
December 13, 1994
Background
The Mississippi Legislature created the Reading Improvement Program in December 1982 as part of a landmark statewide education reform effort known as the Education Reform Act. Also referred to as the Assistant Reading Instructor Program, the Reading Improvement Program sought to improve basic skills acquisition by placing assistant reading instructors in kindergarten through third-grade classrooms.
As of the close of the 1993-94 school year, the state had expended over $340 million on salaries and fringe benefits for assistant reading instructors since the program's implementation in the fall of 1983, and over the five years ending with the close of the 1993-94 school year, local school districts had spent an estimated $18.75 million.
[*Source: undated pamphlet entitled "Mississippi's Reading Improvement Program," produced by the Mississippi State Department of Education]
The Reading Improvement Program Has Not Been Effective
In the eleven years since implementation, the program has achieved none of these results. PEER's review of program effectiveness shows that, on average, students with access to assistant reading instructors scored about the same on standardized reading tests as students without assistant reading instructors (see Exhibit A, below). While PEER found a slight improvement in reading scores for students in extremely low-scoring school districts, these scores, as well as average scores statewide, remained below the program goal of exceeding the national average. Retention rates in the primary grades have remained about the same, while dropout rates have remained constant in the lower grades and increased in the higher grades (see Exhibits B and C on page x).
The dramatic results that reportedly were achieved by the Mississippi school district which piloted the Reading Improvement Program-e.g., an increase in reading scores of 36 percentile points-have not been replicated on a statewide basis. At the time that the Legislature adopted the Reading Improvement Program as a cornerstone of Mississippi's education reform efforts, State Research Associates, a public affairs consulting group, reported to Mississippi's governor: "We believe the use of teacher aides holds greater promise of improving later school achievement than any other single program we could recommend at this time." The obvious question is: what went wrong?
Possible Reasons for the Program's Lack of Effectiveness
The initial expectation that the statewide Reading Improvement Program had unlimited potential and could even approach effecting a 36 percentile point gain in achievement test scores statewide was based on improper interpretation of test score changes in the district where the program was piloted. Attributing the entire increase (from the 23rd to the 59th percentile) in first-grade reading test scores in Lee County to the presence of assistant reading instructors in the classroom hinged on a misguided assumption.
In fact, many other factors were at work during this period which could partially explain the increase--e.g., reading scores were improving statewide, even in districts without assistant reading instructors, and Lee County's population was changing. The published research on the pilot program only examined changes in test scores during one school year and arrived at a much more moderate conclusion as to the program's effectiveness.
The outcome on a relative measure such as percentile ranking depends not only on how much better Mississippi students perform, but on how much better or worse students across the country perform. At the same time that Mississippi has been trying to improve its early childhood education efforts, other states have been trying to do the same. For Mississippi to improve its percentile rankings, positive changes in Mississippi student test scores must exceed changes in the scores of other states. For this reason, Mississippi's success should be gauged by an absolute measure, such as percentage of students mastering a given skill, rather than assessing basic skills achievement using only a relative measure such as national ranking.
The Reading Improvement Program may offer opportunities for success, but only if it is properly implemented. The law establishing the program mandated implementation plans, ongoing program evaluations, and uniform statewide training of both teachers and assistant reading instructors. Program administrators have fulfilled none of these mandates. Without plans, these administrators have not established in operational terms what the program is supposed to accomplish. Without evaluations, the administrators are unable to separate assistant reading instructor classroom practices that work from those that do not and direct program resources accordingly. Without proper training, assistant reading instructors cannot provide the assistance necessary for attaining positive program outcomes. Adequate training is especially critical given that most assistant reading instructors have a high school education and no experience in classroom instruction.
When Mississippi first adopted the Reading Improvement Program, the State Department of Education devoted considerable time and energy to its implementation. The department developed an in-depth orientation program, held regional workshops, and attempted to ensure rigid adherence to all components of the law. However, in the mid-1980s, before the elements were fully executed, the department's support for the program collapsed. By 1994, the Reading Improvement Program had lost its identity. Assistant reading instructors had become just another classroom resource. While the department continued to issue program guidelines and to provide training to assistant reading instructors and their supervising teachers on an "as requested" basis, the State Department of Education no longer ensured adherence to the program's statutory requirements, including those requirements upon which program funding is contingent.
Summary Questions and Answers Regarding the Reading Improvement Program
How much money has the state spent on the Reading Improvement Program?
During the eleven years of program operation, the state has expended $340,105,672 in Minimum Program funds and an estimated $18.75 million in local school district funds (for the five years for which data was available for PEER to use in estimating local costs).
Has the Reading Improvement Program been effective, especially in improving reading?
No, the program has not been effective in achieving its three primary objectives of increasing standardized test scores above the national average (50th percentile), reducing the percentage of pupils retained in the primary grades, or decreasing the student dropout rate. In fact, since the Reading Improvement Program began, statewide standardized test scores in reading have remained below the national average, the percentage of pupils retained in the primary grades has remained relatively constant, and the student dropout rate has remained constant in the lower grades and increased in the higher grades.
How do districts utilize their assistant reading instructors? Do districts use them as substitute teachers?
Local districts use their assistant reading instructors in a variety of roles, including reinforcement of basic skills and performance of clerical duties. Most of the districts in PEER's district review use the assistants as substitute teachers on a routine basis.
Are relevant parties complying with program statutes?
No, both the State Department of Education and the local school districts have failed to comply with provisions of the Reading Improvement Program law regarding evaluation, program accountability, and training. Further, the state department has not enforced the law's preconditions to program funding.
What is the potential effectiveness of the Reading Improvement Program?
The potential effectiveness of the Reading Improvement Program as implemented is minimal. The concept itself may have potential for improving student achievement, but moving from a promising concept to a program that substantially improves student achievement would require extensive planning, intensive staff training, and continuous monitoring and adjusting at the state and local levels.
Policy Options
While program improvement is one response to the problems described in this report, other options are available to the Legislature. PEER offers the following four policy options: