A Compliance and Management Review of the Dyslexia Pilot Programs of the Mississippi Department of Education
Executive Summary
Introduction
MISS. CODE ANN. Section 37-23-15 (1972) authorizes the Department of Education (MDE) to adopt pilot programs for testing eligible students in public schools for dyslexia and related disorders. The Legislature established the pilot programs in response to the growing concern of the federal government and educational community regarding learning disabilities that negatively affect students in the public education system.
For purposes of this review, PEER focused on dyslexia pilot program grants awarded by the department for FY 2005 (July 1, 2004, through June 30, 2005).
The Department of Education established the dyslexia pilot programs as a competitive grant program for school districts beginning in FY 1997. Grant funds may be used for consultant fees, supplies, and travel. Salaries, benefits, and salary supplements are non-allowable expenditures for the pilot programs. Program funds may not be used to support teacher units.
To participate in the pilot programs, school districts must submit responses to the department’s annual request for proposals that it mails to all school districts. The department, through a selection committee, evaluates the proposals to determine which will be funded. Once the department receives its state funding for the pilot program for the upcoming fiscal year, the department staff determines the grant fund amounts and awards grants to the districts selected. From FY 1997 through FY 2006, the Department of Education funded 181 grant proposals to deliver services in sixty-one different school districts. Since the pilot programs’ inception, the state has invested $2,218,737 in providing services to students with dyslexia.
Conclusions
According to Department of Education staff, the department uses its regular grants process to award dyslexia pilot program funds to school districts. The FY 2005 request for proposals (RFP) for the pilot program stated that priority would be given to proposals that “hold the most promise for successful implementation and raising student achievement.” The department’s selection committee evaluated proposals submitted by interested school districts in response to the FY 2005 RFP. The five committee members independently scored the proposals received in accordance with selection criteria contained in the RFP, with additional points available for replicability and sustainability, for a possible 110 points.
No Documentation of Determining a Cutoff Score
Because the Department of Education does not document its rationale for establishing the cutoff score that committee members use in awarding grants for a given year, a third-party reviewer cannot recreate the process used for establishing that score.
For Fiscal Year 2005, the evaluation committee recommended that thirteen of the twenty-eight school districts that submitted proposals receive dyslexia grants.
To make award decisions, the evaluation committee utilized an average “cutoff” score of 95 to select districts to receive grants for FY 2005. During the past five fiscal years, cutoff scores utilized by evaluation committees have varied from 85 to 95. According to the department’s Dyslexia Coordinator, the department selects a cutoff score that the staff believes is appropriate for a particular fiscal year.
While the department has a legitimate need for utilizing a cutoff score for selecting meritorious proposals and the cutoff may justifiably vary from year to year, PEER determined that the department does not document the procedures used to guide the establishment of the cutoff score. Therefore, a third-party reviewer such as PEER could not recreate the process used for establishing that score. In the absence of documentation of the selection process, it is not clear whether the cutoff scores truly result in those proposals with “the most promise” receiving grants for dyslexia pilot programs.
Partial Funding of Winning Grant Proposals Compromises Pilot Program Concept
With one exception in the last five fiscal years, the Department of Education has not fully funded proposals of districts selected to receive grants, thereby compromising the utility of the pilot programs in identifying best practices.
For FY 2005 pilot program grants, the department chose to fund 85% of each participating district’s requested amount, resulting in total awards that amounted to $236,815, the approximate amount appropriated by the Legislature for the dyslexia pilot program grants. Over the past five fiscal years, the department has chosen to award grants to all but one of the pilot programs at less than 100% of the amounts requested, with percentages ranging from 46% upward.
The department applied these percentages with no evidence in the files that the selection committee had determined the effect that a reduced funding amount would have on a district’s proposed dyslexia pilot program. With the exception of a grant to the West Point School District in FY 2006, the department has not fully funded any district’s grant request for the past five fiscal years.
In view of the fact that these funds for educational programs have been offered to districts since FY 1997, PEER contends that the department should have been able to establish a monitoring and evaluation program capable of determining which programs are stellar educational programs, so that at some date in the future, when dyslexia programs might be expanded, districts will know which programs have been the most effective. However, the department has never embraced the need for determining which programs are most effective, instead using funds appropriated by the Legislature and allocated by the department as grants to fund many proposals, possibly of varying degrees of quality, at less than the amounts of funding that might have been needed to accomplish the goals of the proposals.
While the department’s funding of specified percentages of the program’s proposed grant amounts may accomplish the department’s intention to fund as many proposals as possible within the constraints of funds appropriated by the Legislature and allocated by the department for dyslexia grants, such could alter the structure of the pilot programs and compromise the utility of the programs in identifying best practices.
Inadequate Evaluation of Dyslexia Pilot Programs
The FY 2005 RFP stated that the department would award grants to districts whose proposals raised student achievement. In an effort to accomplish this goal, the department required the participating districts to submit the following information:
Districts Did Not Measure Program Effectiveness Against Program Objectives
The Department of Education did not ensure that school districts that received dyslexia grants during FY 2005 measured their programs’ effectiveness against the objectives included in their responses to the department’s request for proposals, which was a condition of the grant agreement.
As noted above, the department requires districts to describe the specific objectives against which their programs will be measured. The districts are also required to describe the evaluation methods, including pre- and post-assessment, that they would use to determine the effectiveness of their programs.
The department did not ensure that districts complied with some of the evaluative requirements, as follows:
Department Did Not Evaluate Overall Effectiveness of Pilot Programs
For FY 2005, the Department of Education did not evaluate the effectiveness of the districts’ dyslexia pilot programs to determine whether the state’s investment in the programs actually yielded improved student performance.
Although PEER found evidence that the districts submitted quarterly reports and pre- and post-test results for FY 2005 to the department as required, PEER found no evidence that the department analyzed the submitted information and drew any conclusions as to the programs’ effectiveness. In addition to the fact that the department did not enforce requirements that districts provide certain evaluative information (see previous section), the department did not prescribe a standard format for the districts to submit pre-and post-test results of individual students; as a result, districts developed their own formats that varied significantly.
Without assuring that the districts compiled and submitted the information mandated by program requirements and submitted pre- and post-test results in a standard format, it would be difficult at best for the department to develop conclusions regarding the overall effectiveness of the dyslexia pilot programs. In the absence of an overall evaluation of the dyslexia programs, it is not clear how the department can make informed decisions in selecting districts to receive grants in the future.
Inadequate Administration of the Pilot Programs’ Grants
Because it reimbursed districts’ grant expenditures in FY 2005 without enforcing all requirements of the grant agreement, the Department of Education did not ensure that all dyslexia pilot program grant funds were properly spent or that participating students’ reading performance had actually improved. Also, the department did not utilize the audit provision of the grant agreement to ensure appropriate expenditure of grant funds.
PEER reviewed FY 2005 records of the Department of Education to determine whether the department enforced grant requirements and found that for some of the participating school districts, the department did not have some of the required information on file.
No Assurance of Expenditure of Funds for Program Purposes
Because it did not ensure that all participating districts submitted their end-of-the-year expenditure reports or supplied supporting documentation for expenditures, the Department of Education did not ensure that school districts participating in the dyslexia pilot programs during FY 2005 spent funds in accordance with grant requirements.
Although the department’s request for proposals (which is incorporated by reference into the grant agreement) required each participating school district to submit end-of-the-year expenditure reports by August 5, 2005, the Department of Education did not ensure that all districts submitted their reports by the deadline. The department received end-of-the-year expenditure reports from only four of the thirteen participating districts by the August 5 deadline.
Five of the nine districts that did not submit their end-of-the-year expenditure reports for FY 2005 received pilot program grants for FY 2006 regardless of the fact that they had not complied with the reporting requirements for the previous grant year. The Department of Education does not have a policy that requires a grantee’s compliance with all grant requirements in a previous grant year before awarding a grant for a subsequent year.
Failure to Enforce Some Grant Requirements
For FY 2005, the Department of Education did not ensure that one participating district complied with the requirement that all dyslexia grant funds be spent in their entirety or be returned to the department by August 5, the close-out deadline.
The department’s FY 2005 request for proposals for dyslexia pilot programs stated that one “critical participation requirement” of districts selected to receive dyslexia grants was that the districts would “expend program funds in their entirety according to the approved program budget.”
On November 28, 2005, PEER interviewed dyslexia program staff of the Greenwood Public School District and reviewed accounting records for the district’s FY 2005 dyslexia pilot program. According to the district’s accounting records, the district had expended only $10,705.70 of $22,440 in grant funds allocated to the district for FY 2005. The balance of the allocation, $11,734.30, remained in a district account and had not been returned to the Department of Education.
When PEER questioned the Department of Education’s Dyslexia Coordinator regarding the Greenwood district’s failure to expend all of its grant funds and file an end-of-the-year expenditure report, the coordinator stated that she had sent the district a letter on October 25 requesting the return of the unspent funds. The district did not take action to return the funds in response to the district’s letter. On January 11, PEER again discussed with the Dyslexia Coordinator the Greenwood district’s failure to return the unspent funds as requested by the department. The department sent the district another letter on January 25 requesting the return of $11,734.30 in dyslexia grant funds. The department received a check for this amount from the district on February 3, 2006, and deposited the funds into the state treasury.
No Auditing of Grantees
Following the close of FY 2005, the Department of Education did not audit grantees to determine whether their expenditures complied with grant requirements.
The “Grant for Specified Services” agreement between the Department of Education and districts that receive dyslexia grants states that the department “shall have access to, and the right to audit and examine any pertinent books, documents, papers, and records of grantee related to grantee’s charges and performance under this grant.”
In view of the fact that districts may request reimbursement for expenditures without providing supporting documentation and that at least eight districts failed to provide FY 2005 end-of-the-year expenditure reports to the department by the deadline, post-audit is particularly important. However, the department did not post-audit any school districts that received dyslexia grants during FY 2005. Thus, the department could not ensure that dyslexia pilot program funds were spent properly and resulted in improvements in students’ reading performance.
Recommendations