How innovations are monitored and measured
Conventional approaches to evaluation are not readily adaptable
to the evaluation of systemic educational reforms because of the uniqueness
and complexity of the latter. A reform needs to tackle simultaneously multiple
elements to be effective in transforming the system according to the reform’s
intended goals. Program leaders and evaluators often have a very short time
frame to identify and analyze what is really taking place within a reform
and to show the impact of the innovations in transforming the system so
that attribution can be clearly established.
Given the challenges of evaluating a systemic educational reform, assessment
has been an essential driver of the RCSE systemic reform, and multiple indicators
of change are used for continuous feedback for decision making and policy
development. Each of the RCSE projects establishes multiple metrics and
benchmarks early in the strategic planning process against which to measure
progress in achieving the project’s goals, and its value-added to
the systemic reform of the K-16+ educational system. Quantitative and qualitative
approaches are used to look at the changes, and the information provided
by the different assessment strategies are being used to monitor progress,
design quality control mechanisms, and retool the initiatives as necessary.
The PR-SSI, the K-12 science and mathematics reform initiative, for example, looked at student academic achievement to “take the temperature” of its overall K-12 reform strategy while focusing on the most important element of the system: the students. The PR-SSI looked at student academic achievement from three different yet complementary perspectives: the classroom level, the reform program, and the system. The teacher, who learned about alternative assessment strategies during professional development sessions designed for this purpose, looked at student performance in the classroom and used this information to refocus the content and strategies of daily lessons to strengthen student learning. The program looked at student academic achievement trends across schools and across cohorts of participating schools. The system used data provided by entities such as the College Board to look at the impact of the reform on the overall system.
On the other hand, the Puerto Rico Louis Stokes Alliance for Minority Participation (PR-LSAMP) Program, which aims at improving the effectiveness and efficiency of undergraduate science, technology, engineering, and mathematics (STEM) programs, uses the following metrics to measure progress of the reform: retention and graduation rates of students, number of years to obtain a baccalaureate degree, number of students that enroll in graduate programs and successfully complete a doctoral degree in a STEM discipline, and the Index of Course Efficiency (average number of times students must take a course to obtain a satisfactory grade –A, B or C). This set of measures that assesses academic progress has direct impact on the cost of higher education and, for this reason, has a major impact on the academic CEO’s and their setting of institutional priorities.