“There are three kinds of lies: lies, damn lies, and statistics.”
The Year Two Mid-Year Evaluation Report on the Baltimore County Public Schools STAT (Students and Teachers Accessing Tomorrow) initiative was recently released by the Johns Hopkins School of Education’s Center for Research and Reform in Education, and the resulting 69-page document would not disappoint Mr. Twain.
The report, as one might expect from JHU, is clearly written and reasonably thorough, with data parsed, presented, and charted as needed. The report opens by explaining that its purpose is to evaluate “implementations and outcomes” of the STAT program, “…relating to the goals of improving student achievement and preparing globally competitive students” (page 3). However, the very next paragraph clarifies that no, the report “does not examine the achievement of student outcome goals” (3) but rather presents information on the level of professional development offered and a host of “measureable outcomes” (3) from classroom observations. The report offers nothing about pedagogical effectiveness, the thing that actually improves student achievement. It also leaves many larger questions unanswered, however, providing a scrim of meaningless data to stand in as proof of effectiveness for a pedagogically dubious program.
The information on professional development in the STAT program was gathered through teacher surveys, and the results are obvious: there has been additional and broader professional development opportunities provided to teachers in Lighthouse Schools, and a majority of teachers have taken advantage of those opportunities. It would be foolhardy to roll out a multi-million dollar initiative like STAT without some kind of training, and the report finds that yes, there has been training offered, in large, small, and one-on-one settings. However, the stickier questions are not even asked: what kind of professional development was completed? How effective was it for classroom practice? What were the goals? How were they met?
In the section on measurable outcomes, the results of several classroom observations provided data on classrooms, teacher practice, digital content, student engagement, and P21 skills. Now, just because something can be measured does not make it a valuable metric. Take this example from the classroom environment findings: a “majority of classrooms observed in fall 2015 were physically arranged to support collaborative learning, displayed materials to support independent thinking to some extent, and had materials referencing the general subject or content area being taught” (25). What is described here is basically a standard classroom; this is expected practice in K-12 environments, as no caring teacher anywhere ever left a drab room of blank walls when working with children. This so-called “measurable outcome” tells nothing about STAT effectiveness; it’s a good bet that a majority of classrooms were that way before the program even existed. What was interesting in this section, however, was the finding that “students may be less likely to move around the room…considering the availability of information and resources accessed through devices” (25). This is certainly not a positive finding, though proponents of the “just ask Siri” school of research might disagree. What is implied here is that students do not move around much, as they supposedly can get what they need from the screen in front of them. This is not school, this is training for dystopia.
In examining teacher practice, the report found that “nearly all classroom teachers exhibited coaching behavior with students at least occasionally” (28). This is also a measurement of little meaning, as nearly all teachers who work with students in general spend some amount of time in coaching behaviors, teaching behaviors, and other required classroom roles. Maybe a few might hide behind their desk all day, or perhaps even under it, but these metrics were not included.
Perhaps the most useless metric in the entire report is the one involving digital content. The information was provided by Engrade, the McGraw-Hill property that created the software platform on which BCPS One sits; it is clear they have been logging a copious amount of student data, as they regurgitated some of it for the report to state the obvious: teachers and students in Lighthouse Schools are accessing digital content more frequently. The creation of teacher tiles (program links) for BCPS One increased; teachers in Lighthouse Schools are almost certainly required to be using the platform, so it is little surprise that they have been. What is surprising is the equating of “student engagement” with “increased student tile views within BCPS one” (47). Essentially, there have been more teacher and student clicks (of a mouse or browsing button), which tells absolutely nothing about the quality of material that is being clicked upon. Maybe it’s whack-a-mole. Maybe it’s spam. But hey, there’s a lot of clicking going on, so it must be good.
Measuring teacher and student clicks and passing it off as a useful metric is absurd. Clicks tell nothing about quality of materials used or quality of learning outcomes; this is a prime example of being data rich yet content and context poor.
The final section of the evaluation examined the use of P21 skills, which include “problem solving, project-based approaches to instruction, inquiry-based approaches to instruction, and learning that incorporates authentic/real world contexts.” It is important to stop for a second here and note that these ideas do not need to be branded with the Partnership for 21st Century Learning “P21” moniker. These ideas are not new to the 21st century and stretch back to the truly innovative theories of John Dewey and genuine progressive educational thought (which should not be confused with modern “progressive” education that advocates high-stakes testing and computer-drive personalized learning). The Partnership for 21st Century, a lobbying group for educational technology business interests, has glommed onto these ideas with the hope that they will lend some veracity to their organization. They don’t.
It is interesting to note, however, that the STAT report found that “P21 skills were least frequently observed overall” (42) out of all the metrics examined; perhaps the classroom focus has relied too much on technology and devices, crowding out more pedagogically effective methods such as student collaboration, problem-based learning, and other more engaging practices.
It is also worth noting that for an APA-style document, the year two midyear STAT report does not present a single reference or citation. Perhaps this is by design or request, or perhaps it is because the ideas that underpin the STAT initiative have a poor or nonexistent research base. The report presented a whopping three sentences of recommendations for improvement of the program, to include a focus on professional development “specific to desired teaching and learning activities that are less frequently practiced” (48) and a clarification of the role of the STAT teacher.
This issue of clarification was raised by a section in the report that noted a theme of lack of trust of the STAT teacher from some teacher survey responses. A few survey responses were quoted, which included: “the STAT teacher at our school has become evaluative and administrative in nature. It’s very clear that things shared/things seen in classrooms are shared with administration and hold weight”; “many teachers are concerned as to whether STAT teachers are going back to administrators and telling them about problems in the classroom. Are they judges or mentors?”; “she does not keep confidentiality about what we are working on…I am NOT going to ask for help because it is reported to the principal and spoken about later as a weakness” (22). These comments speak volumes about what is left unsaid by the STAT report—the BCPS administration does not operate the program on a principle of support but rather on one of threat and expected compliance.
As requested by a reader in the comments, you may also be interested in the complicated relationship between those who wrote the STAT Evaluation Summary and those who pay these same evaluators, otherwise known as conflict of interest: