2014 Bunkum Honorees:
The ‘Class Size Reductio ad Absurdum' Award
Comparing international test scores and drawing ominous conclusions is quite the rage. Also, as GEMS Educational Solutions found, it is a great way to garner credulous coverage from The Economist and the BBC. All that’s needed is a pile of data and a mathematical model, and one can do creative things like rank countries’ educational systems based on their “efficiency.” It apparently matters not how much sense it all makes, as long as it can be puffed up with something that sounds sufficiently intimidating, such as a stochastic frontier analysis, to lend an air of gravitas to an inherently silly idea. So armed, GEMS set out in The Efficiency Index to rank 30 countries. When the data emerged from the GEMS statistical grinder, researchers concluded from the resulting mince meat that to get a 5% increase in PISA scores, teacher wages would (on average) have to go up by 14% or class sizes would (on average) have to go down by 13 students per class.
The most entertaining part of this report is the efficiency index itself, which purports to list optimal wage levels and class sizes for each country. For four countries, the optimal class size is estimated at fewer than two students per teacher. The teacher salary part of the report is almost as risible. The Swiss, we discover, should cut teachers’ wages nearly in half to achieve that nation’s “optimal” teacher salary. Indonesian teachers meanwhile would see their wages triple. As explained by our reviewer, City University of New York economist Clive Belfield, such anomalies expose the weaknesses in each of the study’s three key elements: “the output measure is questionable, the input measures are unclear, and the econometric method by which they are correlated does not have a straightforward economic interpretation.”
Meanwhile, even those of us in the research community who have long pointed to the benefits of (smart) class-size reduction can savor the irony of an “efficiency” report suggesting that two students is an optimal class size.
The ‘What the World Needs Now is Choice Sweet Choice' Awards
The past decades have seen school choice expand through charter schools, vouchers, tax credits (neo-vouchers), and various other mechanisms dreamed up after feverish evenings of reading Milton Friedman. While it’s true that all this choice has increased systemic inequities even as it has failed to improve educational outcomes, that’s no reason to stop or slow down—at least not according to two Bunkum-winning reports: Expanding the Education Universe: A Fifty-State Strategy for Course Choice from the Fordham Institute, and Federal School Finance Reform: Moving Toward Title I Funding Following the Child from the Reason Foundation.
The Fordham report urges policymakers to bring the blessings of market competition to the selection of classes, envisioning a future when students design their own selection of online and off-line classes, offered by a variety of public and private providers. Unfortunately, the report makes no effort to actually evaluate the merits of the idea. It “assumes, without solid evidence, that course choice, electronic educational provisions, and the like are viable, effective, and proven methods,” according to reviewers and University of Southern California scholars Patricia Burch, Jahni Smith and Mary Stewart. “Accordingly, the piece rests entirely on assumptions and assertions.” The reader is left to ponder the next step after course choice. Perhaps our children might purchase individual lessons (and grades) through a bidding process on eBay?
Or they can just take federal Title I funding and head off directly to a private school, as proposed by the Reason Foundation in its report Federal School Finance Reform: Moving Toward Title I Funding Following the Child. Those lost in the antiquated “Good Ole Days” of the Great Society may have mistakenly thought that the purpose of Title I was to provide equal educational opportunities via compensatory services for needy children. It turns out that it’s really about school choice, if one takes the perspective of the Reason report, which argues that Title I funds, and other funds that states and districts may wish to contribute, should be distributed as a form of voucher (this is one version of what is called, “Title I portability”).
Under the current system, Title I is the federal government’s primary way of assisting schools serving large numbers of students meeting Title I eligibility criteria. So moving money away from those schools of concentrated poverty may seem counter-productive to the untrained eye. It might also seem odd to be telling low-income families that the way for them to receive better educational opportunities is to take $2,100 or so, somehow supplement it with their own money, and find a private school. Reviewer Gail Sunderman, a senior research scientist at the University of Maryland, concluded that the report’s recommendations are “likely to exacerbate existing inequities between schools within the same district rather than improve them.” But surely Sunderman and those who share such concerns simply fail to appreciate the miracle of school choice.
The ‘Back-Tracking via CTE' Award
To Lexington Institute for Updating Career and Technical Education for the 21st Century
The U.S. has a long history of tracking low-income students and students of color into dead-end vocational classes that prepare them for neither college nor a career. Every so often policymakers notice this, wring their hands, and say “never again.” Yet hope springs eternal in the human breast. So we can trust that a new generation of politicians will pop up with “innovative” ideas about how to create vocational tracks that produce completely different results.
Not ones to disappoint, a collection of think tanks and elected officials of both major parties are now promoting the idea that Career and Technical Education (CTE) schools will prepare their students for jobs in our New EconomyTM. We encourage the Lexington Institute to accept its Back-tracking Bunkum Award on behalf of the many other think tanks and policy wonks, across the centuries, that have discovered and rediscovered this most derivative of ideas. The Lexington report, Updating Career and Technical Education for the 21st Century, offers advice on how to unleash CTE’s potential to meet the needs of employers and employees alike in our rapidly changing economy. It is built on the Petrillian assumption that we should separate academically talented children from those in need of a non-academic alternative.
Reviewers Marisa Saunders and Jaime del Razo, both of the Annenberg Institute for School Reform at Brown University, concluded that the report manages to over-reach and under-reach simultaneously: It uses “a few poorly developed examples to make broad claims about key attributes of successful programs;” at the same time it does not capture the potential of high school CTE programs to bridge between academic features and the potential to use CTE approach to make learning more relevant and engaging. By replicating the harmful mindset that career education is somehow in conflict with college preparatory curricula, thus requiring separation of academic and voc-ed students, the report reinforces longstanding divisions by social class that funnel students from lower socioeconomic backgrounds disproportionately toward a vocational track, while affording those from higher socioeconomic backgrounds greater access to higher education and the higher incomes that come with it.
The 'It’s Never Too Early to Revise History’ Award
To Sonecon, Inc. for The Economic Benefits of New York City’s Public School Reforms, 2002-2013
Michael Bloomberg hadn’t yet left New York’s City Hall before a series of publications were released that put a rosy gloss on his administration’s educational record. One such report, The Economic Benefits of New York City’s Public School Reforms, 2002-2013, was produced by Sonecon, Inc. a Washington, D.C., economic advisory firm that claims to apply “methodologies that produce analytically unassailable results.” Sonecon’s chairman, Robert Shapiro, authored the report along with Kevin Hassett, the American Enterprise Institute scholar who co-authored the visionary (i.e., delusional) 1999 book, “Dow 36,000.” The Sonecon report credited New York City education reforms during the Bloomberg years with boosting the City’s economy to the tune of $74 billion. “While such estimates are always an exercise in some level of speculation, this report relies on highly inappropriate assumptions to reach its conclusions,” reviewer and NYU economics of education professor Sean Corcoran explained. “Specifically, it attributes all gains in high school completion and college enrollment to the reforms, applies national statistics on earnings and college completion to the marginal graduate in NYC, and extrapolates cross-sectional associations between graduation rates and home prices at the zip code level as the causal effect of higher graduation rates.” For example, breaking down the report’s math, Corcoran finds that the estimated impact of the Bloomberg-era reforms on property values is equivalent to “two-thirds of the entire increase in residential property values between 2007 and 2013.” Finally, let’s not forget that the Bloomberg-era reforms were preceded by a landmark court ruling which, Corcoran notes, “helped drive a large increase in state resources for the City’s schools”—yet that case is not mentioned in the Sonecon report. The “back of the envelope” estimates that the Sonecon report makes, Corcoran concludes, “are pure fantasy.”
And finally…
The ‘Fractured Fraction Award for Using Erroneous Numerators and Denominators to Get Predetermined Results'
To School Choice Demonstration Project and University of Arkansas Department of Education Reform for Charter School Funding: Inequity Expands
To University of Arkansas Department of Education Reform for The Productivity of Public Charter Schools
The University of Arkansas’ Department of Education Reform (EDRE) wins our Grand Prize for two reports. The first,Charter School Funding: Inequity Expands, argues that charter schools are underfunded. The second, The Productivity of Public Charter Schools, argues that despite being underfunded, charter schools still manage to produce better outcomes than district schools. The reports are impressive indeed—as long as we’re collectively willing to overlook the researchers’ bungling of their calculations of both the numerator and the denominator of their equation.
The charter school funding report was reviewed by Rutgers University school finance professor Bruce Baker. He noted the report authors’ misunderstanding of intergovernmental fiscal relationships, and he explained how this misunderstanding produced an erroneous assignment of “revenues” between charters and district schools. A district’s expenditure is frequently a charter’s revenue, since charter funding is often received by a pass-through from district funding. Thus, the EDRE report doubles up on the assignment of revenue to the public school districts: their own plus the charters’. The Arkansas authors also fail to take into account the fact that districts often retain responsibility for direct provision of services (such as transportation) to charter school students. Perhaps it should come as no surprise that the report suffers from vague documentation of its research methods and data sources. For example, it treats “all revenues” (not defined) as expenditures (which they assuredly are not). When numbers were not handy, report authors drew on murky “additional data sources.”
A productivity calculation looks at results per expense, so the enigmatic revenue calculation of the first report is used by the EDRE authors as their denominator. Their numerator computation is not much better, since it’s cobbled together with entirely inappropriate comparisons of student population characteristics.
That second report was reviewed by University of Colorado Boulder research professor Gene Glass, who found that it used state average NAEP scores without bothering to consider the well-known population differences between charter schools and non-charter public schools on demographic variables such as poverty (free lunch eligibility) or special-needs status. As Glass asks in his review, “If one is calculating ‘bang for the buck,’ what is left if neither the bang nor the buck can be believed?”
For these stunningly incompetent analyses, the University of Arkansas’ Department of Education Reform has thoroughly earned the 2014 Bunkum Grand Prize for shoddy research.
Posts
No comments:
Post a Comment