Call for Umpire Accountability & the 97% Plate Score

With fan and player fury on the rise, Major League Baseball's internal system for evaluating home plate umpires—called Zone Evaluation or Z-E—holds that MLB's umpire staff is 97%+ accurate in calling balls and strikes, all while wall-to-wall extracurricular studies place umpire accuracy at relatively lower figures from the 80 percentile-range to the low 90s.*

Why does such a discrepancy exist and why isn't Z-E's 97%+ score better known or accepted?

For one, Zone Evaluation rarely, if ever, sees the light of day outside of MLB's internal umpire operations, while the FoxTrax, K-Zone, Brooks Baseball, and other demonstrations are easily accessible and common to the game. If one system monopolizes the public eye, as non-ZE components do, one could surmise that the public would be more likely to adapt to it.

*You'll notice that the first paragraph ended with an asterisk. That's because 97% was the Z-E score from 2016—that's how closely guarded the secret is...the more recent figure isn't public (though one could argue the 2016 figure isn't exactly common knowledge, either). And before anyone calls and asks how I got that number—which, again, is usually a closely guarded trade secret—it was provided to Gil (LeBreton, of the Fort Worth Star-Telegram), who perhaps got it from HBO's Real Sports with Bryant Gumbel, who perhaps got it from MLB (or any iteration thereof).

That's right, for one shining moment, the guarded Z-E score of 97% from 2016 was public-facing.

K-Zone says it's a ball, but f/x disagrees. Why?

With fandom's increasing call for "umpire accountability," it bears note that the league constantly evaluates its umpires through Z-E, on-site supervisors, or the Supervisor Umpire Review and Evaluation system (the SURE system), to name a few methods, and if umpires fail to meet standards, there are mechanisms to address this. Naturally, this entire process is behind-the-scenes, leaving the common fan to decry a supposed lack of accountability when, in private, umpires are held to an internal standard.

In regard to Z-E's 97%—the league's evaluation of umpires—HBO's report clashed with MLB's ZE number in rather short order, seemingly throwing out MLB's baby with the accountability bathwater.

It's not personal. As Boyle, O'Rourke, Long, and Pavlidis wrote in 2018, "There are myriad issues with implementing a ball-and-strike-calling system like this for real games. The primary issue is that using machine measurements to call balls and strikes will simply shift disagreements with the call from the umpire to the machine, or to the machine’s operators."
Related Link: Robo Strike Zone: It’s Not as Simple as You Think (Baseball Prospectus, 1/29/18)
Related Post: Gil's Call: The Blame Game (Umpire Scapegoating) (8/8/14).

It's all about answering the following question:
Did a sphere traverse a perpetually in flux prism?

The conflict, summed up in nary 31 words, via LeBreton: "MLB claims its umpires call 97 percent of balls and strikes correctly. But according to [Yale University Economics Professor Toby] Moscowitz and HBO, the study showed that only about 88 percent of the calls were accurate."

The 2019 Study: Most recently, Boston University Master Lecturer Mark T. Williams in the Boston University School of Business' Finance and Economics Department authored an article entitled, "MLB Umpires Missed 34,294 Ball-Strike Calls in 2018. Bring on Robo-umps?"

The Trend Toward Good: League-wide, Williams concluded that umpires are trending toward greater plate accuracy, finding that the league-wide "bad call ratio" decreased every year during the period for which data was processed (2008-2018), though Williams cautioned that instead of championing this trend to demonstrate improvement in the officiating ranks, MLB would be wise to continue to implement technology and other approaches to drive the error rate further downward.

We discussed the discrepancy in 2018.

With a variable Williams' team termed a "bad call ratio" (BCR) of 9.21% in 2018, the inverse—90.79%—best corresponds to the concept of a plate accuracy score...but the Williams-found 90.79% score is still well below 2016's Z-E of 97%—or even 2012's Z-E of 95%.

Why? The public numbers and the private numbers are based on the same pitch-tracking data, so there must be some difference in how the two sectors interpret the data. What are the rules?

We previously discussed the issue of this discrepancy as a player-umpire disconnect in October 2018, but there's more to it and a new study of ball/strike decisions renews interest in the subject with what can best be said is a frustrating gap between officiating-centric circles or the league itself and, well, at-large fans of the game. The rules difference is simple: the various parties aren't speaking the same statistical language.
Related Post: UEFL f/x vs K-Zone and the Player-Umpire Disconnect (10/4/18).
Related Post: Analyzing Strike Zone Analysis - Not So Easy or Simple (10/27/16).
Related Post: Dude, What Happened Last Night? About Pitch f/x Error (8/30/16).

Because of William's conclusion of umpires trending toward greater accuracy (via a trend toward lower error), we can be surmise that 2017 and 2018's ZE scores likely improved over 2016's 97%, which itself was an improvement over the 95% Z-E score in mid-2012.

CloseCallSports is attempting to procure an interview with Williams to discuss his findings and implications for baseball officiating, which we hope to bring you in a follow-up article and, hopefully, podcast episode.