Replay History - Overturned Calls Outnumber Upheld

For the first time in Replay Review history, overturned calls outnumber upheld decisions, marking the first time in expanded instant replay's MLB history that over 50% of all Replay Review decisions have resulted in a call's reversal.*

For the first time in MLB history, RAP is below 50%.

Why is this? Time for a sabermetric analysis.

When baseball first introduced expanded replay prior to the 2014 season, the MLB Rules Committee noted that the new Replay Review system's purpose was to adjudicate the game-changing play and to correct the "obvious miss," noting that Replay Review was never designed to be used for the "bang-bang play." From the very beginning, Replay Affirmation Percentage (RAP) was designed to be a low number.

Ahead of the 2015 season, I wrote, "The sure thing of overturning an 'obvious miss' turned into somewhat of a coin flip because teams simply didn’t use the new technology correctly, perhaps because video coordinators, bench coaches and/or managers—notably those in Toronto, Tampa Bay, and the other 'lower half' teams [TOR & TB had low rates of success in challenging calls]—didn’t know what to look for, didn’t know the rules, or simply were unable to think like an umpire."

Upon further consideration, I will add another confounding variable: the "greed" factor, or desire to overturn a 50-50 call in order to secure an advantageous outcome. We've seen that a lot over the past few years, as evidenced by the 27.9% of calls in 2014 that resulted in "stands" outcomes, which is the literal definition and manifestation of the 50-50 play.

Are John Gibbons' Jays being 'greedy'?

On a more overtly "greedy" note, nearly one-fourth of all calls under Replay Review were confirmed in 2014 and, putting everything together, that is the biggest reason why we never before have been in the situation of having over 50% of reviewed calls be reversed: because teams knew not what to look for, lack of rules knowledge, incapability of umpire empathy, and/or the "greed" factor and "what if"-itis in which teams play faulty logic roulette in catastrophizing, "what if this ends up being a game-changing play?"

2014 ended with 47% of all Replay Reviews resulting in an overturned call, while approximately 49% of Replay Reviews resulted in a reversal in 2015. The 2016 figure presently is just north of 50%.

So when tmac sent me an e-mail regarding this RAP revelation and 50%+ overturn mark, I went to work and dove into Replay Review history. It is a short history, but its lessons are nonetheless notable.

Teams and umpires are getting better in replay.

As teams and umpires better learn how to use the system as intended, cumulative RAP is on its way down—slowly and tentatively, but nonetheless statistically significant. We're on pace for approximately 1,500 replays by the end of the 2016 regular season. Given that we had only 1,335 Replay Reviews in 2015 and 1,274 replays in 2014, it is apparent that the trend of subjecting more and more calls to video scrutiny is continuing, and teams are becoming better at choosing the right calls to review while forgoing a challenge of something I'll call "the obvious get," which as you might have surmised is the exact opposite of what MLB calls "the obvious miss."

Umpires are discussing calls more often.

Umpires, too, have become slightly more selective about reviewing calls and aren't sending "the obvious get" to New York as often as they had in 2014 and 2015 (statistics show that Crew Chief reviews, as a proportion of overall reviews, are slightly down in 2016). Similarly, a greater percentage of calls the umpires themselves initiate reviews for are being overturned, compared to the past two seasons. Note that this data has been normalized to account for the old Rule 7.13 HP collision replay procedure—initiated by the Crew Chief in 2014, but switched to a Manager's Challenge in 2015.

On a similar note, we also saw Replay Review overturn bona fide slide calls with more gusto at the start of 2016 than it had with home plate collision calls in 2014 and 2015.

Some calls, like this passing play, are missed.

So, the question on everyone's mind must be, "are umpires missing more calls these days?" And the answer is, as the aforementioned analysis indicates, more complicated than a simple "yes" or "no." By the strict definition of the word "more," an appropriate response would point out that while more calls are being overturned by replay this season, there will, all else equal, be more than 150 reviews by season's end than in 2015, and 275 more reviews than in 2014. In conclusion, the big takeaway is not so much that umpires are getting or missing more calls (though we have seen umpires derelict in their duties and replay used to correct these errors, but this isn't to imply a causal relationship), but that both teams and umpires are becoming more adept at the intended purpose of the Replay Review system.

Remember, RAP is designed to be a low number and its objective is to correct "the obvious miss" and to review the "game-changing play."

We break the story here because I predict that if this news item is reported in the traditional media, the lede will suggest something to the effect, "Replay shows umpires are the worst they've ever been." It is an irresponsible headline with no objective analysis just as "Umpires are better than ever" is an irresponsible conclusion. Without analysis, both are meaningless.

That is, unless you are able to reach in and pull out your QuesTec, Pitch f/x, and Zone Evaluation data that shows the average umpire's strike zone has steadily improved ever since the mid-2000s.

*Note regarding methodology: Through games played on 7/26/16, there were 926 total Replay Review decisions, with 462 upheld (Confirmed or Stands) and 464 reversed (Overturned) calls, meaning that the percentage of overall calls upheld by Replay Review dropped below 50% for the first time in MLB history. However, the minimum sample size for total Replay Review decisions used to compute this figure was 10 (e.g., this does not include the first few days of a young season in which, say, three calls were overturned and only two were upheld on Opening Day).