Two Different Ways To Be Wrong: Sequencing and Bad Projections

Baseball analysts are frequently wrong. Everyone who writes for this website picked the Nationals to win the NL East, for example. We also split between Detroit and Cleveland for the AL Central, with no votes for the Royals. Predicting baseball is difficult because it’s a game with many variables and lots of randomness. It is probably very unlikely that a world class chess player would lose to a novice in any one game, but it’s especially unlikely they’d lose more often than not over 100 games. In baseball, there are so many things impacting single games, and there are 2,430 games, so predicting a full season is especially challenging.

And as has been noted in a lot of places, we didn’t do a great job predicting the 2015 season. The Rangers, Astros, Blue Jays, Royals, and Mets weren’t exactly consensus playoff picks. Whoops!

This has led to plenty of push back against sites like ours and serves as a criticism of the work we do when it comes to predicting the game. Presumably, if we can’t accurately predict which teams will be good and bad, you might not want to put a ton of stock in what we’re saying. Surely, no reasonable person would hold anyone to a standard of perfection, but whiffing often can be a sign of a flawed process.

I’m not going to litigate exactly where our projections may or not be flawed in this post, but rather, I want to separate out two very different components of overall wrongness. In fact, there are essentially two ways in which our overall estimates of the league can be incorrect and you should understand the forces at play when determining how much stock to put into the work done here, and at other sites like Baseball Prospectus.

Let’s build an example. Let’s say we predicted the Mets would win 81 games and then let’s say they wind up winning 93. That was a bad prediction. There’s no official range of acceptable failure, but if they won 83 games, it wouldn’t make a lot of sense to criticize the process. But 12 feels like a big gap. Let’s say we predicted the Twins would win 74 and they win up winning 84. Ten also feels like a big gap.

In both cases, our methodology got it wrong back in April. Now any good statistician will tell you that if you make 30 predictions, it’s likely that some will miss high and some will miss low even if you have an excellent model. There are unpredictable and random events that influence the game. But let’s acknowledge that even if you expect to miss on  some predictions, missing is still missing and a model that has 10-15 game misses could have flaws, so we should explore.

In reality, the ways in which we failed to predict both outcomes are different and they can show us different things about the nature of forecasting.

The Mets, A Story In Incorrect Projections

The Mets weren’t supposed to be a bad club going into the season, but they were pegged to be relatively average. That wasn’t just what our model said, it was the general consensus of the baseball world. Not only did FanGraphs writers pick the Nats over the Mets, everyone at ESPN did as well. Yet the Mets are going to win about 93 games and the NL East. Setting aside the Nationals’ collapse, how did the Mets beat their projections? The projections failed to properly forecast their performance.

This happened in a variety of ways. The Mets got better performance from certain players who we expected to be on their roster, such as Jacob deGrom (2.6 projWAR to 5.1 WAR) and Curtis Granderson (1.7 projWAR to 4.9 WAR). The Mets also added players to their roster who we did not include in our preseason forecasts, like Yoenis Cespedes (2.7 WAR).

In other words, we failed to predict the Mets success because we 1) didn’t do a great job predicting the individual performance of key Mets players and 2) didn’t properly include the performance of the players who joined their organization along the way.

On point #1, that’s our mistake. We generally do pretty well predicting individual player performance, but sometimes we don’t and it can have an overall impact. It’s not possible to get every player right, but certainly we do better sometimes than others.

On point #2, that’s pretty hard to build into a forecasting system, but it is a flaw of the process. We predict 90 wins for one club and 80 wins for another, and even if we nail both clubs through July 1, both are likely to modify their rosters in meaningful, and someone predictable ways. We obviously can’t predict that Cespedes would go from the Tigers to the Mets in April, but the idea that good teams will get better and bad teams will get worse is a thing we can anticipate. Fault us or don’t, but it’s part of the problem.

So in general, we got the Mets wrong because we got the Mets players wrong.

The Twins, A Story In Sequencing

Now certainly, there are aspects of the Mets story woven into the Twins story. Of course we missed on individual players and didn’t get the playing time exactly right. But in general, the individual performance of the Twins is right in line with our preseason expectations.

Wait, you’re asking, how is that possible if they’re going to win 10 more games than we predicted? It’s possible because the Twins are generally hitting and pitching on part with our expectations about their abilities, they have simply benefited from clustering their hits and outs in helpful sequences.

You’re probably tired of hearing about sequencing, but it’s an important thing you shouldn’t ignore. Imagine a team that has a .340 wOBA. You can either hit .340 every game or hit .400 in some and .280 in others. You can break this down into innings or week, it doesn’t matter. The idea is that clustering hits leads to more runs than spreading them out, and then inverse is true for pitching. A poor hitting team can theoretically score a lot of runs if they make sure all of their hits stack on top of each other.

This doesn’t mean the Twins didn’t earn their wins, it just means that the flaw in the model is different in their case.

In the Twins case, we accurately predicted their inputs to the model, but the way the season turned their inputs into outputs didn’t happen the way we expected. Based on everything we think we know about baseball, the Twins aren’t actively controlling that and teams can’t learn to beat those odds in a meaningful way. Perhaps we’re wrong about that, but generally we think it’s basically just randomness. Or at the very least, it’s accidental and fragile.

If I tell you a team is going to have six singles, two doubles, a homer, and one walk in a game and will allow three runs on defense, would you give me credit if that exact thing happens? You would, probably. But what if I told you they would win 5-3 and somehow lost 3-2? You might not want to give me credit, even though the majority of teams that have those hits in a game a score 5 runs (numbers are illustrative only).

A single-single-HR is worth three runs but a HR-single-single is worth one. Order matters, but controlling order does not appear to be a skill, so predicting it is essentially impossible. Maybe we are wrong about the assumptions, but it doesn’t seem like anyone should have been able to predict the Twins would be the team to hit and pitch poorly while also winning when most teams like that don’t.

We predicted they would play like a .457 club, and according to our context neutral BaseRuns model, they have played like a .452 team. So in general, we got the Twins right, but put the pieces together incorrectly.

Conclusion

The take away point is not that our models are perfect and you shouldn’t view them with a critical eye. The point is that there are different ways to be wrong about a baseball team’s season. It’s wrong to lump the Mets and Twins together into the same bucket of incorrect predictions. As an analyst, I would stand by the Twins projection as a correct one. The Twins played the way we expected, they simply did so in the right order (for them). The Mets actually played better than we thought they would.

Team projections are tricky because there are two layers. You have to get the players right and then you have to get the way the plays fit together right. It’s a hard business and you shouldn’t have illusions about our abilities. If you make plenty of predictions about baseball, you will make many incorrect predictions because it’s hard to predict baseball.

That’s okay. Most of us are here for fun and we’re doing this to better understand the game. For clubs, there are millions of dollars at stake, but for the average person, it’s all about enjoying the game in a certain way. Lots of people who read this site like this kind of statistical approach. If you don’t, that’s fine. It’s not for everyone. We don’t hate your team or augment the projections to tell a good story. They’re a tool and we enjoy them, even if they sometimes don’t perform very well.