Sign In
  • Support FanGraphs
    FanGraphs Membership
    FanGraphs Shirts
    FanGraphs Mugs
    Gift a Membership
    Donate to FanGraphs
  • Games
    Ottoneu Fantasy Baseball
    Signup, FAQ, Blog Posts
  • Blogs
    Blog Roll

    FanGraphs
    Podcasts: FanGraphs Audio | Effectively Wild | Chin Music

    FanGraphs Prospects

    RotoGraphs
    Podcasts: The Sleeper and The Bust | Field of Streams | Beat the Shift

    Community Research

    Archived Blogs: The Hardball Times | NotGraphs | TechGraphs | FanGraphs+
    Archived THT: THT Live | Dispatch | Fantasy | ShysterBall
    Archived Podcasts: Stealing Home | Doing It For Bartolo | OttoGraphs | UMP: The Untitled McDongenhagen Project
  • Projections
    2023 Pre-Season Projections
    ZiPS, ZiPS DC
    Steamer
    Depth Charts
    ATC
    THE BAT, THE BAT X
    2023 600 PA / 200 IP Projections
    Steamer600
    2023 Updated In-Season Projections
    ZiPS (RoS), ZiPS (Update)
    Steamer (RoS), Steamer (Update)
    Depth Charts (RoS)
    THE BAT (RoS), THE BAT X (RoS)
    3 Year Projections
    ZiPS 2024, ZiPS 2025
    On-Pace Leaders
    Every Game Played, Games Played %
    Auction Calculator
  • Scores
    Today
    Live Scoreboard, Probable Pitchers
    Live Daily Leaderboards
    Win Probability & Box Scores
    2022, 2021, 2020, 2019, 2018, 2017...
  • Standings
    2022 Projected Standings
    2022 Playoff Odds, Playoff Odds Graphs
    ZiPS Postseason Game-By-Game Odds
    AL East
    AL Central
    AL West
    NL East
    NL Central
    NL West
  • Leaders
    Major League Leaders
    Batting: 2022, 2021, 2020, 2019, 2018, Career
    Pitching: 2022, 2021, 2020, 2019, 2018, Career
    Fielding: 2022, 2021, 2020, 2019, 2018, Career
    Splits Leaderboards
    Season Stat Grid
    60-Game Span Leaderboards (Special)

    KBO Leaders
    Batting, Pitching

    Minor League Leaders
    AAA: Triple-A East, Triple-A West, Mexican
    AA: Double-A Northeast, Double-A South, Double-A Central
    A+: High-A Central, High-A East, High-A West
    A: Low-A West, Low-A East, Low-A Southeast
    R: Appalachian, Gulf Coast, Pioneer, Arizona
    R: Dominican
    WAR Tools
    Combined WAR Leaderboards
    WAR Graphs
    WPA Tools
    WPA Inquirer
    Rookie Leaders
    Batters 2022, Pitchers 2022
    Splits Leaders
    Batters: vs L, vs R, Home, Away
    Pitchers: vs L , vs R, Home, Away
  • Teams
    Team Batting Stats
    2022, 2021, 2020, 2019, 2018, 2017...
    Team Pitching Stats
    2022, 2021, 2020, 2019, 2018, 2017...
    Team WAR Totals (RoS)
    AL East
    Blue Jays  |  DC
    Orioles  |  DC
    Rays  |  DC
    Red Sox  |  DC
    Yankees  |  DC
    AL Central
    Guardians  |  DC
    Royals  |  DC
    Tigers  |  DC
    Twins  |  DC
    White Sox  |  DC
    AL West
    Angels  |  DC
    Astros  |  DC
    Athletics  |  DC
    Mariners  |  DC
    Rangers  |  DC
    NL East
    Braves  |  DC
    Marlins  |  DC
    Mets  |  DC
    Nationals  |  DC
    Phillies  |  DC
    NL Central
    Brewers  |  DC
    Cardinals  |  DC
    Cubs  |  DC
    Pirates  |  DC
    Reds  |  DC
    NL West
    D-backs  |  DC
    Dodgers  |  DC
    Giants  |  DC
    Padres  |  DC
    Rockies  |  DC
    Positional Depth Charts
    Batters: C, 1B, 2B, SS, 3B, LF, CF, RF, DH
    Pitchers: SP, RP
  • RosterResource
    Current Depth Charts
    AL East
    Blue Jays
    Orioles
    Rays
    Red Sox
    Yankees
    AL Central
    Guardians
    Royals
    Tigers
    Twins
    White Sox
    AL West
    Angels
    Astros
    Athletics
    Mariners
    Rangers
    NL East
    Braves
    Marlins
    Mets
    Nationals
    Phillies
    NL Central
    Brewers
    Cardinals
    Cubs
    Pirates
    Reds
    NL West
    D-backs
    Dodgers
    Giants
    Padres
    Rockies
    Offseason Tools
    2023 Opening Day Tracker
    2023 Offseason Tracker
    2023 Free Agent Tracker
    In-Season Tools
    2023 Closer Depth Chart
    2023 Injury Report
    2022 Lineup Tracker
    2023 Payroll Pages
    2022 Probables Grid
    2022 Schedule Grid
    2023 Transaction Tracker
  • Prospects
    Prospects Home
    THE BOARD!
    THE BOARD: Scouting + Stats!
    How To Use THE BOARD: A Tutorial
    Top Prospects List
    Top Prospects
    2023 2022
    AL
    BALCHWHOU
    BOSCLELAA
    NYYDETOAK
    TBRKCRSEA
    TORMINTEX
    NL
    ATLCHCARI
    MIACINCOL
    NYMMILLAD
    PHIPITSDP
    WSNSTLSFG
    AL
    BALCHWHOU
    BOSCLELAA
    NYYDETOAK
    TBRKCRSEA
    TORMINTEX
    NL
    ATLCHCARI
    MIACINCOL
    NYMMILLAD
    PHIPITSDP
    WSNSTLSFG

    • 2022 Preseason Top 100

  • Glossary
    Library
    Batting Stats
    wOBA, wRC+, ISO, K% & BB%, more...
    Pitching Stats
    FIP, xFIP, BABIP, K/9 & BB/9, more...
    Defensive Stats
    UZR Primer, DRS, FSR, TZ & TZL, more...
    More
    WAR, UBR Primer, WPA, LI, Clutch
    Guts!
    Seasonal Constants
    Park Factors
    Park Factors by Handedness
  • Sign In
Help Support FanGraphs


Become a Member No Thanks
Already a member? Log In
  • Intro
  • Features
  • Offense
    • Complete List (Offense)
    • OBP
    • OPS and OPS+
    • wOBA
    • wRC and wRC+
    • wRAA
    • Off
    • BsR
    • UBR
    • wSB
    • wGDP
    • BABIP
    • ISO
    • HR/FB
    • Spd
    • Pull%/Cent%/Oppo%
    • Soft%/Med%/Hard%
    • GB%, LD%, FB%
    • K% and BB%
    • Plate Discipline (O-Swing%, Z-Swing%, etc.)
    • Pitch Type Linear Weights
    • Pace
  • Defense
    • Overview
    • Def
    • UZR
    • DRS
    • Defensive Runs Saved – 2020 Update
    • Inside Edge Fielding
    • Catcher Defense
    • FSR
    • RZR
    • TZ / TZL
  • Pitching
    • Complete List (Pitching)
    • ERA
    • WHIP
    • FIP
    • xFIP
    • SIERA
    • Strikeout and Walk Rates
    • Pull%/Cent%/Oppo%
    • Soft%/Med%/Hard%
    • GB%, LD%, FB%
    • BABIP
    • HR/FB
    • LOB%
    • Pitch Type Linear Weights
    • SD / MD
    • ERA- / FIP- / xFIP-
    • Plate Discipline (O-Swing%, Z-Swing%, etc.)
    • Pace
    • PITCHF/x
      • What is PITCHF/x?
      • Pitch Type Abbreviations & Classifications
      • Heat Maps
      • Common Mistakes
      • PITCHf/x Resources
  • WE/RE/LI
    • RE24
    • Win Expectancy
    • WPA
    • LI
    • WPA/LI
    • Clutch
  • Principles
    • DIPS
    • Regression toward the Mean
    • Replacement Level
    • Sample Size
    • Splits
    • Projection Systems
    • Linear Weights
    • Counting vs. Rate Statistics
    • Park Factors
    • Park Factors – 5 Year Regressed
    • Positional Adjustment
    • Aging Curve
    • League Equivalencies
    • Pythagorean Win-Loss
    • Luck
  • WAR
    • What is WAR?
    • WAR for Position Players
    • WAR for Pitchers
    • FDP
    • fWAR, rWAR, and WARP
    • WAR Misconceptions
  • Business

Understanding Projections, “True Talent Level”, and Variability

by Piper Slowinski
February 23, 2011

This is the second in a series of posts about projections. The first part was about the methodology behind each projection system. In this section, we look at what projections are actually telling us.

If you’re new to projections and want to use them to, say, help with your fantasy team, it’s easy to make a common mistake: underestimating the built-in variability in projections. Many people – and I used to be among this group myself – view projections as hard and fast guesses at a player’s production this next season. Most people get into projections as a result of fantasy baseball, so this makes sense; we all want to know which player is going to hit 30 homeruns this next season and which will steal 40 bases. However, projections are actually measuring something different than a player’s expected production: they’re measuring a player’s true talent level.

This might seem like an arbitrary distinction, but trust me, it’s not. As we all know from our day-to-day lives, having a “true talent level” at a particular skill does not necessarily mean you’ll perform at that level every single time in the future. Our minds love to ignore variability and instead treat outcomes as solely talent-driven, but the world doesn’t work that way. Let’s consider a couple examples.

Let’s start with something simple: flipping a coin. We’d expect that a normal coin would have a “true talent level” of landing heads 50% of the time, right? If you flipped that coin 100 times, though, it may be that you’d end up with 53 heads and 47 tails….or with 45 heads and 55 tails. You’d be most likely to end up with a result close to 50/50, but it’s no guarantee that things would end up precisely at the coin’s true talent level every single time.

If that’s not convincing enough (after all, there’s no “talent” involved in whether a coin comes up heads or not), consider basketball. Each of us has a specific true talent level for hitting free throws, some better than others. I’m horrible at basketball, so I’d place myself around a 30% talent level, meaning I’m likely to make around three baskets out of every ten. If I went out and shot five different sets of 10 free throws, though, I wouldn’t necessarily make three every single time; my scores may look something like 2, 4, 5, 3, 1. This variability doesn’t mean my true-talent level is wrong: it just means that there’s no guarantee we’ll perform exactly at our true talent level every time we perform. Sometimes we may over-perform, while others we may under-perform.

You’ll sometimes hear people refer to projections as “50th percentile projections”, which expresses exactly this concept: 50% of the time a player will over-perform their projection, while 50% of the time they’ll under-perform it. Players and teams are most likely to perform at a level close to their projection level, but that’s no guarantee. Variability for teams and players typically follows the normal distribution:

This is the statistical way of showing how likely a person (or team) is to perform close to their true talent level. Say we simulate this upcoming season 1,000 times with the Red Sox as a 94-win true talent level team. According to this graph, we’d expect the Sox to finish 68% of those seasons within one standard deviation of 94 wins, and 95% of them to fall within two standard deviations. “Standard deviation” simply refers to the predicted amount of variability with a team or player, with major league teams typically having a standard deviation of six games over a 162-game season.*

*Someone smarter than me, feel free to disagree and disprove this – that isn’t my calculation, so I’m not wed to it. I’d argue that standard deviation varies on a team-by-team basis, with some teams having a slightly tighter deviation and some having a slightly wider one, but as an average six games seems to make sense.

In other words, say we project David Wright to hit 20 homeruns this season. That projection isn’t saying that he’s going to hit exactly 20 homeruns, but instead that he’s 68% likely to hit within one standard deviation of 20 homeruns. With that in mind, Wright hitting either 15 or 25 homeruns wouldn’t necessarily prove the initial projection “wrong”: it just means that Wright’s season varied from his projection, and we can use that information to better project his true-talent level going forward.

How do we measure that variability? What’s the typical standard deviation for homeruns, batting average, wOBA, etc? Those questions I don’t have a specific answer to, as there are many conflicting issues in play. For one, projections are only best guesses at a player’s true talent level, and there are many things we can’t know or measure: the exact effect of injuries, exactly how a specific player will age, the exact amount of talent residing in a player. We can make good guesses based on historical trends and what we see on a day-to-day level, but “true talent level” is an ethereal concept and can’t be measured like you can a temperature. As such, projections will never be perfect.

Also, the standard deviation for specific statistics can be different depending on the player. Adam Dunn has almost exactly 40 homeruns for six seasons in a row, but there are also players like Jose Bautista and Ben Zobrist that have had wildly varying power numbers over the past few seasons. Each aspect of each projection therefore has a different level of confidence to it, and you have to be able to assess that confidence by looking through a player’s career and determining if you think that projection has a wide or tight standard deviation.

So the next time you start looking through projections, remember to take variability into account. Our minds love to eschew probability and uncertainty – why do you think casinos make such a killing? – but understanding this concept can keep you from drawing faulty conclusions from projections. Embrace uncertainty, and it might help you beat the house (or beat your friends at Fantasy).





Food Metaphors, Replacement Level Style
 
Pitchers and Injuries: It Happens

Piper was the editor-in-chief of DRaysBay and the keeper of the FanGraphs Library.

5 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
RMR
11 years ago

Great article. I think it’s also important to recognize the flip-side of the normal curve, the extremes. Some people fall in to the trap of using an extreme outlier or two to “prove” that a method of estimating true talent is “wrong” or “stupid”.

However, given 100 players with true talent estimates, we would actually expect a few guys to put up performances way out in the tails (2+ SDs from the mean) — even if we nailed the true talent estimates.

The whole probability distribution concept and it’s associated considerations (e.g. sample size) is just fundamental to understanding statistical analysis. Great article, Steve!

0
Steven Ellingson
11 years ago
Reply to  RMR

I draw a 2 of hearts, a 4 of spades, a 9 of clubs, a jack of hearts, and a queen of clubs.

The chances of me getting that exact hand are 1 in three million! I must not have it!

0
RMR
11 years ago
Reply to  Steven Ellingson

Or maybe you have a special skill called “ability to draw 2 of hearts, a 4 of spades, a 9 of clubs, a jack of hearts, and a queen of clubs”. We can call this “clutch drawing with no hold cards”.

2
nick
11 years ago

Also, probability distributions are never correct – in the way that science is never “correct”. They are always needing to be updated when new evidence arrives. The truth always is in need of updates – it is never static.

I’ve always wondered how well projections worked for the 1969 season. How would those projections have turned out? All the distributions couldn’t have factored in the lowering of the mound. I’m guessing that not having that data would have skewed everything. So projections are always limited to the data you have before hand and sometimes your prior sampling data is not going to be able to predict the new population probability distribution going forward. Especially with rule changes and things like new ballparks.

0
Epee9
11 years ago

The expected fluctuation from true-talent level from chance over a 162-game season is 6.4 wins. The observed standard deviation from 0.500 win record for all 30 teams (2001-9) is 11.9 wins.

Since 0.500 must be the true-talent average for MLB as a whole, that leaves SQRT(11.9^2 – 6.4^2) = 10.1 wins per team per year to be explained by other factors.

I estimate (long calculation) that salary differences account for another 6.4* wins/team/year. That still leaves 7.8 wins/team/year to be explained.

At the team level, the breakdown appears to be: 28% luck, 28% salary, 43% something else.

0
You are going to send email to

Move Comment

Updated: Saturday, January 28, 2023 7:05 AM ETUpdated: 1/28/2023 7:05 AM ET
Player Linker - @fangraphs - Contact Us - Advertise - Terms of Service - Privacy Policy
sis_logo
All major league baseball data including pitch type, velocity, batted ball location, and play-by-play data provided by Sports Info Solutions.
mlb logo
Major League and Minor League Baseball data provided by Major League Baseball.
Mitchel Lichtman
All UZR (ultimate zone rating) calculations are provided courtesy of Mitchel Lichtman.
TangoTiger.com
All Win Expectancy, Leverage Index, Run Expectancy, and Fans Scouting Report data licenced from TangoTiger.com
Retrosheet.org
Play-by-play data prior to 2002 was obtained free of charge from and is copyrighted by Retrosheet.