I skimmed through Aaron Schatz’s Football Outsiders preview of the Ravens – Patriots game today. As usual, he gives a lot of interesting numbers. Football Outsiders has better data than anyone else. Just like everyone in the world, Schatz picks the Patriots as the clear favorites in the end; I agree, though I think the Patriots’ edge is subtle. The Patriots have home field and finished second to the Packers in True Wins with 12, but the Ravens played a harder schedule (7.8 average True Wins for Ravens opponents versus 7.3 for the Patriots, including a game against the Vick-less Eagles).
I often find that stats-based previews list a lot of numbers that don’t mean much. Here’s an example from Schatz’s piece:
The biggest strength of the Ravens defense this season has been the pass rush, and it’s stayed strong over the last few weeks even as the rest of the defense faded a bit . . . The big improvement has come from a couple of situational pass rushers: fifth-round rookie Pernell McPhee (6.0 sacks) and third-year veteran Paul Kruger (5.5 sacks), who played very little in his first two seasons. There are also more sacks coming out of more complex blitzes; the Ravens had just one sack from a defensive back in 2010, but six in 2011.
McPhee and Kruger primarily play in third-down sub packages, which is a big reason why the Ravens have the league’s best Adjusted Sack Rate on third downs at 11.6 percent. Tom Brady is sacked almost twice as often on third downs, with a 4.4 percent ASR on first and second down but 8.7 percent ASR on third and fourth down, so we can expect a couple of drives to end when Brady is caught in the pocket with nowhere to go on third down.
The first paragraph tells us that the Ravens are getting a decent number of sacks from multiple sources. Of course, sacks are driven by good coverage and good pass rushing, but they are still semi-useful for measuring pass defense. The second paragraph I find less useful, since QBs should rationally take more sacks on third down (instead of throwing the ball away), especially when they have many yards to gain. For example, it’s likely that the Ravens have a strong third down sack percentage because teams generally face long third down distance against them. Opposing QBs willingly hang in the pocket and get smashed because first down routes take longer to develop. Similarly, Tom Brady’s higher sack rate on third down is probably normal and intentional. Losing 7 yards on a first or second down sack can kill a long drive, while losing 7 yards on a third down sack hurts field position only a little.
Here’s a different sort of example:
Tom Brady excels against big blitzes, so the Ravens want to limit those and bring just four or five pass rushers. Based on our current numbers, Brady was actually a little worse against five (7.4 yards per play) than he was against four (8.1 yards per play) and the Ravens bring five on 29 percent of passes (fifth in the NFL).
The first line is a great point, which Gregg Easterbrook (TMQ) often makes as well. Big blitzes are really risky. You might get a sack or two, but you might also give up a bomb, which is a game changer. My issue here is with the last two lines. First, Brady’s 0.7 yard difference in yards per play against four and five man blitzes seems too small to highlight and may reflect game situation (e.g., teams probably blitz five in longer distance situations, where I just argued that QBs are more likely to risk a sack in order to gain the necessary yardage).
Second, even though the Ravens rushed five a lot this year, they may not rush five a lot today (and even if they do, it might not matter much based on my last argument). Teams playing the Ravens were generally behind and faced lots of clear passing situations, giving the Ravens lots of chances to blitz.
In short, I’m saying that Schatz’s numbers provide some general context, but they say nothing about causation. The relationships he hints at suffer from “omitted variables bias” (i.e., decisions and plays are not executed in a vacuum; they are affected by many other variables during the game).
Are there ways around this problem? Sure. One approach is to stop trying to predict every aspect of the game and focus on the big picture (my True Wins statistic and Football Outsiders’ DVOA statistic are two examples of this approach). There is a strong temptation to analyze each little match up, since this humanizes the analysis and makes for good reading, but such a level of accurate detail is difficult to attain.
Another approach is to look at predictable events that have a very strong effect on game outcomes. Two examples are players out due to injury and turnovers. Heading into a game, we generally know who is able to play, and turnover rates have some persistence each season. Turnovers in particular are quite predictive of game outcomes: teams that won the turnover battle won 80% of their games between 2005 and 2011 (teams that tied the turnover battle won exactly 50% of the time).
Unfortunately, neither of these approaches help a team (or its fans) figure out whether to blame the offense, defense, coach, or quarterback for a loss. For that level of analysis, I have a suggestion. Check out the stats, but watch the game too and pay attention to blocking, coverages (the all 22 camera view would help), tackling, and everything else besides the quarterback and the running back. You’ll get a lot farther by watching closely than by reading up on statistics alone.