Bill Barnwell is up to his usual tricks at Grantland. This time, he’s tired of hearing that Flacco is an elite quarterback and wants a new measure of quarterback value. Flacco gets credit for piling up wins, which Barnwell thinks is unfair:
For whatever good or bad Flacco provides, he has spent his entire career as the starting quarterback of the Baltimore Ravens, who perennially possess one of the league’s best defenses. He also has Ray Rice and a solid running game to go alongside him on offense. It’s safe to say that a win by, say, Cam Newton usually requires more work from the quarterback than one by Flacco.
I agree with this wholeheartedly. In response, Barnwell tries to capture quarterback value by creating an “expected wins” measure based on points allowed by the defense and comparing this to actual wins. He argues that a quarterback with more actual wins than expected wins is doing well because he is scoring more points than average.
An example helps explain the concept. First, Barnwell notes that teams have won 86.5% of games recently when allowing between 8 and 12 points. Imagine a team that allows between 8 and 12 points in all 16 games. They are expected to win 86.5% of those games, or 13.8 games. If the team won 14-16 games, Barnwell would argue that the quarterback is doing better than average, while if the team won fewer, Barnwell would argue that the quarterback is doing worse.
As hoped, Flacco is unimpressive by this measure (while the usual suspects — Tom Brady and Peyton Manning — are top dogs). He has 44 wins in 64 regular season games, but because the Ravens D is so good, an average QB would have managed about 42.
Before going farther, I’ll warn you: these numbers are pretty meaningless. I’ll start by explaining some basic issues with Barnwell’s stats and finish by discussing why none of this has much to do with Flacco’s quality.
He Pulled a Barnwell!
So what are the problems with Barnwell’s approach? First of all, it’s way too complicated for what it accomplishes. Why not just look at average points scored in all starts? The Ravens have scored 1,632 points in Flacco’s 68 games, which is about 24 points per game. NFL teams averaged 22 points per game in 2011. This comparison makes exactly the same point as Barnwell’s fancy stats: at face value (more on this below), Flacco seems just above average.
Second of all, it’s easy to come up with circumstances where Barnwell’s measure goes in the tank. The funniest is a quarterback who scores 100 points per game while his defense pitches shutout after shutout. We’ll call him Geno Smith (if only he had that defense!). Smith, as I’ve defined him, is the best quarterback of all time, which is easy to see from his points per game. But since teams always win when allowing zero points, Barnwell would find that Smith is an average quarterback (he wins every game, just like the average guy).
A good QB with an excellent defense will never look that great, in fact, since there’s little room to improve on an 80-100% expected winning percentage (Flacco’s two wins above expectation are a bit more impressive through this lens). The same is true at the other end of the spectrum. A good QB will have a hard time improving on expected winning percentage if his team gives up 40 points per game. If he averages an excellent 30 points per game, he might win a couple games against an expectation of one or so, but that’s a small improvement.
So, Barnwell’s approach is unnecessarily complex, and the complexity may even reduce its value (similar to WAR in baseball, which I attempted to revamp recently). But is it meaningful at all?
He Pulled a Double Barnwell!
After using his statistics to argue that Flacco is not among the elite (“[Matt Ryan] appears to have grown into a more complete quarterback than Flacco”), Barnwell has a throwaway line at the end:
Of course, there are plenty of limitations to this simple[*] metric. It doesn’t incorporate game style or factor pick-sixes/other touchdowns not allowed by the defense into the discussion.
Indeed, this metric might overvalue Flacco because of the Ed Reed effect (Flacco gets credit for Ed Reed’s pick sixes) and the Ravens’ strong running game (more free points for Flacco). However, it also might undervalue Flacco because of “game style,” or rather, “game strategy.” Because of their good defense and consistent running game, the Ravens generally play low risk/low reward on offense. If they limit turnovers and score 25 points on offense, that stands up more often than not (the 49ers do the same thing with Alex Smith and Frank Gore). The Ravens don’t seem interested in having an “above average” QB because they win their division every year with “average” performance. Without situational analysis, we don’t really know whether Flacco is good because he doesn’t get the chance to excel in the traditional sense.
The impetus for Barnwell’s article is correct: commentators shouldn’t use Flacco’s win-loss record to argue that he is elite. The Ravens defense is largely responsible for that record. Unfortunately, Barnwell’s complex measure isn’t helpful. All we know from Barnwell’s numbers is that Flacco executes his half of the Ravens’ game plan well — his 47-21 record proves that.
I discussed this with my PhD friend Chris, and he had this to say:
It really doesn’t solve the good defense issue at all. The Cowboys D “gave up” 34 points last night. But this was because Tony Romo had 5 interceptions; Chicago had a ton of possessions and great field position. In Barnwell’s system, Romo would have been barely penalized for losing that game, because when your defense gives up 34 it’s hard to win. But the loss was almost entirely his fault.
I believe this is usually how it works; teams with bad QBs give up a ton of points, because a bad QB loses the field position battle, gives the other team extra possessions, and can’t keep the offense on the field, leading eventually to a tired and ineffective defense. This is one of the primary mechanisms through which bad QBs lose games, so any attempt to improve on traditional stats that doesn’t account for this is probably pretty useless.
That’s a great point — it’s the reverse of the Ed Reed effect described above, with the QB transferring responsibility for poor play to the defense in this case.
*I disagree with the word “simple,” but I’ve covered that already.