NFL combine drills actually pack (some) predictive power

ESPN the Magazine asks, of the NFL Combine’s influence on the Draft, “How do you weigh a week of drills against three or four years of a player’s work?”

Citing regression analysis of combine metrics dating back to 2006, by Jeff Phillips, MIT grad and principal of the Parthenon Group, columnist Peter Keating writes:

If you look at certain combine stats, they explain on average 20 percent of how well players perform during their first three pro seasons.  That’s probably a weaker relationship than most team executives would want, but it aint zero…  Phillips found that different measures matter for different positions.  For instance, 40-yard dash time – sometimes derided by analysts who argue that players don’t actually have to run 40-yard dashes in games – is the only skill that’s significant for all positions.  A players’ weight is important for offensive linemen, defensive linemen and linebackers, while scores in the three-cone drill (which measures agility) matter for running backs and defensive backs.  Other metrics are narrower in their predictive value… definitive answers haven’t emerged yet from the fledgling research.

The data don’t predict a player’s ceiling, the “perfect storm awesomeness of Adrian Peterson or Patrick Willis.”

The raw data simply don’t know what kind of system a player will enter, or talent he’ll have around him, or luck he’ll have with injuries, or intangibles he possesses.  But (the) stats do a pretty good job of separating the potential stars from likely busts…  So looking at extreme cases from the class of 2014, Jadeveon Clowney’s 40 time was 0.3 of a second faster than any of the five best defensive linemen drafted in the past eight years, and his Phillips stats are better than 99 percent of players at his position.  Among less famous prospects, keep a draft-day eye on Brandin Cooks, a receiver from Oregon State, whose blazing speed helped him achieve the third-best blend of stats among all wideouts since 2006… [picked Rd 1, #20 by the Saints] watch Minnesota’s Ra’Shede Hageman too [picked Rd 2, #5, #37 overall by the Falcons].  Just 11 defensive linemen over 300 pounds in Phillip’s database have shown better speed than Hageman did at the combine, and eight of them have gone on to successful NFL careers.

On the flip side, Ha Ha Clinton-Dix could go in the top 10 but he was below median in every key component of Phillip’s statistics for defensive backs at this year’s combine [picked Rd 1, #21 by the Packers].

We recently made a crucial distinction in another post on the topic of data and decision-making, entitled The greatest comeback ever and the limits of decision models:  some outcomes can be influenced and some cannot.  Big data may help make accurate predictions or guide knotty optimization choices or help avoid common biases, but it doesn’t control events.  Models can predict the rainfall and days of sunshine on a given farm in central Iowa but can’t change the weather.  A top draft pick may or may not develop based on the system, surrounding talent, &etc.

In our experience the best results often come from a combination of deliberation and intuition.  Too much data can lead to analysis paralysis, common sense can be a shockingly unreliable guide, and those who rely on intuition alone tend to overestimate its effectiveness.  (They recall the times it served them well and forget the times it didn’t.)

In the wake of the last financial crisis, BoE Director of Financial Stability Andrew Haldane deployed an analogy about a Frisbee-catching dog to explain how complex (and sometimes frivolous) attempts at regulation push the limits of data modeling or even the nature of knowledge itself.  The dog can catch the Frisbee despite the complex physics involved because the dog keeps it simple:  run at a speed so that the angle of gaze to the Frisbee remains roughly constant.

So while old-fashioned intuition is not out of date it’s also unwise to rely only on one’s instincts to decide when to rely on one’s instincts.  The dog’s doing just fine, but if it involves more than a Frisbee he might want to crunch some numbers too.

Specifically about this year’s draft:  we’re never quite sure what to make of the draft, it’s so over-hyped.  Clowney seemed like the obvious first pick and Manziel is high risk, so that worked out as expected.  A pretty efficient market overall given the information available to the teams…