With the Scout Camp series coming to it’s end, I thought it might be interesting to take a look at the method of evaluation and the cause of the variations between much of what has been written here and what is has been written on other draft sites. The purpose of this article is to explain the way in which I go about evaluating players, my thoughts on the evaluation of college players in general and where that differs from other analysts.
It will be rather long, so strap in.
The Harsh Truths
Scouting is not a science. It’s not even close. There are very few things that can be said to approach actual known entities with almost everything being an estimate of wildly varying accuracy; as a result, almost every statement made regarding the draft will be some shade of wrong. So why even bother?
While very few things can be said with a high degree of certainty, what we eventually end up with after evaluation is a relativistic description of players; this is really where the core of scouting begins. It is quite hard to describe concepts such as pad level and hand placement to a high degree; to quantitatively asses these skills is inherently inaccurate, but what can often be done is create a standard through comparison with others. However, while this qualitative method can be effective, and does form the basis of almost all current NFL scouting, it is important to not forget the inherent subjectivity of such assessments. Every scout and analyst will see the same tape differently, and will have slightly different methods of evaluation in this regard. What this means is that when anybody, myself included, uses statements such as “X player is…” or any other definite statement, they are either fundamentally unaware of the limitations of their own evaluations or simply opting to drop the “I think…” from before every statement.
This in itself is not a problem, it would be fairly tiresome to read “I think..” a few dozen times throughout each evaluation, but the issue comes when people then don’t absorb the pervading uncertainty when reading the evaluation. While a few people being overconfident about the power of scouting isn’t going to do much real harm, if enough people have enough confidence in the scouting process, it creates this atmosphere of legitimacy. However, what should be made clear at this point is just how bad we are judging talent, even at the highest levels.
Photo Credit: Brian Spurlock/USA Today Sports
By The Numbers
If we are to judge just how ineffective the current scouting process in the NFL is, the best place to start would be to look at how effective draft picks have been. In order to do this, I am going to look at first round picks since 2011, an arbitrary starting point, and examine how many have worked out. I chose first round picks for this, partly in order to keep the numbers manageable, but also because these are the players where there should be the greatest consensus about their ability. While there are the occasional out-of-the-blue picks, the majority of the players taken in the first round are highly rated across most draft boards with few surprises coming about the players taken on draft night. So how do these players fare?
First, we’ll need to define how we judge success, and for first round picks I have decided, again with a certain level of arbitrariness, that a good definition of a solid first round pick is somebody who averages an Approximate Value each year of 5 or more. AV is a valuation statistic from Pro Football Reference, it is far from perfect, but at this level gives a vague indication of player success, especially when we will be comparing players across different positions. Am AV of five represents a decent but unspectacular season for an NFL player, this is the value that Tyler Larsen and Jonathan Stewart received last season, and so for a first round pick to fail to live up to this level would likely be seen by most to be a bust; this isn’t a particularly high bar for them to reach.
Of the 223 first round picks over the past seven years, 120 have averaged an AV of 5 or more for their careers so far; that’s good for a 53% success rate. The numbers are also extremely consistent year-to-year. 2013 unsurprisingly had the lowest success rate, but that was still 47% and no year managed to get as high as 60%. Even 2011, the draft class that has become famous for being loaded with talent, had its fair share of busts; let’s not forget that Jake Locker was drafted eighth overall. Now, there are some who will argue that teams often reach for a quarterback in the first round, and that swinging for a signal caller is what drags these numbers down to some extent; however, the numbers just don’t bear this argument out, first round quarterbacks actually have a success rate of 60%, they are in fact dragging the numbers up.
Interestingly, this success rate continues to be low even at the very top of the draft.
While there does appear to be a dropoff at the end of the draft, even in the top 10, the numbers are very poor. What I mean to show from this, however, is not that everything is impossible and that we should all give up, but rather that the conventions and techniques that have been used in most of scouting in recent years are far from perfect, and that maybe, just maybe, it might be worth trying something else.
Up Next: Why Do We Fail So Consistently?
things such as spacial recognition and reaction time are important… too many tests are static and not enough are dynamic… high pressure functioning and failure tolerance are also important things to test more specifically