Confirmed Bias
Before I start this section I should first thank my friend and colleague Dan Kreso; If you find any of what is written here interesting, and hopefully you will, I strongly suggest that you check-out his complete piece on scouting biases.
“Humans can come to conclusions from very little evidence, and more troublingly, are not aware of the lack of quality or quantity of the evidence. In other words – we jump to conclusions.”
-Dan Kreso
If we never came to any conclusions, scouting would be a fairly pointless affair, but the way in which we come to conclusions is often highly flawed and leads to significant errors in evaluation. A major reason for this is confirmation bias, or the idea that we look for things which confirm our prior conceptions of a player; these conceptions can either come from a few memorable plays we are aware of, a particular theme in the first few plays we see or simply from a known reputation of the player and (subconsciously) when watching tape, we look for things which fit into our preconceived view of what the player is like.
“Rationally the first observation should not be given more weight than any other. But with the “positive test strategy” this is indeed what happens! If you see a wide receiver you have often watched before, and one you believe to be a reliable catcher of the ball, drop a pass, it is an uncharacteristic error. If he makes several drops, then he is just having a down patch in form. For a different player, where the sequence is reversed, however, it is easier to think of the wide receiver as a bad catcher who is having some uncharacteristically good form.”
-Dan Kreso
The other significant source of error is that of uncertainty; NFL prospects often enter the league having never been asked to exhibit skills that will be important at the next level, and it is up to scouts to try and judge their potential ability in these areas. It is very difficult to evaluate what you can’t see, and trying to infer things from why you don’t see it is also strewn with potential error; a running back might not be used as a receiver because they can’t catch, or it might be that the scheme they play in simply doesn’t make use of running backs in this way. Teams can often probe these uncertainties during private workouts but, as with confirmation bias, there is only so much that can be done from an outside perspective. The best than can be managed is, in suitable cases, to attempt to extrapolate from what is seen on tape such as in attempting to judge a quarterbacks’ ability to process reads. However, as with confirmation bias, possibly the biggest improvement that can be made at this stage is to be aware of the unknowns.
The way in which this manifests in my evaluations is in the frequent use of the term developmental. This is used to account for when players fail to demonstrate a suitable skill set, but also fail to demonstrate the lack of one. When you see me rank a player like Josh Sweat as a developmental piece, it is due to the fact that he needs to improve his hand usage significantly in order be consistently effective in the NFL; as much as he flashes with quickness around the edge, his occasional struggles turning the corner are potentially worrisome at the next level. I’m not saying he’s not a good (potentially great) prospect, but the skills are not prominently on display, so he will still have to develop to be successful; NFL.com gives Sweat a “Round 2-3” valuation and has the FSU edge rusher mocked to the Panthers with the 24th pick.
Some skill sets, such as the use of running backs as receivers, are very useful but not integral, whereas a player who will have to move from a 3-4 OLB to a 4-3 DE in the NFL or vice versa, will be integral to their potential effectiveness. Still, this is still only a very broad and vague inclusion of this uncertainty, and the lack of certainty in this area is something that I try to stress throughout my evaluations.
Then there is the fundamental uncertainty in the evaluation of all skills; trying to accurately evaluate a players performance, even in a certain skill on a certain play, is impossible, and it is important for us to understand the uncertainty in our own judgement as well as in the player. While this is hard to account for on a prospect by prospect basis, an important skill in any area is an awareness for your own flaws and over the course of several years I have, like most I assume, done my best to identify ways in which my previous evaluations have been flawed and looked to correct these errors. Again, this is certainly not a completed process and there are likely to be many inaccuracies in my evaluations, just as there are for any scout or evaluator. Show me a scout who tells you they have evaluated every prospect correctly and I will show you a liar who has most likely deleted their past thoughts as soon as they are proven wrong.
Up Next: What Do We Do To Eliminate These Biases?
things such as spacial recognition and reaction time are important… too many tests are static and not enough are dynamic… high pressure functioning and failure tolerance are also important things to test more specifically