There is a lot more that goes into trying to measure defensive value. A hit is easy to measure. With defense, you have to take into account where the ball was hit, where the guy was standing before the ball was hit, his ability to get to a baseball quickly and catch it, how quickly the ball is released once it is caught, how hard it is thrown, how accurate the throw is, and a bunch of stuff I'm probably missing.
We can get a good indication and rank order of guys who are better and worse at those things by position using statistical analysis, even though the difference between them may be small and unimportant for several players and larger for others. But most of the time if a guy is not good based on the eye test, he's never going to be in the group of players for that position for which there is enough data to get useful analysis.
The thing about statistics is that the numbers don't mean anything until they get large enough for predictive value. Then when they get super large, small and unimportant differences may be statistically significant but have little practical difference.