It's not fraught with difficulty, it's fraught with impossibility. Here's the thing, even if we assume for the sake of argument that you can form an accurate judgment of a given players fielding, that's not enough. In order to be useful, you have to compare it with something. In order to rate Theriot's defense at shortstop as average, above average or below average, then by definition you must watch 95% of every shortstop's plays, form a judgment for each player and then make meaningful comparisons between them. Attempting to do this on your own, inside your own head, is ridiculous. And while that process may lend itself to consistent evaluations of players from year to year, that is because those evaluations are consistently meaningless.