Baseball is just a lot different than football and basketball. A lot of times the best "athletes" aren't the best baseball players. Wade Boggs and John Kruk would not have played above high school in anything but baseball. Naturally speed helps in the outfield and on the base path but even then there are baseball abilities that makes it less important. Football and basketball you can at least out athlete people but it doesn't work in baseball.

Baseball is also expensive now. It is very unusual to see someone like Hunter Renfroe who doesn't do the travel team almost year round. I know several people that is what they use most of their vacation days on is baseball tournaments.