Ole is a term of affection in the South.

When someone is referred to as "A good ole boy" it doesn't have a thing to do with his age.

"Here comes Ole Bob" is not a comment on his age, either. There doesn't have to be a "young Bob".

Ole Miss is the same thing. It is an affectionate nickname.