Jaime may not have wanted Casterly Rock then. Jaime would have lost all 3 of his children. That type of grief changes any person