Context matters.

If someone put up a statue of Jefferson somewhere to celebrate that he was a slave owner or serial rapist, or for the purpose of intimidating black people, it should come down. I'm not aware of any such statues though.

We shouldn't erase the history of how he treated his fellow human beings though, so any monument to Jefferson or Washington's accomplishments needs to be placed in proper context. That might mean including some kind of reminder plaque, or maybe just doing a better job of teaching the full story in schools. If we stop seeing the Founding Fathers as flawless gods and instead just regular people like us who stepped up when it counted, we can realize that we are just as capable as them of accomplishing great things ourselves.