Do you speak up when you feel taken advantage of or abused?

I have never been abused, but sometimes people have taken advantage of my silence or kindness. I bide my time because I know that they will meet their karma soon. I will often just distance myself from the person.