Unless I really have something I need to do, I usually don’t walk away from someone who is annoying even though I would like to, I don’t want to come off as rude. But if I really do have something pressing, I’ll just say so! I’ll say something like, “I’m sorry to interrupt, but I have to (insert whatever I need to take care of).”