Next step, "Why should you be bothered by them?" hehe..More confrontational, I know. I think behind not being "bothered" by someone still implies that there is something wrong with "them" it's just that it isn't a bother to you. At the same time, I like that your choice doesn't call anyone out. I just wonder if there is a more affirming way to state this point.