To have a specific reason, objection, or prejudice that makes you dislike or oppose someone or something.
"What do you have against eating vegetables? They're good for you!"
I have nothing against the man personally.
— Common idiomatic usage; widely found in English political speech and journalism