It's possible for someone to be neutral and still feel some adverse effects, but keep in mind that feelings of mild aggression, disgust, or comradery would still lean more towards one team than another.

Alternatively, whilst your character may not consider themselves to be on a side, those mild symptoms may mark them for or against their own team. How consumed or engaged your character is with their side is up to you, but that mark would ensure adverse responses from others, if that's an option you'd like to play with!