You can’t prevent that.
But any system that actively enforces party lines should be automatically disqualified as a legitimate electoral system. It strengthens the power of the dumbest, least informed voters at the expense of rational voters willing to actually understand who candidates are.
That’s not a bug. It’s the entire aim of an electoral system.
The people who aren’t the extreme ends of two poles and actually have policies the majority are in favor of are the people who are supposed to be in office. I shouldn’t be choosing between “arrest people for using birth control” and “eat the rich and disband the police”.
You also don’t get progress in any direction when both parties are spending half their time unraveling everything the last group did.