"Engaging with the lever in any way makes me culpable for the death that occurs" seems to be the rationale behind most of the voting naysayers, so, and I am quoting wikipedia here, their rational is quite literally one of the positions of the trolley problem;
'...since moral wrongs are already in place in the situation, moving to another track constitutes a participation in the moral wrong, making one partially responsible...'
Their refusal to engage with the trolley problem has made them ignorant not only of how their position IS one of the sides they refuse to take, but also that the other option recognises their desire for moral purity at the expense of obligation, for what it is;
'Under some interpretations of moral obligation, simply being present in this situation and being able to influence its outcome constitutes an obligation to participate. If this is the case, then doing nothing would be considered an immoral act.'
People who refuse to vote because "they're both bad" may be taking the high ground, but there are still people on the tracks.