Men get tripped out over stupid shit. And a lot of it is what will their buddies think. (For example, "wow she must be fucked up in the head, massive daddy issues, good luck with that")
The rest is just men being dumb. One of the most satisfying relationships I ever had was with a full-on sex worker - and I don't just mean sexually, although...in that regard, yes I was fully satisfied. And she kept asking if i would develop a problem with what she did for a living. I kept saying no, and she kept not believing me based on past experience. We didn't work out for other reasons...
Some people are just not open to non-conformism, but the ones that are, are more likely to have the emotional intelligence and strength to meet and challenge as an equal.