I think the main reason is that there is no real reason to wear it. There are lots of alternatives that dont have the same connotations, so why would somone choose to wear something with pseudo-military overtones, when they could choose something else that is more "friendly"? Green trousers dont draw the same attention and standard issue lightwieghts pass the "friendly" test without raising an eyebrow, are available from most surpluss stores, are cheap and tough - so why buy camo?
Camo is military clothing. There is no significant % of the population that hunt, as in the US, which letitamizes wearing it.
So people question the motives of those that do wear it, and assume they have some unhealthy military obsession. There is a crazy fringe element surrounding bushcraft that is occupied by camo-wearing people with a fantasy/desire to survive a post apocalyptic world in their bunker, stuffed with MRE's and armed to the teeth. I think most people would like to distance themselves from these loonies.
Camo itself isnt bad, but that's not really the point. It's the desire to not be associated with the crazy people that stops folks wearing it.