Most people are way less observant than you would think. As I others have said, movement, shine, shape etc also all come into play.
But you would be surprised by how many people wouldn't notice you in the woods if you are wearing normal clothing.
Olive, ranger green, grey, coyote brown...