I sort of partly agree with both sides of the argument.
Never really bothered me, I only store tap water in mine and it does have a "distinctive taste" I'd also go as far as saying it has a "distinctive smell" as well!, but I'm just used to it.
It definitely happened in my US army water bottles I had too.
I think it would happen in any plastic container.
is it perhaps to do with the gradual breakdown of the rubber washer?
funny story, just after My TA training I was told of another guy that joined a few weeks after me, he stored Cola in his 58" bottle.
when it came to cooking his rations and personal Hygiene (washing/shaving) he suddenly had a very confused look on his face (and I imagine a few press-ups)!
strange really, because it was commonplace in WW2 to carry tea or lemonade in the water bottles.