Yes, that may well be the case. No one is arguing with that.
But if you have it in your pack away from eyes then no one sees it, no one gets uppity.
You carry it on display, whether legal or not, you are courting potential trouble. That's the reality.
You may well say 'well, I'm willing to...