You don't have to do anything wrong, all the police would need would be an allegation, which could be completely false. It's not unheard of, for example, for people who are anti-shooting to make false allegation against a law abiding SGC/FAC holder.
I think it would certainly be wise to keep...