If I were in your shoes, I'd have a chat with Sagepay, then my card issuer if I didn't like what they had to say.
From looking at this guy, he seems more of an incompeten than a scammer, but having been in your situation before, I know I'd have alarm bells going off about the whole scenario.
Incidentally, I'll actually be driving within about 3 miles of his house on Thursday afternoon.