If you can "punch it out with a stick" (without drilling it) then you didn't have it fitted very well to begin with.
I was thinking the same thing. Any axe handle I fitted wouldn't shift with a steel punch and a club hammer. You have to break up the wood with a drill, chisel etc.
I remember Mors Kochanski telling me that he filed deep nicks into the metal wedges he fitted so that he could hook them out with a screwdriver. I tried it and it really does work, but I still couldn't get the wooden wedge out (but admittedly drilling the wood out was easier with no metal to hit)