Coastal wildfowling is not expensive but there is a learning curve. If you join BASC you can get the wildfowling booklet that lists all the clubs, and membership is fairly cheap. In Scotland there is still free wildfowling without need for a permit on some Crown foreshore. Some nature reserves like Eden Estuary, Montrose basin, Tyninghame and Lindisfarne allow permit-controlled wildfowling. The permit is free at Eden, and not expensive at the others. You must have a shotgun certificate and liability insurance (automatic with BASC membership) to get the permit.
I don't pay for my rabbit shooting. You just need to cultivate relationships with landowners
My half-gun in a syndicate costs me £250 a year - that gets me 2 days on the rough shoot (pheasants, woodock, hare, pigeons) plus a few days decoying pigeons over peas, plus some deer stalking.
Hind stalking (with a rifle, obviously) on Arran is pretty cheap via BASC (about £250 for 5 days, if I recall right).
To get the shotgun certificate you would need to budget:
Application fee about £50 or so
Gun cabinet about £100
BASC membership for insurance (not mandatory but very wise) about £50 a year
Gun. I paid £50 for a boxlock side by side. You could spend slightly more on a pair of Purdey sidelocks.
As Buckshot mentioned, some shoots will give their beaters a free day of shooting late in the season in return for their work.