All I can do is echo what others have already said.
Shop around and have a look in your local army surplus shop.
To be honest, I don't actually have a bivvy bag. Always keep meaning to get one...but...I've never had one and never missed one. Just feel like I have to get one. If I was to get one it'd be an ex-army one because they are cheap, available and olive drab (I don't like sticking out). But I haven't got one and probably never will.
Here's what I use:
Sleeping mat - get one of the nice thick army surplus ones. MUCH better than the civilain ones in Millets et al. And only cost £10.
I use...wait for it...an inflatable mattress! Yup. Bloody comfy it is too. I got a hikers mattress in Blacks. It also costs £10. I put this on the sleeping mat to stop it puncturing. And it never has yet - and I've been using this combo for nigh-on ten years. A hikers mattress is different from a normal inflatable mattress in that it's a third the size and weight (a bit bigger than a bog roll) and takes just a minute or two to blow up. It looks a bit like a beach lilo (which a friend of mine uses!) - in that it's a series of individually inflated tubes. I tried to use a self-inflating mattress but absolutely hate them - they're expensive, much heavier and too thin.
And then comes the sleeping bag. This is probably where you don't want to scrimp. In my view. I really don't know where and how much mine cost. Have a look at the Highlander range. Get a 3 season bag as well.
Tarp - I got mine in a local surplus shop. Cost £25. I'd say that one suggested by Mesquite is a real bargain. Snap one up now, or put it on your Christmas list!
I use either paracord to suspend the tarp, or tent poles and bungees - all depends on time of year, location and whimsy.
As someone else said - I reckon £100 will do it. Most of that will (SHOULD!) go on the sleeping bag. Again, Christmas is coming - so put it on the pressie list.
Don't know what other kit you need. As recommended, the Forsts Mora Clipper knife (£8.50 on ebay) is a REALLY good option. Billy cans can be either bought or you can use stainless stell utensil pots from Wilkinsons/Morrisons/Woolworths etc - drill a couple of holes either side at the top and use a bit of chain from Homebase to suspend it over the fire. They cost just £3, compared to £15+ for a 'proper' billy.
Do you have a rucksack? If not, or if you need a newer/bigger one, you have two options: military surplus or civilian. Army surplus are cheap, tough and available. But you can look a bit conspicuous when in public. I prefer to avoid that. Millets own-brand sacks(can't remember the name) are a good option, and avoid help avoid looking too dysfunctional when out in public. I recently bought a Highlander Forces 66. It was only £45 and I'm really chuffed with it. Highly recommmended. I got mine in plain green rather than camo - a sort of half-way house between looking either too military or civilian.
Anything else is extra. And you'll only know what else you need when you go out.
Sorry if some of that was patronising/obvious - not sure of the extent of your experience. Have fun.