Some of the best advice I was given when I first started out was that I should take actions to stay warm before I begin to feel the cold.
Some tips for the OP:
Bring lots of layers rather than one or two thick pieces of clothing then in the evening as the temperature drops, begin adding layers before you feel the cold. If you are too hot you can always take a layer off but it is much harder to get warm once you get cold.
If there is plenty of wood around then build a reflecting wall behind your fire to direct heat back at you, similarly if you have a tent or tarp try to position it to provide another reflecting wall behind where you are sitting. Using this technique I've been forced, many times, to strip off layers and sit past midnight next to the fire in just a t-shirt and shirt.
A good sleeping bag is worth its weight in gold, one of the most important pieces of equipment we take in my opinion.
Extra insulation underneath you is essential and worth more to you than extra insulation on top.
If you do begin to get cold then stoking up the fire, cutting more firewood, making a brew and doing some simple exercises are all good ways to help warm back up again.
I hope you give it another chance OP, with a bit of time, learning and a little money for kit then you can make camping quite comfortable.