Leave them be. I've got them in my garden and they're a constant source of well tilled soil. I collect all the earth they dig up and stockpile it to mix with other soils to fill raised beds and pots. I don't know where I'd be without them at the moment
You do have to be careful though, especially in a dry summer like this one has been, if you're watering certain areas of the garden the worms collect in those areas and attract the moles. All of a sudden your plants will end up a foot higher than when you planted them...