Down has to be "contained" in quilted pockets (which incidentally create cold spots at the stitching lines), otherwise the down would all fall to the bottom of the bag! However, each of the pockets has a small gap to the surrounding pockets to allow the down to be manipulated back into place (useful if a seam leaks, but also creates the opportunity for ongoing transmission of down around the bag, so some quilted pockets are overloaded, and some relatively empty, again creating cold spots). When it gets wet, it doesn't just lose all insulating value, but clumps up. Careful natural drying followed by a never-ending "push the down back into equal pocket volumes" then ensues!
Clumping can also occur as the down gets compressed resulting in broken feathers etc, mixed with body oils/sweat that you pump out nightly. Compression occurs not only when you leave it in a compression/stuff sack for any length of time, but also in use courtesy of our body weight. Which is why down bags need regular professional care/cleaning if they are to retain their insulating prowess. I admit that, over the years, I've tried to wash my own down bags. It is a right pain, and the drying/manipulation process never-ending. So I took to sending them off to a professional cleaner (suggest you use a down specialist rather than a high-street dry cleaner). It's not cheap, but better warm than freezing!
A dry, new (or freshly cleaned) down bag is a thing of beauty and comfort. A dirty down bag will miss it's temperature rating by tens of degrees. A wet down bag is the beginning of hypothermia