As you say, it is one of those ideas which needs to be considered in balance with other factors. However at the moment it isn't considered at all. Take for example the subsidies which farmers receive for 'keeping fallow land in a fit state for agriculture'. Basically this means that in order to get the cash, they have to ensure that the land (which they aren't using anyway!) is prevented from developing any significant habitat value (or for that matter, recovering agricultural value).
Now, in the UK specifically we don't have the significant issues with grazing patterns which arid grassland areas have (desertification when there's too many grazers, and worse desertification when too few), but over-grazing areas of land and then, even though you don't want to use it anymore, uprooting everything that doesn't look like grass, keeping hedgerows cut back (though not laying them properly, because it's too labour-intensive), draining it, etc, is a nonsensical way to manage the land; it keeps habitat value low and does little or nothing to advance the recovery of fertility.
Overpopulation doesn't really cover it either; at least it is an oversimplification. When natural gas supplies run out, we'll have a problem. But until then we can use the Haber process to create nitrates out of thin air, and produce all the food we need. We could do a lot more by focusing on avoiding waste, and by that I don't really mean the amount that private individuals throw away, but rather the waste currently created by our supply chains. The patterns of ownership of the food supply have a lot more to do with the famines that you see in the developing world than the carrying capacity of the planet - much as the Irish were dying of starvation even as boatloads of food sailed for England, we encounter famines (of staple foods) in the developing world, while cash crops are raised successfully for export. Saudi Arabia, for example, is busy buying up tracts of the best farmland in Ethiopia. I can't see that panning out well for Ethiopians.
Incidentally, I think that the patterns of ownership probably have a lot more to do with the degradation of farmland generally - nomadic people and migratory animals moved on and this permitted land to recover. Once you start building fences, you create the conditions for the degradation of land.