Inside the industry, the people who make the decisions on how to produce music and dtermine the general "sound" of music, they are making very specific decisions on that finished based on two primary driving factors ---
How does it sound on the overly dnamically compressed radio broadcasts?
How does it sound on the low bit rate MP3 and internet radio services AND crappy headphones?
If that is the target audience, then what we get is crap. If you want to have a top 40 hit, that's the market you have to cater to. It is so rare that top quality sounding recordings return a healthy income for everyone involved. It is a business, after all.
I've recently been in studios where the initial mix is astounding and everyone in the room is very impressed in what they hear over the $10,000 studio monitors in the acoustically perfect room. Then the engineer will run it through a realtime lossy compression engine as is used for internet radio, and everyone wil cringe in pain at the result. So, the mix is tweaked until a compromise between the perfect reproduction sound and the internet radio sound can be reached. Neither is ideal, but it is what it is.
One solution is to return to the days when multiple mixes are produced - one licenses solely for internet radio, another for online music stores with low bit rates, and a final for CD/LP and high bit rate online stores (sometimes a last mix for dance clubs). The problem with that approach, as has already been learned, is the inability to control where a version is broadcast or distributed. Since digital is so easy to send wherever and whenever, the owners of the content find it very difficult to control where it gets played in the standard distribution model.