Why doesn t volume control settings allow greater range on low end iPhone

Maybe my ears are really sensitive, but for every iphone I've had, the lowest volume is not low enough.  My phone is often louder than I want in my headphones even at the lowest setting.  I find myself trying to lower the volume but ending up in silence.  After putting it back up to the lowest setting, I end up clicking down again a few minutes later because it's too loud for my surroundings and situation.  It's a huge volume jump from the lowest setting to completely off, and alot of times the lowest setting is too loud (even with normal buds or headphones).

The volume limiter setting only limits the upper end, but doesn't change the lower end (just makes the 16 set increments smaller in that reduced range).  I know there is also an option to turn down the volume of individual songs in iTunes options, but this problem applies to all aspects of the iphone beyond just my music library (Music apps, Videos, etc.).

Why hasn't apple expanded the lower volume limits?  There has to be a way, or is this not possible?  Can anybody think of a way to fix this problem?