Your amplifier is loud enough at nine o'clock on the dial, the usable range is a sliver, and down at the bottom one channel is quieter than the other. That is not a fault — it is a volume pot being asked to work at the very bottom of its travel, where its two halves stop tracking each other.
The fix is to throw away some level before the pot has to, so you can turn it up into the range where it behaves. This works out how much, and what to build.
The obvious one is a resistor in series with one leg of the cable. It is one component and it works — but it adds its own value straight onto the amplifier's output impedance. Dropping a 32Ω in-ear by 12 dB from a 1Ω output needs about 98Ω in series, which turns that 1Ω output into a 99Ω one. For a headphone with flat impedance that is merely a level change. For a multi-driver armature in-ear it is a tone control you did not ask for, and the interaction simulator will show you how big.
The better one is an L-pad: a resistor in series and a second one across the headphone. It needs two components and a moment with a soldering iron, but the output impedance the headphone sees is now set by the shunt leg, so it can be made lower than the amplifier's own. This is exactly what an iFi iEMatch is, and it is why calling that device an “impedance adapter” is misleading — a plain adapter raises output impedance, an L-pad does not have to.
Attenuation placed after the amplifier drops the music and the amplifier's own hiss by exactly the same number of decibels, because both pass through the same divider — so on its own it changes nothing at all. What fixes the hiss is what you do next: turn the volume back up. The signal at the amplifier's output rises while the noise generated after the volume control stays where it was, and the ratio reaching your ear improves by the pad's attenuation. That needs two things to be true: that you have volume to spare, and that the hiss comes from the output stage rather than rising with the gain setting. A digital volume cut cannot do this at all — it quietens the music and leaves the amplifier's noise floor exactly where it was.
A gain switch does the same job with no soldering, if you have one. Going from a 6× setting to unity is 15.6 dB, which is usually more than enough. It works by moving the pot's operating point, not by changing anything about the imbalance itself.
And if you would rather solve it in software: cutting digitally is free and reversible, but check what it costs in bit depth first — on a 24-bit path the answer is usually nothing at all.