The latter bit sort of answers the former there, in a lot of cases - there are other cases where measurements aren’t representative of on-head performance, but a lot of times folks extrapolate a bit much in terms of how a measurement will sound, IMO. “doesn’t sound like it measures”, to me, means one of two things - either 1, the measurement doesn’t match what’s going on on your head, at your eardrum, or 2, the expectation you had about how measurements correlate to sound wasn’t quite spot-on.
Just to check, do you mean the high Q/sharp notch that’s right around 3khz, or the general dip in that band that Resolve showed? High Q notches like this one are generally known to be pretty difficult to subjectively detect (there’s a section in Harman’s How to Listen for finding them which is pretty neat!), because the thinner the band that’s notched out is, the less likely that a significant amount of musical information is going to fall there.
The generally being below target in the ear resonance band I’d expect to be comparatively much more audible, and to dominate the subjective impression of that part of the upper midrange.
Oh absolutely - measurements will most certainly lead your subjective impressions (or at least prime them). Of course, so too will the design language of a product, the circumstances you hear it under, etc - I wouldn’t say we “shouldn’t” measure before we listen, but if we do, it can definitely have an impact on what we hear. However, if we wanted to avoid impacts on what we heard, we’d need to blindfold ourselves and have volunteers carefully lower headphones into position on our ears, to avoid the other biasing effects 
It’s absolutely a significant effect and worth considering, however, and generally I think best practice as a reviewer is starting the listen > measure > listen loop with listening.
For consumers, however, unless you have easy access to a panoply of demo headphones, measurements are among the best option for narrowing down purchase choices at the moment - in that case, I’d definitely recommend that any of us with a budget constraint look before we spend, even if that means coming in with some priming for what to hear (perhaps it’ll push us in a positive direction, even - nothing wrong with biasing you to like your headphones more).
IMO it would be more ideal for “virtual demo” type systems to be used, either via binaural recording or equalization presets, but these have a lot of issues in implementation - they’re more intuitive to users, but they’re very, very hard to get right.
Kind of a complex one. I’ll quite strongly avow that anything which is audible in sound is measurable. That does not, however, mean that everything which we perceive when listening to a product is measurable in its sonic output. The classic example here is the visual impact of a speaker on perception - Sean has a great example with larger vs smaller speakers in this blog post - but for headphones ergonomics, styling, brand image, and probably even little details like personal history with a brand will likely have some influence on what we hear. Our subjective experience of sound is a complex interplay of psychology and acoustics, and we can’t (yet?) measure the brain side of things.
Perhaps when we have probe microphones that come with a cap of brain probes to monitor neural activity, we’ll be able to perfectly predict how a headphone is heard from measurements, but for now, I would say at least that we can predict the aspects that are based on its actual sound output. That said, I don’t think that 100% of information is contained in a single frequency response plot on an averaged ear - I talked a little bit about this with @Resolve on his podcast, although I didn’t get as far into the weeds as I’d planned to.
Absolutely agree, and I’m not sure how anyone could disagree with this position (unless you didn’t believe in biases, I suppose, but that seems a bit…eccentric).
As far as DACs and amplifiers - presuming we’re talking about equipment that’s “sufficiently linear” under the conditions it’s being used - I think you will find that experiments in limiting people’s exposure to such products to only the signal effects alone tends to support the idea that a lot of what’s heard there isn’t coming from the transfer function of the DAC or amp. Which, of course, is fine - if it sounds better to you, that’s good, right?
Please don’t feel rushed to reply - I’m kind of at loose ends this last day or so, so I’m being hyperactive on forums.
Profound!!! lol
I think that the only reasonable answer you can give there is “to some extent”. We definitely have measurements which we can reliably correlate with people’s perceptions . They don’t cover the full extent of what people perceive when listening to things at present (or, at least, won’t perfectly predict how people will rank different options), and there are areas where I think new ways of analyzing and gathering data could be useful¹, but we definitely can get a pretty useful picture from our current analyses, and our data acquisition probably isn’t the major bottleneck to improvements.
So now I’m more of “if I like it better, it is better” IE: correct for me.
