"Myths About Measurements" Discussion Thread

I wonder what everyone’s thoughts are on some of the confusion and arguments over headphone targets?

Personally I think it stems from a lack of understanding on why there is no “standard response curve”. When I researched on different philosophies and methods to EQ headphones, I came across these types of comments:

Categories of opinions on EQ (and headphone response):

  1. People who don’t trust EQ - No EQ (Manufacturers know best, sound signature of equipment is a good thing and most pure)

  2. People who only trust measurements (Harman is the “Standard”, cannot trust our own ears). 2a. Use Oratory and ignore the rest of the FR. 2b. Use Amir’s simple 2-3 filters and ignore the rest of the FR

  3. People who trust their ears (I just want it to sound good). 3a. Add a bass and/or treble shelf. 3b. Use Oratory and then sweeps to tame treble peaks. 3c. EQ by ear. 3d. EQ using sweeps to correct for personal HRTF and HpTF, then adjust tonality to taste. 3e. Use flat speakers as reference (Flat speakers is the one true reference) 3f. Use equal loudness curve corrected sweep as reference. 3g. Use pink noise only

I’m sure I’m missing a few more, feel free to add to my list.

The point being there are many schools of thought, many of them not accounting for the limitations inherent in mixing, mastering, manufacturing of headphones, and differences in how people hear sounds with headphones compared to speakers. Most people intuitively jump to the conclusion that “duh, sound is simple, you hear a tone play and it just needs to sound like what the artist envisioned”.

It’s no wonder why there are so many myths and misinformation on headphone audio quality. Not to mention the added uncertainties by manufacturers in order to justify expensive products, and consumers justifying their purchases.

You Headphones.com guys are doing a good job with your new graphs with more variability built in, but I think there are still areas of communication which could be improved upon to get your point across better. For example, an article or a short video rundown with intuitive graphics on why headphones (and not normal day to day sounds and to a large extent, speakers) creates specific wave cancellations and peaks in the cup + ear. I think starting with this one fundamental fact would open a lot of people’s eyes as to your reasoning to why there’s no one true curve or one true headphones. Do it like how some Youtube exercise channel put a link to a 5 minute warm up on all their videos. Put a link to this short explainer video in all your review videos.

4 Likes

A 5 minute warmup video for understanding audio is a fun idea haha. Maybe we can do something like that.

But yeah I tend to see it as like… expanding circles, with the ultra nerd niche stuff is in the center. I remember at CanJam SoCal last year having a chat with one of the engineers from a well-known manufacturer who sat in on Dr. Olive’s presentation with us. I asked him what he thought, and based on our commentary and interjections there his response was “you guys are SUCH nerds”. And while he was joking around with me, it’s a good signal that we really shouldn’t expect people to be nearly as interested in this stuff.

My longstanding criticism of the existing science, including Dr. Olive’s work, is that it’s done a particularly poor job of actually communicating this stuff to the masses, and a lot of unfortunate narratives have developed as a result - narratives that now we have to do a lot of work to untangle. That’s… at least part of the impetus for Cameron’s recent video.

It’ll fall on deaf ears over at places like ASR, where they position themselves as being interested in science, but so far I’ve only seen one reasonable critique of his video there, and the rest are just playing the authority game or doing the usual “I didn’t watch the video but it doesn’t match my worldview so it’s stupid!” head-in-the-sand thing. I was joking with Cameron on a recent live stream saying “they won’t believe you because you’re not old enough”, but that literally is the way they’ve criticized some of his content.

Anyway, in my crude model of the audio community landscape… the more time you spend learning/researching this stuff - the more you learn from people like Blaine and Oratory - the closer you get to that ultra nerd center, and the further out you go the more you see various confusions and misunderstandings - I imagine like what you see from the groups you mentioned. The thing is, even within the niche of the niche center point of this model, there is still room for interpretation.

And that’s not to say “we know everything”, because we certainly don’t. More so, just that the deeper you go the more you’re able to agree on certain fundamentals that you’ve been able to test or demonstrate that the wider and more populated circles are still unsure about. You converge on foundational concepts, and this goes to things like the value of EQ, ‘technicalities’ being FR at the eardrum, the metrics that matter vs the ones that don’t, and so on.

With that said… a lot of the stuff in that ultra nerd niche is also still conjecture. People like Oratory, Blaine, Joel, Griffin, myself etc… we’re likely to all agree on this stuff, but it hasn’t been tested at a sufficient scale to prove it. It is entirely reasonable to criticize our positions here on that basis. I think maybe the closest to actually testing what we claim is true with regards to some of this would be the work from Fabian Brinkmann - I think I linked it a while back, which is a good start.

But, as Cameron said in his video, there’s really little incentive for science to actually investigate this stuff to the degree that we might want. We’re quite fortunate to have enthusiasts like Dr. Sean Olive still looking into some of this to the degree he is able.

5 Likes

I think @GoldenSound said something quite on point in your recent podcast. For cars, people intuitively understand that acceleration metrics is related to tire grip and surface grip coefficient of friction. Even if you don’t understand the physics (and tire physics is another giant can of worms, as difficult to understand and simulate as headphones), at least the concepts of it is intuitive enough in an everyday sense. Everyone has walked on slippery sand, oily, or wet pavement at some point.

I think as a community there’s a lack of intuition. Partly because in everyday life people don’t encounter sound wave cancellations, or when they do they don’t notice it. And then on top of that, when this topic is being talked about, it’s rolled inside mildly difficult to understand terms like HRTF, or being vaguely talked about in terms of “oh our ears are shaped differently, therefore we hear differently”. Which actually led to some people down the illogical path that “Since that is true, everyone has a giant dip and wave cancellation in the treble region, affecting everyday sounds or from speakers”.

It also doesn’t help that there aren’t any good textbooks on headphones. Listening to stereo content with headphone is a bit of a barren wasteland in terms of the amount of academic research done or being pursued. No legitimate papers or books I’ve read on audio mention many of the terms that audiophiles like to throw around. In fact a lot of those concepts seem to be completely ignored or swept aside by audio researchers because the concepts themselves are very ill-defined, too personal and therefore not readily quantifiable.

I truly believe that as a community there are some pretty low hanging fruit left in terms of accessible educational videos which resonates with the layman or newcomers. We need to build up some intuition about why the treble region of headphones FR is so wild, why people want more or less bass, why it’s a bit of a fool’s errand to chase “flat response” or “studio accurate” response in headphones, especially as a hobbyist who doesn’t produce music when accuracy matters less with a personal listening device.

Gotta get some MEMS mics in your ears and your 3d printed head soon!

That’s right, there are some small and isolated progress being done by the industry, like Grell’s idea for the OAE1, Dan Clark’s idea for the AMTS. I think the audiophile world is overlooking some of the new designs being done in the gaming headset market. I kid you not, the soundstage of the GSP600 closed back is truly amazing. And after extensively EQing it to fit my head and taste, I find it at least equally as enjoyable as my E3 and Aeon RT Closed (both EQed too). I find it utterly amazing how none of the audiophile closed backs have this sound stage or design. Granted without EQ these headphones are pretty crap, but it’s just an example of how parts of the industry does experiments and research which goes unpublished and therefore, not widely understood and adopted.

1 Like

Had a quick question about the video, in 10:15 @GoldenSound talks about how the spatial enhancement feature on that particular DAC, and how SINAD/THD/etc. don’t showcase this change in sound.

However this would appear as a change in the FR output of the DAC right? I don’t see ASR post FR metrics of DACs, but unless I’m misunderstanding something, wouldn’t the change in sound still be relatively easy to communicate with FR as an added measure?

1 Like

FR itself is also unchanged.

There are various more dynamic factors that won’t be able to be accurately represented in the more basic measurements.
Even just basic music production techniques like a dynamic EQ or M/S processing would in most instances not be evident in these tests.

That’s not to say they can’t be measured, they easily can, but just you’d need to construct/use a different test to see if these are occurring and characterize them.

I do plan to do a simple music recording vs reference test in future measurements similar to what was shown in the video. As this will then show immediately if something in normal operation is occurring that is not obvious from other measurements.

2 Likes

I’m still a little confused on why exactly it wouldn’t show up in FR, I’m assuming it might have something to do with using test tones as a testing method rather than music or m-noise?

Otherwise though, looking forward to the video.

1 Like

Various things will have an effect without changing FR.

Higher distortion won’t change FR. (Since even if you have high enough distortion to meaningfully alter the magnitude of the signal, it’s altered the same amount at all frequencies unless the THD vs freq is nonlinear.)

Crosstalk won’t affect FR since if measuring both channels at once, again the magnitude alteration will be constant. And if measuring just one channel crosstalk into the other channel has no effect anyway.

More dynamic changes that either only kick in or whose modifications are dependent on the content of the signal or the relationship between the signals in the left/right channels won’t necessarily alter FR either.

Again in music production there are huge varieties of tools like delay, reverb, M/S, compression that won’t directly affect FR. If FR was always the thing changing then EQ would be the only tool ever needed.

Even in situations where other factors are nonlinear vs frequency, so for example a device with way higher crosstalk at high frequencies which would mean that more of the signal from the right channel leaks into the left, thereby either superimposing and increasing amplitude at higher freqs in the other channel, or phase cancelling and reducing it, the fact that crosstalk could be at say -40dB (which would be considered very high), that only changes the actual FR itself by a theoretical max of about 0.08dB.

The reason that when looking at headphones the FR is for the most part all you usually need to look at is because they are almost always minimum phase devices, and a completely linear-time-invariant system, so there’s almost never any unpredictable or unusual changes occurring besides what’s described by the magnitude response itself (besides a couple IEMs that have been shown to behave as acoustic expanders for example). But with a DAC there’s a lot more stuff that can be intentionally or unintentionally altered without changing FR itself.
Even from an unintentional standpoint, an audibly high level of jitter for example won’t directly change FR since you’ve just got X amount of jitter added no matter what you’re playing, and so the magnitude of any given sine is still going to be the same.

4 Likes

This helps out a lot, appreciate the explanation.

1 Like

Great post Luke. I’m so much a 3c. I find that when I’ve taken the time to do Harman, it comes fairly close to sounding good - IF and ONLY IF the recording engineer didn’t muffle the sound by sticking his head up his ass.

With the right equipment, I don’t NEED EQ - but often find that Oratory does help. But my ears, equipment and my preferences are exclusively my own. Hell, your 2013 BEATS may sound good to you, who am I to say? If you’re a baby bird, regurgitated worms probably taste better than medium rare Delmonico does to me.

That doesn’t mean I don’t have a real interest in seeing what’s been done here with newer measurements and a good sense of humility about how different people hear and that we don’t know everything.

1 Like

I know I’m necroing this post, but this is my formal stance on what is happening in audio. We saw the same thing when NwAvGuy beat the drum about output impedance years ago: it went from “well, less than 10 ohm, for sure”, to everything trying to be a super perfect voltage source. Which is good…so long as it’s good, and it’s the same with SINAD. In isolation, getting an inaudible “improvement” to such specifications for free is all upside. The issue is that you run into things like “having less unconditionally stable amplifiers”, which is very much a modern problem (you did not see a lot of solid state amplifiers catching fire in 2003, I will tell you that).

In that respect, I do wish that people would stop obsessing over the easily-measurable but audibly-irrelevant and focus more on the stuff - like phase margin/stability, fault state protections, etc - that might actually stop somebody’s headphones from melting someday, but I guess I tilt at windmills there.

@GoldenSound obviously has a different view than I do here, due to perceiving some differences in “transparent” equipment. I would still very much like to do some extended testing to isolate what he might potentially be hearing, but I personally view that as out of scope here.

As a footnote, 100dB SINAD isn’t actually “enough”. Technically, 140dB SINAD isn’t “enough”. The reason is simple: SINAD is the level of Noise And Distortion relative to the Signal, which is valid only for a given level, which doesn’t necessarily match “the absolute loudest you can expose your ears to”. A 120dB SINAD dominated by noise referenced to 2Vrms would mean that the output noise is approximately .000002Vrms, or -114dBV. A Campfire Andromeda has a sensitivity of approximately 143dB/V, meaning that the residual noise of such a device on the Andromeda would be 143-114 = 29dBSPL, which would be an audible hiss.

This is a fairly niche case and mostly reflects the shocking irrationality of Campfire Audio’s design process, but it is technically true that to have a DAC or amplifier whose noise is unconditionally inaudible with headphones, it must have a noise residual which is below the threshold of hearing when used with the most sensitive headphone, which gives a rather absurd value of something like 18nanovoltsRMS

4 Likes

I am under the weather, so my thoughts aren’t especially coherent at the moment. But the gold standard for me is still the in-ear response of neutral speakers in a semi-reflective room.

If you’re looking for targets, models, reference points, baseline levels, benchmarks, or touchstones to use for headphones and EQ, my advice is to start there.

Is it accurate to say that one should care about high SINAD levels if they intend to EQ sub bass up? I’ve seen some graphs indicating this is a region that can introduce a lot of distortion with PEQ

Not SINAD… that’s an antiquated index score for evaluating source equipment. Harmonic distortion and excursion limits for headphones, sure that may matter a bit with regards to EQing up the bass. Harmonic distortion for headphones is generally going to be higher than what the amp or DAC imparts - unless we’re talking tube amps.

Also, with passive headphones, noise isn’t something to consider either. Think of SINAD as a noise and distortion index for DACs and amps, but you have little to no reason to care about this unless you’re treating it as an “is it broken” index.

2 Likes

Even if we’re talking about tube amps, you can very easily make a headphone amp with a lot less distortion than your average headphone, it should be noted. You can of course end up with something grossly nonlinear, but that’s not something that has to happen

1 Like

I really appreciate Cameron’s video!! Very well thought out and concise! More of this please!! I tried the 2k dip and it actually sounded better to me - hard to describe other than more clarity - less cluttered and sooothing - not sure if it is placebo but for now I like it. Is it because of the cancellation due to the distance between the ears? If so I think Resolve may need to try the dip at 1k - just kidding of course - I love what you guys are doing - keep up the great work!!

Thanks! that clears it up for me

I also have watched Cam’s vid. I can’t remember if he mentioned MDAQS though, which is a more perceptually based method of rating some of the above, developed by Head Acoustics. I don’t really have an opinion on it, pro or con. But have looked at the some of the results in the Sound Guys reviews (their recent Sony MDR-M1 review includes this chart, for example). And I think Headphones.com could probably also use this.

1 Like

Yeah the guys talked about the pros and cons of a single number rating in their 4 hour long podcast (why are they so long, lol) so I can’t see them adopting something like that for various reasons.

I had a look at MDAQS and tried to find out more information about it on HeadFi but no one replied to me about it. It’s an interesting concept. Jude on HeadFi said the OAE1 preview video (unlisted, linked from Drop.com’s page) that MDAQS score was the highest ever for the OAE1 in terms of immersiveness.

Granted I also think the OAE1 has amazing imaging performance after EQing, but the stock FR scoring highest for immersiveness is definitely something I find a little bit hilarious.

@Mad_Economist and @Resolve may have more to say on the topic, but I believe we are all generally fairly skeptical of MDAQS.

A lot of what it does is fairly unsubstantiated in terms of whether what they’re calling timbre or immersiveness actually correlate to what people are experiencing when they describe those subjective effects. (OAE1 having the highest ‘immersiveness’ score is probably a good example of how this metric may not be particularly well tuned…)
And a lot of other aspects of it sort of present particular metrics/measurements as ‘new’ when actually it’s just a different way of showing existing data we already have. That or are IMO arguably incorrect in approach. Temporal behaviour and frequency domain behaviour in a linear-time-invariant minimum phase system like headphones are directly linked and you can calculate one from the other. So measuring or evaluating ‘temporal’ behaviour as a standalone metric is from the get go a questionable approach.

But beyond that, there is also just the issue described in the myths video which is that trying to boil down a complex experience with many aspects and contributing factors into one score (or even three), is fraught with problems. And personally I do not think MDAQS has solved them.

4 Likes

Yeah I met with the guys from MDAQS a while back and they gave a good presentation on it at one of the CanJams. My understanding of it is that the scores are based on a perceptual model that are calculated by an algorithm for headphone metrological data that it gets fed. I was interested in it because perceptual models are useful tools in the project of trying to predict performance for people.

But at the moment my skepticism stems from a few issues:

  1. It’s unclear how this is any different or better than just looking at FR against a target. Yes, you’re scoring it, but those indices are generally comprised of FR. And since headphones are minimum phase, it remains unclear as to the output’s relationship with other data. It may make sense for other types of audio devices.
  2. Jude seems to think because you do see positive scores for certain products that you wouldn’t expect from existing predictors, there’s something there. However… I’ve also tested some of those products, and many of them are quite bad. So at that point it becomes unclear to me how good of a predictor it actually is, or if it even does a better job than PPR (which admittedly can also yield bad results).
  3. Without being able to verify what’s ‘under the hood’, it becomes difficult to say “yes, this is providing useful information that I can’t get from other indices”.

I remain open to the possibility of there being something to this, and generally I’m interested in what they’re doing. So I want to make it clear that I’m not ruling it out or anything, it’s just that I’m approaching it with a bit of skepticism as to its usefulness for headphones.

4 Likes