While we try not to bash other sites directly here, you point out some things that I bit my tongue about when I read over the links. I confess never to have previously looked at the site.
But your point about the methodology is good, and can apply elsewhere also.
It puts me in mind of 1980âs era Consumer reports when they reviewed something you knew well. They seemed to miss the point on a lot of items. At one point, they used to say that anything more than a speedometer and fuel gauge on a car instrument cluster was âconfusingâ. Then later if thatâs all you had, it was âuninformativeâ.
I think there is a pretty good mix here. In general, you need to find someone whose ears and and reviews you trust. There is a lot on the site about individual hearing differences - your ears or perception might really be different than what they are measuring.
Yeah, sorry I didnât mean to be bashing the site so much as scrutinizing the methodology for recommendations.
I do personally look at their measurements to compare, so at the very least I find that quite useful. And I hope they do figure out their conclusions/recommendations, because the project on the whole is laudable - just at the moment it has some distance to go
The graphs are the graphs, but whatâs âbetterâ really depends heavily on preference and what one is used to. The Sennheisers have a v-shaped signature, and it sounds like this is what you prefer. Nothing wrong with that. Also, comparing in-ears and over-ears based on frequency response is difficult, because they interact with our ears in very different ways (in-ears bypass the pinna), so the Rtings graphs need to use a different target response curve for in-ears than over-ears. That makes it a bit of an apples to oranges comparison.
I actually think that Rtings does a pretty good job of measuring the things we know how to measure that have a bearing on performance, itâs just that boiling it all down to a single number misses the nuances. Take for example the comparison between the Bose QC35 and Stax SR-L300. By the single composite sound rating, the Stax win by a tiny margin at 8.1 vs 8.0. However, when you look at the individual categories you start to see a fuller picture. The Stax get dinged for poor sub-bass extension (not surprising since theyâre open vs the closed Bose), and because Rtings weights frequency response so heavily that makes a big impact. OTOH, the Stax edge out the Bose in the mids and treble, and handily trounce them in Soundstage and THD, unfortunately these arenât weighted very heavily in the final score. Though I donât quite follow their explanation of how they measure soundstage, Rtings soundstage rankings actually line up pretty well with my personal experience and reading of reviews. And IMHO, THD is the closest thing that we have to a measurement of âresolutionâ or âqualityâ of the sound. I think IMD would be a useful addition, but no one seems to measure that on transducers.
So, if youâre looking for âresolvingâ cans with a nice soundstage and donât mind the missing sub-bass, Rtings clearly points you in the direction of the Stax over the Bose. If OTOH you want a lot of sub-bass for electronic music, movies, etc. then picking the closed-back QC35ii over the open-backed Stax doesnât seem ridiculous to me.
Itâs a good analogy. Just like with Rtings, I always find that to get the most out of a Consumer Reports review, one needs to dig into the details. If you just go by the overall score, youâll always end up with a Honda Accord, even if what you really wanted was a convertible sports car or a light duty pickup.
I actually had to praise them for what theyâve been doing with soundstage evaluation - not that I know if itâs accurate. You donât know unless you try, and theyâre certainly trying to lay the groundwork for a lot of headphones that donât get this kind of evaluation.
At the risk of sidetracking this a bit, on the THD front my experience has been that it canât possibly be the only thing we have to measure detail and resolution, simply given there are headphones that are more resolving that have higher THD than ones with lower THD. At the moment I donât think it can be ranked so easily because of how significant frequency response and tonality are to resolution as well.
I think youâre right that weighting is the cause of their problems, and all it would really take is to include a reviewers preference scale where the performance categories are identified. But thatâs again why I think theyâre prioritizing evaluations and recommendations based on the tools they have to evaluate - assuming itâs exhaustive. I canât help but think that if it were exhaustive, they wouldnât end up with such ridiculous results.
One thing I found out via Discord chat about this very topic this week was that the compensation curve isnât actually the harman target, which I had suspected on Rtings. Itâs actually a combination of the Diffuse Field and Harman targets mixed together-- however (and this is personal preferencesâŚ) they choose the Harman low end and the DF upper end, which makes it, essentially, a U or slightly V-shaped compensation curve.
My ideal compensation curve is somewhere close the exact opposite of this, with slightly elevated bass region DF target through the mids and a Harman like treble response with extended upper highs.
I havenât seen anything else that I find particularly convincing. Much like the overall rating, I think THD is best examined in detail rather than a composite number. Weâre pretty insensitive to distortion at low frequencies but much more sensitive at higher frequencies, so the profile of distortion to frequency matters. I find that the relationship of THD to SPL matters as well. All things being equal, stuff sounds better at louder volumes if for no other reason than they take us further above the noise floor. Unfortunately, many headphones (like the HD58X) start to distort as volumes get too high, which I think places an effective upper bound on how much they can âresolveâ under normal listening conditions. Others, like the LCD2C, actually distort less as SPL increases, and I do in fact find them to be quite âresolvingâ when listening at higher volumes (except for the missing frequencies in the high mids, but that can be alleviated with EQ).
That qualification for THD seems super important to note. But my experience has also indicated that there can be quite distinct improvements in detail and resolution where there isnât a measured improvement in THD. For example, the Focal Elegia has lower THD @ 1khz than the Utopia, but the latter is categorically superior in terms of detail capabilities. Similarly, the Clear boasts around the same as the Utopia, and itâs still very obvious which one has better resolution. I also wouldnât say that the Utopia has the better or more âaccurateâ frequency response either. Of course, it might be the case that higher frequencies would differ enough where it confers resolution benefits to the Utopia, but I kind of feel like itâs just better throughout the entire range as well.
Moreover, if you look at the THD for the Nighthawk, itâs at least as good as both, and thereâs no world where that headphone does as good a job at detail retrieval as the Utopia (even if EQâd).
The Nighthawk seems like a great example of how frequency response affects perceived detail. That fat bass is going to mask a lot of higher frequencies. Iâve never heard the Nighthawk, but I wouldnât be surprised if it sounds a ton more resolving after EQâing to a more neutral frequency response.
I own the NightHawk Carbon. The NHC comes from the factory with pleather pads and velour pads. The bass response drops from extreme with the pleather pads to warm-normal with the velour pads. With velour one can hear more of the treble, and the fundamentals of the drivers become apparent.
A major consideration with all dynamic drivers is construction material, as they often take on undertones to match the material. I take this as a function of natural resonances emerging from constantly and rapidly moving driver mass back and forth.
My Focal Elex have metal drivers and were distinctly brassy/tinny for the first 20-30 hours. They remain intense and brassy when played on a noisy amp.
My Sennhesier HD-600s have a slight boxy TV speaker or boom-box tone from the plastic drivers.
My NightHawk Carbons have biocellulose drivers. Itâs a product that results from bacteria working on âfood,â and the tonal properties resemble paper more than anything in my personal experience. Speakers with paper cones (and the NHC) tend to have a self-damping lack of precision. The NHC generally flatten out dynamics and flood the whole spectrum with a bassy blanket.
Ultimately I agree with the gist of DIY Audio Heavenâs trashing of the original NightHawkâs treble; itâs not very good for the original MSRP:
âThe âdarkâ character masks the treble issues this headphone has.
With EQ some improvements can be had but a stellar quality it wonât reach.â
Yeah that was my conclusion as well. Iâve personally owned both the Nighthawk and the Nightowl, which has tightened things up a bit. But even with EQ I canât compare it to any of the Focals when it comes to detail. To me, this explanation is precisely whatâs missing from measurement tools, and in my opinion requires qualification for proper headphone evaluations.
Now the question of whether or not someone enjoys that is an entirely different one. I can completely understand why people are drawn to them over some of the more resolving headphones.
I enjoy Fransâ reviews in large part because he correlates what he measured with what he hears. As usual, he has an explanation for the poor treble resolution.
The rising edge is nice and fast and 30kHz is about equally loud as 5kHz.
BUT âŚ. it doesnât come near the ârequiredâ level. On top of that there is quite some ringing following the pulse. A short âpulseâ which our brains would like to use to determine the âlocationâ of the sound thus is too low in amplitude and because it is ringing also âsmearsâ that energy over time costing resolution.
So looking for ringing in CSDs or impulse response seems like a potentially useful adjunct to distortion measurements.
Iâd also be curious how the bass distortion looks when EQd down. If itâs still in the 1% range, I wonder if thatâs high enough to mask some detail in the midsâŚ
Itâs interesting you mention CSDs because that was the first place I started looking. It turns out the usefulness of CSDs is currently at best still inconclusive. I know Tyll used impulse response and square wave response for certain things, but itâs unclear if those were predictive for resolution. I think there were some notable examples like the HD800, but it was less predictive when it wasnât being helped by frequency response. The gist of what I discovered for CSD is that for the majority of headphones, the time domain is directly proportional to frequency response - which means that thereâs no special information derived from CSDs that isnât already given in frequency response. At least⌠thatâs the theory.
Iâve been a bit cagey about my personal views on this because Iâve seen counterexamples that contradict the theory, and Iâm also on the fence about how thatâs applied to the headphones of today - or how itâs applied in practice. But I can certainly see the other argument as well, that itâs as of yet unclear how audible time domain information is. It could be one of those things that isnât noticeable when you focus on specific elements of the sound, but redounds to benefits or detriments in the supervening experience. That is to say, even if you canât find it on its own, it might still have a significant impact on how a headphone sounds overall.
Gotta say the Shure SE425 were probably the worst purchase decision Iâve ever made in the world of headphones. They were supremely uncomfortable, exceedingly boring sounding and worst of all, i couldnât return them.
