Blind Testing Source Equipment - The limits of discernibility

When I insert the male jack of a headphone into a DAC, depending on the size and material of the DAC box, the friction generates a characteristical sound that is not the same from one DAC to another.
Using male to female cords/adapters that will stay in each DACs during the test, then holding female plugs in the same hand to switch the male plug of the headphone from a female plug to another should get rid of this, and might help with a more consistant time required to plug/unplug the headphone.

4 Likes

like the custom “Hartaudio” cables with the Multi-Kit system.

a third person could name the cables x/y/z and hide the interconnects under a sheet, so the blind tester and his assistent are not knowing which is which

so your assistent could write down your answers on a pre-filled list with a previously build sequence by a random generator, she is following

not sure at all, if my bad english is any helpful with describing this somehow complicated setup

3 Likes

yes they do very well and trained people can get good, as is seen with some of the specialised “magicians”

1 Like

That’s a great idea!

The Hart cables are relatively inexpensive as well. I think @Resolve should give it a try.

3 Likes

Mine are finally shipping btw. Yeeeyyy…

Now I wonder if there’s a A/B Box with mini-XLR inputs around. Make or buy decision? :thinking:

Definitely buy for me. :blush:

2 Likes

oh a mini-XLR switcher box? - intersting question!

I would use adpters
and also will buy one like the cheap passive noobsound 3to1, for example

there are sme with RCA in/out and some with XLR in/outs

for both(XLR and RCA) I found the 2to2 in/out, yet

though some guys will have way more experiance with this sort of splitter and can certainly recommend something better suited

2 Likes

I have done alot of blind AB testing over the past years…

For me its about trying to tell the difference in devices, can I really hear audible differences, Thats one “test”…

The other is which one do “I” prefer?

If its really hard to discern differences, good or bad and/or if it takes a long time or is hard to tell…are the differences really that great ?? If I have to spend hours of straining to guess at these things its telling me something: Both devices are pretty similar for all intensive purposes!

Then its down to which one do I like from a cost and/or a subjective analysis.

Some folks pride themselves in being able to hear very slight diffrences…ok thats great, magic ears etc…but do they really matter…do you like or perfer one over the other…being honest with yourself.

One point thats made over and over is there are a lot of good choices for gear that are pretty similar over a wide price range…

Some folks have tried to shock the audio world with “hey my $100 thing is just as good as your $1000 thing” with this kind of testing…some of us like this and some of do not!

It goes back to can I hear a difference without a bias and if I can which one to me do I like…(and keep?).

Great discussion!
Alex

6 Likes

I found a well written piece in a publication called AudioXPress. It takes on the “measurement vs perception” debate in a thoughtful manner. The piece is in two parts, with the first outlining the history of blind and ABX testing in the audio industry, which is now well established and dates back to 1996 when the Audio Engineering Society (AES) published “AES20-1996: AES Recommended Practice for Professional Audio—Subjective Evaluation of Loudspeakers," which has since been revised to provide guidelines for the subjective evaluation of all sound system components. There is no reason to reinvent the wheel so I suggest either reading the AES publication or this article, which is a nice synopsis that is easy to read.

The second part gets into the debate and ends with a link to an ABX listening test one can do to determine if you can hear the difference between two amps that measure similarly, one being solid state and the other tube.

The article is a good read and thought provoking. The listening tests are fun.

I should add that I don’t stand on either side of the measurement vs perception debate and like many on this forum straddle the two positions; wavering between the two as I dive deeper into this hobby.

This is a link to the first part with a link to the second part at the end of the page.

7 Likes

Take Hart audio ecosystem, for instance. The only common interconnect is a mini-XLR connector. Hence, the question. Of course, that would only apply for those folks who actually own those type of cables.

Cheers.

1 Like

I agree that blind tests conducted by just anybody probably don’t serve much of a useful purpose in terms of making recommendations and in this context I think that many who are fanatical proponents of it place far too much emphasis on expectation bias as a factor that undermines one’s ability to make a reasonable sighted evaluation. So when I go to forums like this for advice I am not that interested in reviews that were done blind.

That said, for me, as a consumer, who often finds disparities in opinion by reviewers thus making reviews sometimes more confusing than helpful, the benefit of blind testing would be akin to the type of review Consumer Reports provide when they run consumer products through a gamut of tests. But in order for blind tests to be useful they would have to be conducted in a manner similar to what Harman was doing with speakers when they were testing for preference; in a controlled environment according to AES guidelines, with trained listeners tested to not have hearing loss, and repeated with different groups of listeners to ensure consistency in results. The idea being that if one person hears something in a certain way and another hears it differently who is to say that either is wrong or that they were not influenced by external factors. But if many people who are trained to understand how to listen can corroborate what they hear and what they prefer in an environment controlled to exclude external factors then that to me would make a recommendation much more trustworthy.

Ideally the testing should be done by a research body that has no commercial interest in the products being tested, kind of like Consumer Reports or the BBC or the National Research Council of Canada who for many years funded research into phycoacoustics and audio reproduction.

4 Likes

Yes, and I think I may incorporate it into my reviews/comparisons a bit. I think there’s room for a “here’s how I hear the product, here’s what I think, and here’s how well it was distinguishable in a blind test”. That way it covers more bases than just the type of evaluation it gets in a blind test, but also satisfies those who specifically do want that kind of comparison.

2 Likes

I think that would be good for both the reader and yourself. As you mentioned earlier, doing a blind test does make you think a bit more about your evaluation and is a good exercise in honing your listening skills.

1 Like

I am in agreement with you in that I am not sure what the ultimate benefit of this type of testing is. What is the translational benefit to the audience and how is it perceived by them?

I think A/B testing in this context is still very subjective and multivariable even if you were to blind it as you are only blinding yourself. One person doing a blind test is still a sample size of one so I am not sure it can ever be considered anything more than an interesting data point for a subjective measurement.

Its a fun thing to attempt though and can create some findings which may lead to more interesting discussions especially along side of value propositions. Imagine the fun if you put in a DAC you had never listened to but I think that was besides the point of what you were trying to determine.

1 Like

Yeah exactly. It’s also limited by the fact that it’s a short-term test. That’s not enough to be comprehensive either. The process for identifying audible differences in sighted comparisons generally involves the ability to listen for specific things. So you can run a sighted test where you’re paying attention to or looking for certain qualities, and you can do this 10 different times and look for 10 different things. In a blind comparison, those differences need to jump out at you in order for it to really be noticeable. And the problem there is that with high end DACs, very rarely do those differences jump out at you. It’s a subtle difference at most.

2 Likes

I think the amount of effort and consideration this would require to really work properly is fairly large when you stop and consider everything. This is also something that can easily get bogged down in esoteric discussions, which I am not necessarily against, but can muddy the larger picture

My opinion is just an N=1 but you are already one of the better and most consistent reviewers out there from what I have personally seen so adding this testing should just be a sideline to what you are already putting out there. Unless you are getting a big audience demand for it I would just do it as an item of interest or if there is an important context for it like a manufacturer claim that their $100 DAC is no different than an Yggy or something. However, if this is something of keen interest for you that you wish to explore go for it, I’ll still watch.

1 Like

I have been recently thinking about those occasions where I have a visceral reaction to the music I am hearing, and trying to correlate those experiences in my mind to the objectively ‘best’ sounding setups - theoretically the goal of blind listening.

I’ve concluded that it isn’t always the most revealing DAC or cans that do it for me, which causes me to wonder if I would be listening for the ‘best’ sound or just the one that resonated with me through some combination of factors.

An example: the track ‘Will the Circle Be Unbroken’ from the Nitty Gritty Dirt Band’s Vol. 2 album of the same name (not the original classic album). Listening on my Meze 99 classics from multiple sources, this track comes alive in a way that other cans can’t seem to replicate. I can hear more detail on my Elegias, and the experience is perfectly pleasant, but it is not as musically effective. To circle back to the thread’s premise, then, I would pick this setup 10/10 times despite it being objectively ‘worse’. But the differences in a test will not always be as pronounced, particularly in those cases where you’ve caught fire in a bottle with the synergy of the recording, DAC/Amp, and headphones.

So for me, blind listening may not be a tool I’d use for trying to identify and upgrade to better stuff, but rather to help hone my understanding of what sounds I like and how best to maximize the chances of finding those combinations.

Ramble over. :slight_smile:

3 Likes

For me, at least, I appreciate blind testing as part of a product review because I am convinced that a disappointing number of audio reviews are not successfully avoiding placebo and expectation bias. It’s a comforting addition to a review that goes a long way to reassure me that the reviewer isn’t just being awestruck by a price tag or a billet aluminum chassis and hearing all sorts of things that just simply aren’t real.

It’s a vital counterbalance to all the dubious “difference was night and day” reviews or favorite gear lists that seem mostly just sorted by price, descending.

Reviewers who don’t at least acknoledge the value of unsighted comparisons lose some credibility with me. Not to mention those who downplay or even denigrate the practice.

5 Likes

In the end, for me at least, it’s somebody’s opinion and impressions of something. Some are good at doing it (like yourself) and some aren’t. As long as it isn’t too far on either side of the spectrum of Subjectivity V’s Objectivy I am usually ok. We all have preferences in the end though and most of the time the differences between the different gear are small anyway. Just my opinion of course. But this is a great discussion.

6 Likes

I think that one of the problems of reviews in general is that by their nature of having to describe how a piece of gear sounds, usually by focusing on a few key characteristics and in comparison to at least a reference, they can’t help but give the impression that the differences that characterize a component are much greater than they were actually perceived. One purpose of a blind test by a reviewer might be to provide a reality check on the impression they might be giving in their review. Perhaps it could be looked at as a humbling experience and a reminder to tone down it down a bit.

Agree that you have to be concerned about reviewers and what they deem good and bad and what the influence is. If a reviewer was unwilling to concede the potential value of unsighted testing then they are of little use. The issue for me lies with the implementation of it. However, if we accept it just for what it is, an attempt at blind testing, then great. It’s when it devolves into arguments about the nature of the actual testing that things bog down.

I will accept a little bit of human error/variance in this hobby as the human variables are extensive, Thats just me though, people can love blind testing and I also use that kind of stuff for work so…

1 Like