A/B Testing Experiences

After sending my Schiit Magni 3+ in for warranty service a few weeks ago, I decided to get a cheaper amp to power my HD6XX in the meantime. I ended up going with a FiiO A3 that I found on Craigslist for ~$30.

Fast forward some time and I finally received my Schiit back from service, good as new. But this left me with a decision to make. Do I keep both, or sell one? Do I really need both? If I do just keep one, which one would it be?

The only reasonable answer was SCIENCE!

So I ordered an A/B switch off of Amazon and had my brother help me with a blind A/B test between the two amps with my trusty HD6XX.

We volume matched both amps to a somewhat louder than normal volume using a constant test tone. I put the headphones on, closed my eyes and he started playing AC/DC. Once he clicked the switch several times in quick succession to ‘shuffle’ which amp I was listening to, he started switching from one to the other in random intervals. Once I had gained enough confidence, I told him to stop, and indicated which amp I thought I was listening to.

During the tests, I was never very confident in my decision, but to my surprise, I was able to pick the correct one every single time. To me the only difference seemed to be the clarity in the upper frequencies.

However, using my brother’s HD8 DJ’s, there was absolutely no difference. The results were random.

I figured I’d share my experience and see what everyone else’s experiences are when it comes to direct A/B testing DACs and amps. If you guys have any tips or methods to make this more effective, or if you have any interesting anecdotes, I’d like to hear.

4 Likes

Curious which A/B switch you used?

I used this one

https://www.amazon.com/KNACRO-Switching-Switch-Select-Output/dp/B07V5LS96J/ref=sr_1_28?dchild=1&keywords=headphone+AB+switch&qid=1597157001&s=electronics&sr=1-28

2 Likes

Good stuff, especially blind and volume matching. You might want to try introducing triangulation, both to test/hone your ability to identify and to increase reliability of the results.

For example, have him shuffle but not actually switch. And then have him play 3 samples, two of A and 1 of B, and you have to properly identify. Gain settings sometimes sound different also.

For science!

I think that sounds like a great idea! I just wrote up a post for an automated solution to this.

To my knowledge, apart from having a friend flip the switch, there’s no way to perform a good blind test.

1 Like

I’ll check out your post. For headphones, it’s tough to do blind comparisons independently (as far as I know). It’s easier for blind coffee/tea/libations tasting.

If these comparisons interest you, you might want to read our DNA Starlett tube amp thread where I discuss my A/B comparisons (granted, not blind) of the various headphone outputs (TCR vs XLR, gain, impedance).

That sounds like a good read. Do you have a link?

Here’s the most recent one and I believe you can trace precedents, or scroll up.

I recommend reading the articles in the “Big Sound 2015” on Innerfidelity. Tyll Hertsen’s contributions, including the conclusions for which I’m providing the following link, are fascinating, as are the other participants’ reflections; in fact, the whole thing is well worth reading: https://web.archive.org/web/20170331225330/http://www.innerfidelity.com/content/big-sound-2015-wrap-what-i-learned (note you need to use the internet wayback machine to access the articles now that IF is defunct).

Besides the valuable things the contributors have to say about blind tests, the articles also offer fascinating insights into what constituted the summit of headphone listening half a decade ago. It’s interesting to consider how things have changed since then (or not?).

6 Likes

Thanks for posting that link. It holds a lot of truth.

1 Like

The comments on the article are almost as interesting as the article itself.

I hope everybody clicked through to the post by Mike Moffat of Schiit: https://www.head-fi.org/threads/schiit-happened-the-story-of-the-worlds-most-improbable-start-up.701900/page-516#post-11921090

3 Likes