Fc-Construct's Official Headphones.com Community Forum Thread

The HEADPHONE Community
1

Similar to the rest of the Headphones.com team, this post is my watercooler thread on my general thoughts about audio and the state of the hobby, fun things I’ve been up to in the community, and extremely questionable impressions to be taken with a mountain of salt.

Feel free to browse or ask any questions. A couple facts about me to know my perspective about audio in general:

  1. My introduction into audio was through FOH mixing for church bands. I play the piano but it was under the classical pedagogy. Unfortunately, I’m not really into classical music and more into badly produced rock.
  2. I tend to like IEMs a bit more than headphones but can appreciate both.
  3. I consider myself “source agnostic” - I don’t really care for cables or DACs/amps, but if I hear it make a difference I can accept that. Just that IME differences are once in a blue moon and it’s so minute you’d have to be doing A/B testing for half an hour to hear it.
  4. My educational background is in biochemistry, but I ended up working in marketing for my day job. So a bit of science, a bit of business in the way I see products and how they’re positioned.

Old Topics: N/A

Current Topic: A different approach to a ranking system?

4 Likes
2

The first topic I’ve recently been mulling about in the shower is what the best rating format actually is. There’s half a dozen of them out there, but lately I’ve been leaning towards a style loosely inspired by the Michelin guide. Precogvision and I had tried something like this for our show impressions at CanJam SoCal 2025, but I’m starting to think if I should expand it to all my reviews as a conclusion.

Adapting it for a proper review vs. show impressions, here’s the rating breakdown:

  • Not recommended - The sound of this product ranges from being outright bad to very mediocre. (I don’t expect many products to end up here).

  • Recommended - The starting point for decent. Potentially worth a listen if you have the time. (I expect most products to end up here).

  • 1 Star - Excellent sound and one worth going out of your way for.

  • 2 - 3 Stars - Excellent sound with a special sauce that makes the product stand above the rest and/or has a “justifiable” price.

The reason I’ve been trending towards this style is because I think it forces people to think about the meaning of the actual rating rather than glance at a S/A/B/C ranking list or 7/10 star approach that’s we so commonly see. There are more complex ways of creating ratings, and I’ve dabbled in creating something like that before in an old tier list. But for the purpose of clearly communicating a stance towards a product, I’m starting to find this Michelin-style system to be attractive. It provides a middle ground that gives a position, but also forces someone to engage with the content of the review.

4 Likes
3

That’s boring. You should use ACRS as the base for your ratings.

image
image1024×1536 338 KB

Note that in real life, nothing is 0 or 100, so let’s use numbers that will make people feel OK, but not great. The 70s and 80s come to mind. Then, we need a color scheme, in which the same score can have multiple colors.

No if you say that’s Wo, bro, it’s nowhere near as good as having something Hu or Re. I’m keeping it simple by using 2 letter descriptors. But these descriptors should be unique to your rating system, so people won’t psychologically carry over there feelings from some existing system.

Isn’t this really At (79+1) Green?

1 Like
4

Howdy, Fc. (Hope it’s ok to call you that, or maybe you have a different nickname you prefer?)

I don’t use IEMs. But would enjoy seeing more of your headphone reviews. As far as overall rankings are concerned, I think it’s usually best to follow the KISS rule (keep it simple…). If people want to go deeper, then they can read the whole review. But many will not.

Siskel and Ebert where ahead of their time in some ways with their binary approach of thumbs up :+1: or down :-1:. Meaning I can recommend it, or cannot. But they often qualified their choices. If you try to blend a binary system like that with a numeric rating, I think it might confuse some older folks (like me).

4 or 5 stars seems to be the big trend though on most internet sites. And that is something that is fairly easy for someone old and decrepit like me to process. And on the occasions when it doesn’t provide enough nuance or granularity, you could do half-stars.

I would hope in your reviews that you cover a number of different aspects of the product though, including the sound quality/neutrality, comfort/fit, ergonomics, quality of construction, and so forth. And maybe you’d want to give different scores for each of these things, before calculating the overall.

Just a few thoughts which may or may not be helpful. :slight_smile: Cheers.

2 Likes
5

I like that system. Though where would you fit a headphone about which we might say, “Some will love it.” For example: the Audioquest Night Owl. It might not be the kind of thing that you can say is “recommended,” let alone star-worthy. But is it “outright bad to very mediocre”? I don’t think so. It’s distinct. Some will love it. I rather like it myself.

1 Like
6

In my (pretty extensive) professional experience with rating systems, it’s tough to maintain personal consistency or inter-rater agreement with anything more than 3 categories.

No. Maybe. Yes.

Without oversight and conscious discipline, people tend to reinterpret and rescale anything more complex. They often refuse to use the extremes of a Likert-type scale, or they cluster everything at the top or bottom. Some people are too kind and others are too cruel. Also see university grade inflation, whereby softy teachers who want good evaluations and to get their students into grad school allow Cs to become As over time.

In my experience, a 10-point scale tends to become: 1 to 6 = ignore this product, 7 and 8 = good but clearly flawed, and 9 to 10 shift around given personal taste. Again, it’s tough to be consistent beyond 3 categories.

1 Like
7

I’ve spent hours, days, and years looking at and creating such rating systems. My first thought was that the top two and left bottom were fine except for the confusing rainbow colors (which have no meaning) . The bottom right is indeed nonsense. There’s huge danger in using red, yellow, and green for anything other than No, Maybe, and Yes, as we are so overtrained on traffic signal meanings that they’ll turn into No, Maybe, and Yes for most readers.

Decision algorithms often result in bizarre conclusions…

Summary

Consumer Reports…ASR…RTINGS…

1 Like
8

I like this. How do we translate to headphones? Something like this?

Don’t bother.

Try it first.

Blind buy.

2 Likes
9

No if you say that’s Wo, bro, it’s nowhere near as good as having something Hu or Re. I’m keeping it simple by using 2 letter descriptors.

This is amazing LOL

4 or 5 stars seems to be the big trend though on most internet sites. And that is something that is fairly easy for someone old and decrepit like me to process. And on the occasions when it doesn’t provide enough nuance or granularity, you could do half-stars.

My hesitation with this is that people often look at stars with their own predisposed notion of what the stars mean and runs into the problem where everything ends up being a 3 or 4 out of 5. I don’t think it’s a bad system and I’ve used something before in the past, but I don’t know if I want tot anchor reviews around it.

I would hope in your reviews that you cover a number of different aspects of the product though, including the sound quality/neutrality, comfort/fit, ergonomics, quality of construction, and so forth. And maybe you’d want to give different scores for each of these things, before calculating the overall.

I have something like this in my own personal tier list, but it’s quite complex lol. And doesn’t involve comfort/construction/ergonomics.

If you try to blend a binary system like that with a numeric rating, I think it might confuse some older folks (like me).

Yes this is the major hesitation I have about adopting this system. But I think it can be framed in a way where you see all 5 levels at once so you get a sense that hey, even though its recommended, there’s still 3 levels above it can go.

But is it “outright bad to very mediocre”? I don’t think so.

This is a fair point. But on the other hand, I think it is a bit of a cop out as a reviewer to constantly say “it’s all subjective, you might find it great!” I think at some level you have to put a stake in the sand. Perhaps it should change to “Bad to very mediocre; not for me”.

In my (pretty extensive) professional experience with rating systems, it’s tough to maintain personal consistency or inter-rater agreement with anything more than 3 categories.

No. Maybe. Yes.

Yup I agree. That’s why in all my written reviews, I end the conclusion with a “Should You Buy It?” and it’s always No. Maybe. or Yes. Then I spend the next little while trying to justify it. Maybe I just stick to that.

3 Likes
10

Appreciate your feedback on some of our suggestions, Fc.