Best implementation of Product Reviews?

Yeah I think some of this is just semantics as terminology for these things varies among people. I think no matter what there is going to be opinion and subjective thoughts. Look at Tyll’s reviews. That is how I try and format my own reviews. I like to evaluate build quality and comfort first which will inevitably be a mix of objective observations like materials used, weight, quality of parts, etc as well as subjectively how it feels and fits. For sound, I feel a full review would evaluate the frequency spectrum (measurements may or may not be needed). But it helps to at least talk about quantity and quality of the spectrum heard and how they fit in together. Again, some of this is objective here like quantity of mid-bass as that can be measured. Or notable treble spikes are objective things that can be analyzed. But that will also involve subjective opinions in how you feel that level of bass sounds as a whole in the entire spectrum or whether those treble spikes make the headphone harsh or sibilant. It doesn’t have to be super technical, but there should be some critical thought into understanding why the sound signature sounds “good” or “bad” to you. That really might be what distinguishes a “review” vs an “impression” to me. For me, impressions are more like bullet points of thoughts. You just say stuff like “this headphone has a really nice bass response that works well with electronic music”. Those types of comments are great for getting a feel of a headphone and whether or not to even consider it. But a full review helps me actually understand the details of the headphone and what it is doing such that it sounds a certain way. This can even be applied in the physical attributes as well. And impression would be like “headphone felt heavy and it started to get uncomfortable after 10 minutes”. But a full review would be more critical in thought like “headphone weighs 475 grams so it is on the heavier side of the spectrum, and the headband has a very rigid shape with pleather pad material that doesn’t compress such that it clamps too tight and I get a hotspot on the top of my head that gets uncomfortable after 10 minutes.”

Basically, I feel that writing a full review takes a significant amount of time/work to put together. When getting ready to write a review I often spend hours listening to the headphone, staring at its FR graph, trying different genres, and really trying to understand critically what I am hearing. And then I will spend even more time volume matching against other headphones and doing A/B switching back and forth. I also try it on different source gear and note how things change. It just takes a lot of time to actually fully understand a headphone and quite a bit of work to write out all those thoughts into a full article. Those types of reviews I feel deserve a separate space. Impressions that are those quick notes you can make without having to spend that much time really thinking about. So those I feel work great in the official threads. Hopefully that helps clarify my distinction.

4 Likes

Great thread. I’m hearing the importance of all aspects of a review and or impression both having contributing factors. In all the years I’ve been listening to music (50+) I have never been able to understand the scientific aspects of equipment . At this point I realize that aspect is not important to me.
When it comes to a-b ing equipment with other pieces it is a joy to hear differences and what has synergy to me and share with those who want to read it. I love to report speed, darkness properties of bass, mids and highs soundstage, instrument separation and much more. Talking about air, taunt bass and sparkle, comfort, rolling tubes, and most important to me “Source” of my material. I like knowing what equipment I have will bring me closest to artists performances and not mastering engineers (all too often) poor creative licensing .
What I’m loving thus far in our new community is a sense of no right or wrong or judgements of anyone’s posts or topics. If kept that way we will all benefit from each others experiences and further our collective knowledge.

Taco I would have responded sooner but I hit my daily limit of posts yesterday lol. But I agree with all your points on the distinctions between review and imppression. I also think people can put just as much effort into a impressions review, though I know that doesn’t happen that often. I look forward to giving my impressions on some of my headphones in the near future, busy weekend heading up to Vancouver B.C. I hear they have a headphone bar, I might have to see about convincing the wife to go there with the kiddo.

1 Like

Awesome man, I agree that people can put just as much effort into impressions too, I am looking forward to seeing what you come up with. And I have also heard the headphone bar is an awesome joint worth checking out.

@andrew and I live in Vancouver, if you want to listen to the LCD-3 with the SPL stack!

2 Likes

I used to review using a points system.

If you do that, that’s about all anyone bothers to look at.

A proper review should get into some objective data (doesn’t need to regurgitate everything available on the product’s website), attempt to evaluate it objectively, and also throw in some personal impressions.

I’d avoid a points system like the plague. Not only is it all anyone will pay attention to, it’s impossible to manage long term (or even short term if you have more than one reviewer involved).

What’s an 8.5/10 DAC for me today? I dunno, maybe the RME ADI-2 DAC or a Schiit Gungnir Multibit. An 8.5/10 for me a few years ago? Audio-GD NFB-2.32. Times change.

1 Like

This is the truth! Especially with ever evolving technology! You can have classics but with new technology those classics become stale (not always but generally when talking about technology).

1 Like

Oh man, I will ask the wife, we are only going to be there for the day. Granville Island and china town are priorities for her and the lil monste…the lil kiddo…I might convince her to let me stop by the Headphone Bar. What do think of doing a headphone meet-up, people could bring their gear and spend a couple hours listening, I would make the trip up for sure with with my gear. Could make it an annual thing lol be the Penny Arcade of headphone meet ups lol.

1 Like

Damn. That’s sounds good. Probably should try to do this by location.

Yeah I’ve been wanting to do one for awhile…but didn’t really feel comfortable with the Reddit community lol. I’m in the Seattle area (Snoqualmie/Issaquah). I would be willing to drive up to Vancouver or relative distance to that from Seattle (for the Seattle location).

1 Like

We’re like a 10 min drive from granville island. Headphonebar is about 20 mins from the island. We are doing the Grouse Grind pretty early in the morning tomorrow though so we won’t be around until the afternoon.

We are actually looking at doing multiple meet-ups in both Canada and the US in the future. Its something we’ve discussed at length for a couple of years and looks like something that could be realized this year!

1 Like

Gotcha, have fun with nature’s stairmaster! If I can’t make it work this trip, it’s only a 3hr drive I could always make another trip up. I will definitely make it for a Meetup! Any good hole in the wall must eat restraunt recommendations?

I have used the points system in the past. For me, it helps work through a long list and sort on various features that are most important to my wants.

I agree that it would be nice to have an easy way to aggregate impressions about specific aspects of a headphone. Something I wouldn’t want to do is to have to comb through hundreds of pages of discussion to learn about specific aspects of a headphone’s performance, as I have to do on Head-Fi’s impression threads. That said, except maybe when talking about a specific known reviewer, I don’t think points are meaningful because everyone has different preferences and, frankly, levels of generosity. Yelp reviews are a great example of this. It’s not uncommon for me to see 2 or 3 star restaurant reviews only to read in the comments that the reviewer liked the ambiance, got good service, enjoyed the food and didn’t pay too much!

To me, the key to making a collection of impressions and reviews truly useful is to treat the archive as structured data rather than just a collection of free text. Concretely, that means that an impression/review can and should still include unstructured text, but should also include fine-grained impressions on very specific aspects of a headphone. I would suggest a category/sub-category taxonomy with categories and subcategories such as:

  • Build Quality
    • Finish
    • Headband
    • Yokes
    • Connectors
  • Comfort
    • Headband
    • Clamp
    • Pads
    • Weight
  • Cable
    • Weight
    • Tangling
    • Plug
  • Other Accessories
    • Case
  • Sound
    • Clarity
    • Timbre
    • Bass
    • Midrange
    • Treble
    • Instrument Separation
    • Soundstage
  • Source Pairings
    • Schiit Magni 3
    • Feliks Elise
  • Good for Genres
    • EDM
    • Classical
    • Folk
    • Rock
  • Not Good For Genres
    • EDM
    • Classical
    • Folk
    • Rock
  • Specific Song Impressions
    • Yuja Wang Rachmaninov 3rd Piano Concerto
    • Massive Attack Angel
  • Comparisons
    • DT 1990
    • LCD2C
    • … other specific models
  • Measurements
    • Frequency Response
    • Impulse Response
    • THD

Subcategories should be easy to expend (e.g. adding new amps). If reviewers have impressions about a category that don’t fit neatly in a give subcategory, they could just leave them at the category level. Each of these impressions would be subjective text and an optional numeric rating (since some people like myself may not feel comfortable putting numbers on stuff).

As an oversimplified (and slightly crappy) example, for my HD 600, I might provide the following:

  • Build Quality - Not fancy but functional and durable
    • Finish - Durable but not particularly attractive. The faux marbling isn’t as off-putting in person as it is in pictures, but I would have preferred a solid color scheme like on the HD650 or HD6xx.
    • Headband - Combination of metal and plastic. Don’t bend it too far, as I’ve heard it can break.
    • Yokes - I expect these will last a long time
    • Connectors - The cable is removable. The connectors are a custom thing, but there’s plenty of aftermarket cables available with these connectors. They are a little tight to remove and insert, but that keeps them from falling out.
  • Comfort - Overall very comfortable, I could almost forget I’m wearing them
    • Headband - Very nice, no hot spots
    • Clamp - A little tight, but loosens up over time
    • Pads - Soft, breathable and very comfy
    • Weight - Light
  • Cable - There’s only one included cable and it’s too long for use on the go or even at my desk. I bought a cheap aftermarket cable on Amazon (they’re plentifully available from companies like NewFantasia).
    • Weight - Heavy
    • Tangling - Probably not too bad given its length
    • Plug - Plug has a built-in adapter but it’s odd, the 1/4" slides completely over the 1/8". It took me a while to figure out that there was even a 1/8" in there!
  • Other Accessories - Not well accessorized.
    • Case - nope
  • Sound - Overall pretty neutral though with elevated mids, a little grainy to my ear and not the best bass performance
    • Clarity - Pretty good, although I often hear some sort of hash/grain that makes these sound a bit hazy compared to my DT 1990
    • Timbre - Very natural
    • Bass - A good bit of mid-bass weight and punch, not great extension. The bass distortion makes itself noticeable in that the clarity in the bass is not great (i.e. sounds a bit muddy).
    • Midrange - The highlight of these headphones. Somewhat elevated, but not unnaturally so. Really shines on voices, guitar and piano.
    • Treble - Pretty good, though it has a little more low treble than I’d like (can get slightly fatiguing) and less higher treble than I would like (lacks some sparkle and air).
    • Instrument Separation - Not the best. Complex passages of orchestral music can sound like a wall of sound, with individual instruments difficult to pick out.
    • Soundstage - I’m told it’s narrow, but I almost always use crossfeed so it’s of little interest to me.
  • Source Pairings
    • Schiit Magni 3 - Sounds fine.
    • LG V20 - Sounds fine.
    • Macbook Air - Sounds fine.
    • Feliks Elise - (I’m making this up because I haven’t heard it). Greater bass weight though slightly muddier, less fatiguing treble
  • Good for Genres
    • Classical - The very natural timbre is great on stuff like Segovia’s guitar works, but I don’t love the congested sound on Beethoven’s symphonies.
    • Folk - Excellent. Voices and guitar are front-and-center where they should be.
    • Rock - Generally pretty good, though the faster more technical stuff isn’t so hot.
  • Not Good for Genres
    • EDM - Not so great. Tried listening to Massive Attack’s Angel with this, didn’t do much for me.
  • Comparisons
    • DT 1990 - With their treble EQ’ed down, I vastly prefer the DT 1990 for it’s greater overall clarity, sparklier treble, and much more precise, better extended and punchier bass. The HD 600 pulls a head a little bit on vocal-focused music like folk rock, but for DT 1990 doesn’t fall that
    • LCD2C - The LCD2C’s bass extension and clarity blow the HD 600 out of the water. The LCD2C also exhibits less of the grain and haze that I hear in the HD 600. However, the HD 600’s overall tonal balance sounds more natural and less dark.

The benefit to capturing this stuff in a database is that it opens up the possibility for all sorts of applications.

  • Build Your Own Review - Select which aspects you care about and then see reviews include only those
  • Focused Analysis - For example, see all impressions of the HD 600 with classical, or see all classical impressions for all headphones.
  • Review the Reviewer - Look at all of a reviewer’s impressions of “bass” or “classical” to see how they tend to evaluate this and put their review in context.

Building this database wouldn’t be easy, would require a fair amount of work on getting the taxonomy right and probably require some iteration before it’s just right, but I think it would become a tremendously valuable resource. Especially with the ability to attach crowd-sourced measurements, I’m excited about what this format would bring.

7 Likes

Very nice post. There are some ways to handle this with a bit less structure, but the trade-offs are either

  1. Money or
  2. Time and expertise.

If you have money to throw at it, call in the experts:
https://www.marklogic.com/

If you have time and expertise, go open source:
http://www.exist-db.org/exist/apps/homepage/index.html

I was a content data professional, now 4 or 5 years into recovery, having not kept up with the industry, done xml markup, built DTDs or Schema, or talked with professionals about structured data editorial systems in that time.

I’m going to stop now, because I don’t want to descend back into the depths of driving around, thinking about extensions to content management systems.

Andrew - some practical observations.

  • unless you pay them, writers are like cats. Difficult to herd.
  • even if you pay them, writers are like cats. Difficult to herd.
  • it is possible to use two different databases - your forum here, and a second one that monitors it and builds the proper indices.
  • Expect to spend a minimum of $30,000K and 500 man hours, or $80,000K and 200 man hours, depending on your solution. Even the open source will cost you your time and money to learn it.
    Results, however, can be worth it. The links I posted above to solutions are generally considered top contenders. There are plenty of other players, but they often either try to herd cats (did I mention that was hard?) or use components of the above.

@pwjazz note my clever use of a numbered list in one spot and a bullet list in another, resulting in a style error. :wink:

2 Likes

I like the idea you’re presenting here, in terms of what it would enable.

I can also say, as someone that semi-regularly posts reviews of gear, some of them pretty extensive, that there is no way I’d fill all that out in the form necessary for it to work as structured data. And even if I did, absent a big guide, or additional context (both of which entail additional work on the part of the reviewer) you’ll still get subjective and/or incomplete responses.

For example, unless you have a range of values specified then things like “Cable: Weight” will be useless as what one person considers heavy won’t gel with another person’s opinion. And if you think I’m going to pull out a scale and measure that for every headphone I want to post a review or impressions of … well, let’s just say it’d result in me never actually getting around to making the post at all.

My impressions or reviews might include such things anyway, and I agree it’d be great to have them processable, but having seen how this sort of things tends to go when those having to do the work are faced with that much structure … my expectation would be that you’d get great data on very few reviews. Per @pennstac’s comment, it’s hard to get people to observe that much rigor even when you’re paying them.

And then the degree of time/effort I, personally, am willing to put into a post on impressions or a review is highly dependent on what item we’re talking about. I’m going to say and provide a lot more information about a $4,000 set of headphones than I am a $100 set.

Having a “Headphone Database” with such specific information in, that then links to the more common narrative reviews might service both goals. I think having a list of information that is good to include, with examples, such as that you provided, would also be useful. But I do believe making that a necessary part of posting impressions or a review will simply result in people not bothering and just posting their thoughts in the product threads anyway.


I’d add that, perhaps for gear made available via the Community Preview Program, you might be able to get people to fill out some of that … but I’m hesitant in thinking you’d get everyone to do it all.

1 Like

I agree 100% that entering impressions and reviews should be easy and enjoyable, but I think that’s solvable with good UI design. Certainly none of the detailed categories should be required, like I think the following would be a legit review…

Build - utilitarian and solid

Comfort - lightweight and comfortable with an initially strong clamp that eases over time.

Sound - amazing mid-range, okay clarity, slightly bright with a bit of a haze and slightly muddy bass.

Source Pairing - Sounds fine out of my laptop and smoother and bassier from my OTL tube amp.

Genres - I like these with folk, rock and small ensemble classical

Comparisons - I like the DT1990 and LCD2C better because of their better bass performance and instrument separation.

One thing that this all reminds me of is a survey. In fact, doing periodic semi-structured headphone surveys could be a valuable way to collect impressions.

Lastly, as a reviewer myself I actually think having some structure helps remind me to look for certain things and actually makes it easier to collect my thoughts.

1 Like

Also, I suspect we could use natural language processing to do feature extraction and sentiment analysis on free text. We’re dealing with an extremely specific domain with a constrained core vocabulary and fairly fixed scope of products.

I’m going to disagree here and say that your example is what I would call “impressions” and I would personally choose to leave those in the main official headphone thread. Going back to what I said earlier in this thread: “It doesn’t have to be super technical, but there should be some critical thought into understanding why the sound signature sounds “good” or “bad” to you. That really might be what distinguishes a “review” vs an “impression” to me. For me, impressions are more like bullet points of thoughts. You just say stuff like “this headphone has a really nice bass response that works well with electronic music”. Those types of comments are great for getting a feel of a headphone and whether or not to even consider it. But a full review helps me actually understand the details of the headphone and what it is doing such that it sounds a certain way.”

Let me break this apart why this falls into an “impresion” vs a “review” for me:

What are the materials that make it feel solid to you? Some people only think metal feels solid and that anything plastic feels cheap. Some people think good, scratch resistant plastic feels better made than metal that dents and is too rigid. Think old school plastic Samsung phones vs metal iPhones here. There are arguments for both sides.

Lightweight compared to what? Lighter than HD650 or lighter compared to LCD-3? Does the initially strong clamp make it painful or is it tight but snug and gives a nice seal (e.g. LCD2C)?

Why is the mid-range “amazing”? Is it because of the timbre? It is because it is on the warm side? Is it because vocals are very forward?

There is nothing wrong with your example and I am not trying to pick on you. That is exactly what I expect “impressions” to be. Those are the types of thoughts one could get from a headphone from listening to them for 10 minutes at meet. And those are the types of thoughts I would enjoy seeing in the official headphone threads. I think of it this way. If I am going into the official thread for a headphone I am curious about and haven’t heard myself, I want to see the general opinion on it and see what it is generally about. Those impressions you listed are perfect. That way I could quickly tell if it fits with what I am looking for. If I want a light, mid-focused headphone that would do well with acoustic music it sounds like from your description this is a headphone worth investigating more and doing some research on. And because I know you, I am assuming you were describing HD600, which all of your impressions sound dead on with how I hear it.

So from there, that is when I would start going to the separate “review” repository for HD600 and start digging in to learn more about the finer details and get a better sense of if this headphone really would be the best purchase for me. As an example, I posted the following Campfire Audio Cascade review in the official thread for now until a separate “review” repository is made. I only had the Cascade for about 6 days so maybe 15-20 hours of head time, and this is sort of the minimum of what I would personally write for a full review; my other reviews might be 2 times as long. But this is the type of detail I would expect to see from a full “review”:

https://forum.headphone.com/t/campfire-audio-cascade-closed-back-headphones-official-thread/59/18?u=i_want_all_the_tacos

I admit this is just my pesonal opinion of how I like to see separations between impressions and reviews, and I have no idea if this makes sense and works for others. If I am totally off base and people have better ideas than I am fully open to those too. But at least for me, that is sort of how I am used to approaching new gear and the methods I have used for learning about gear in this hobby.

Ha yes, my example was just to illustrate that if people want to leave low-effort impressions they should be able to. IMHO the difference between impressions and reviews is more one of degree than of kind. Quoting from a DT1990 review of mine:

As I’d hoped, the DT 1990 does very well on metal like Dream Theater’s “Metropolis, Pt. 1” and Rage Against the Machine’s “Bombtrack”, with a punchy and tight bass that does justice to fast double-bass drum and bass guitar work, a strong lower midrange that gives electric guitars a nice crunch and an elevated treble that brings out that guitar edginess that’s so fundamental to metal and which I miss with the HD 600 and LCD2C. The neutral midrange balance also allows vocalists La Brie and De La Rocha to sound clear, natural and authentic while occupying a balanced place in the mix. Lastly, the DT 1990’s dynamic and crisp presentation of percussion, plus a little sizzle, does justice to fine drummers like Mike Portnoy.

I could just as well have said “they’ve got very nice bass, neutral mid-range And sparkly treble.” This would be consistent with my review, just less well elaborated.

Yeah it seems are on the same page here. I’m fine with low effort impressions for the official thread. That’s the place for questions, answers, impressions, general comments, etc. And then the separate review repository would be reserved for lengthier, in-depth reviews like your DT1990 example.