Best implementation of Product Reviews?

42

What I would actually prefer is that impressions and reviews be their own thing and not just a thread. Head-Fi has those long impression threads which quickly fill with irrelevant side discussions and make it difficult for me to find actual impressions about the headphones themselves. I’d like something more like the Head-Fi reviews section, even for quick impressions, with the ability to easily filter through the impressions/reviews, look at them in aggregate, and so on. It would of course also be good to be able to discuss the impressions, but I think that would be better served by separate discussions in response to each posted impression rather than one big thread. And I think we could just tag the full reviews as “full review” so that if I want to just have those, I can filter for it.

Elaborating on the review vs impression thing, it strikes me that a review is really just impressions with context and supporting evidence. If an impression is “I like the bass on the LCD2C but wish it were stronger”, a review is more like “Hello, I’m Bob. I’m basshead who loves my TH-X00. I listen mostly to electronic music, especially jungle and drum and bass. I compared the LCD2C with my TH-X00 listening to X, Y and Z songs, driven from my Magni 3, volume matched at 78 dB, etc… I found that the LCD2C’s bass extension matches the TH-X00, but kick drums lack some of the slam of the TH-X00. I tried X, Y and Z equalizer settings and that improved things, but if you want slamming bass, stick with something like the TH-X00. Etc …”

When I think about how I consume such information, there are a few ways.

  1. Sometimes I just like to read well-written reviews because I enjoy sound and enjoy intelligent people writing about it thoughtfully.
  2. Sometimes I’m actually trying to make a purchasing decision and want to research some specific equipment. I’d like to look at it in aggregate to get a general feel for what people think about the bass performance, comfort, etc.
  3. Having found some general impressions, I might want to dig in some more to learn about some of the people who provided those impressions, what other equipment they own and their general listening preferences, and how they came to their specific conclusions (e.g. what did they compare to, what songs did they listen to, etc.)

Item 1. Is what you would call “full reviews”, 2. is what you would call “impressions” and 3. brings it full circle back to reviews. In an ideal world, this all just becomes different views into the same core information that allow me to pivot, zoom in and out and slice and dice to my heart’s content. If we could achieve that experience for the consumer of the information coupled with the ability for contributors to enjoy writing big reviews as well as be able to give quick impressions here and there as time permits, that would be my personal optimum.

43

That’s exactly what I am referring to when I say “review repository”. Earlier in this thread I specifically used Head-fi’s method as an example in that there is a standard, long running, continuous thread, what we here are calling “official threads”, and then a separate review area where full length reviews are written, and within those people can comment and respond on each individual review, but they are all grouped together in one collective place. That’s what I mean by review repository. So what I am trying to do now is define is what content would belong in the former vs the latter.

Generally for me, the way I approach new gear is I first start in the “official thread” if I know nothing about the gear and usually the very first posts will give a good explanation and sense of if I want to explore it more. Then once it seems interesting to me I will jump specifically to the review area and just read individual full length reviews. And then if I am pretty serious about that product but need some questions answered I will go back to the “official thread” and search to see if my question has already been asked, if not, ask it in there. And then once I actually buy and own that product I will usually continue to keep up in the “official thread” to answer other peoples’ questions or to just pay attention to how others are hearing that gear, how they are pairing with other sources, or mods they are making, etc.

So going back to my original point, those very short, bullet point types of “low effort” impressions that can be had within brief listening session would go in the “official thread”. But the full length reviews that involve critical thinking go into the separate “review repository”.

For me, Item 2 is usually covered in Item 1 if that is indeed a full review. A full review should be pretty comprehensive in talking about build, comfort, sound, source pairings, genres, etc. If you specifically want to figure out the bass response of a headphone the review repository would be perfect since you can read through each individual review and jump to the “bass response” section of it. You wouldn’t have to wade through the noise in the “official thread”. But the “official thread” should be for everything else including questions to specific people or the general audience. I also feel those short, quick impressions work well in there too as often times people give those impressions in response to a question about the gear. It follows a natural flow to give bullet point types of impressions/answers to people discussing specific things within the “official thread”.

Now, if you wanted to dive deeper into a specific full review and ask the specific reviewer more related questions to their review, I like how Head-fi’s system allows those comment chains to build within an individual review. So each individual review does have the ability to have its own related sub-thread, but all the individual reviews are collected together into 1 repository. That is ideally how I would envision the review repository working here.

1 Like
44

Taking a step back for a moment and tabling my proposed solution, I generally agree with what you’re saying but believe that there are three things on which we can improve:

  1. Increase the ratio of structured reviews vs discussion forum chatter. Continuing with Head-Fi as our touchstone, if you look at the Focal Clear, you see 3 reviews vs 3716 messages in the official thread. When looking for review-like information, it would be useful to have a greater diversity of reviewers represented, and I would personally be willing to sacrifice some verbosity in the review in exchange for more reviews. In other words, it would be nice if some of the longer/better impressions buried in the official thread were surfaced as reviews.

  2. Assuming we solve 1, it would be nice to be able to get a longitudinal view of this large set of reviews without having to read each review in depth. The summarized pro/con sections in Head-Fi go some way towards this, but with a bit more structure I think this could become more powerful and useful.

  3. The discussion in the official Head-Fi threads often veers into tangents that make it very time consuming to dig through them for information about the headphone that’s ostensibly the focus of the official thread. I think there are things we can and should do to avoid this happening here.

45

I agree on all your points. And here are some possible solutions:

  1. I believe the reason for the lack of use of the “review” section of a given headphone in Head-fi is because it isn’t clear that there are separate section in Head-fi for reviews. I was browsing and subscribing to dozens of threads on many different products for literally years before I finally realized there were separate review sections. I even posted full reviews myself in the long, ongoing impressions threads because I never knew about the separate review sections. Those review sections just aren’t clearly linked to the impressions thread. I mentioned this at the top of this thread about why Head-fi’s system is good, but why it also has shortcomings. I propose that if we have a separate review repository section that it be directly linked and referenced somehow in the “official threads” such that they are basically tied to each other. And I also think it will take a bit of moderation to move full length reviews to the separate section if they accidentally get posted in the “official thread” area.

  2. The structure issue is essentially what this thread discussion has been about. That is why people have been arguing for or against using a rating system or having templates. I am a bit undecided on this myself as I like having structure, but I also don’t want to force people to have to fill out templates, especially if the thought of doing so discourages potential reviewers from writing reviews. Maybe we can just use a “TL;DR” section at the top of all reviews, which really is the same idea of what “abstract” sections are in scientific journal articles. Something like a quick summary at the top of each review would probably help readers quickly decide if they want to invest a lot of time in diving into the full review or not.

  3. I agree about the tangents, but I actually don’t think the tangents themselves are bad as long as they are on-topic. I think with long running, continuous threads there is just no way to avoid that type of conversation flow. And I actually think it is fun as an owner to just get to know other owners within a thread and talk to each other about how they use that product. But I do agree that Head-fi’s SEARCH FUNCTION is terrible, which is why it gets time consuming trying to find relevant information you are looking for. I think this forum’s built-in layout will actually alleviate many of those problems without even having to do anything extra. For one, try out the search function in this forum, it is pretty solid. It is much more fluid and quick in being able to jump to relevant posts. Second, I believe the way Discourse works is that once a thread accumulates a certain number of posts (can’t remember the exact number) it automatically builds in a summary tool that uses a sorting mechanism to show the most popular posts. A great example of this is the “Songs to test headphones with” thread. Click on that link. In the very first top post it is already showing you “Popular Links” so right away you can just click on links to new playlists that people liked. Next, you will see a button that says “Summarize This Topic”. It now filters out and hides less popular replies. And then if you scroll through the actual posts there are a couple things to look out for. First, on the lower left corner of a post you might see a grayed out “# Replies” label. If you click that it will expand to show who replied to that particular post and what they said. Similarly, if you check the upper right corner of a post and see the “reply arrow username” label it means that post was a direct reply to someone else, and if you expand that it will show you the conversation of what that post was replying to. Those features are really important IMO. Often times I try to do this in Head-fi by skipping the hundreds of pages at the beginning and jumping to the newest posts. But unless a post specifically quotes someone else inside their reply, you lose context of what they were replying to and you have to scroll backward in pages to figure that out. This system here is much better for browsing through long threads to find the most useful information.

46

  1. Agreed that just making reviews more visible is a good first step.

  2. Quick summary is a good start, though I suspect in practice I’ll find myself wanting more structure. An NLP approach for extracting structure from the free text might be the best of both worlds.

  3. Yes, the semi-threaded, infinite-scrolling format of Discourse is a big improvement over Head-Fi’s forum in and of itself, as are the popular links and summarize features and the much better search engine.

Another cool thing about Discourse is that it supports plugins so as we learn more about how it works in practice, we could create a “headphones” plugin that provides some domain-specific functionality to take it the extra mile.

47

It’s late. I probably shouldn’t comment. But I’m going to.

If you want to create a structured data set cheaply, you must feed the authors templates, and the templates must enforce the structures. You want to do a review? Use the template. You want to have your impression kept for posterity, use the template.

Next, you need an editorial component. Machines are not going to substitute. You don’t want them to say “amazing midrange” by itself? The editor has to send it back to the writer for revision. This is something I know. I was Product Manager for an online editorial system designed to build structured content for scientific, technical, and medical clients including McGraw Hill, John Wiley & Sons, Reed Elsevier, Lippincott Williams & Wilkins, and any number of smaller players including professional societies. Please don’t use another headphone blog as your standard of comparison.

The above is why I suggested looking at systems that produce indices that try to impose some structure on less rigorous full text. You are both right in that a headphone blog has certain limited or common content that will make this easier.

But the real problem is the human one. You can get SOME people to follow your structure, but you will lose many authors out of frustration. And there are good reviewers that do subjective reviews. They may be a bit quirky, and probably won’t fit into the proper containers.

Speaking of the editorial component - realize that high-quality content contains elements of both peer review, and copy editing. You need a workflow and a process to move content from creator, through an approval process, and then into an editorial process that includes both content and copy editors.

I really don’t have a dog in this horse race. I don’t think I’ll write much more here, as I don’t have a heck of a lot more to say. If you want my opinion, please ask me, I’ll be reading this, or message me. My background may not be appropriate for what you’re hoping to achieve in this thread, but I’m willing to add my comments if asked.

48

Is there any progress on getting a review implementation up? I am going to try and put up a number of reviews throughout the summer and it would be great to get them put into a new home.