More Updates To The Steam Customer Review System

[Brick Tamland]

Over the past few months, we've been reading your feedback and reports on the Steam Customer Review System. In addition, we've been looking at the different ways customer reviews are being used on Steam and evaluating which aspects of the feature need the most improvement. In May of this year we made changes to highlight the recent reviews on games to better show the current state of quickly evolving products (read more about those changes here). Now we are releasing the next update, which adds more filtering and sorting options for the displayed reviews, and sets some new defaults to highlight the recent, helpful, and relevant reviews.

 

 

 

New Filtering Options & Defaults

We know that you have your own preferences in what information is important when you consider making a purchase. So today's update adds some options at the top of each games review section where you can filter the reviews you see by language, purchase source, and whether the review is positive or negative. This will let you dig in to different aspects of a game's reviews to see what other people have to say about the game and a summary of how positive those reviews are.

 

With the introduction of these new tools, we're setting the default filters to provide the most useful snapshot of a product's reviews for you. By default, we'll show you a summary of helpful recent reviews written by Steam customers in languages you speak. (If you speak more than one language, you can configure your preferences here.)

 

We are also changing the default review score that we show at the top of each product page (and in search results) to not include reviews written by users that obtained the product through a Steam key. Here's why:

 

 

The Review Score

When we introduced the Steam Customer Review System in November 26, 2013, our primary goal was, and still is, to help customers make an informed decision when considering the purchase of a new game. To achieve that goal, we've put an emphasis on written reviews that encourage customers to share their experience in a game so that other's can decide for themselves whether the game sounds like something they would enjoy playing.

 

As the number of reviews on any particular product grew, it became difficult to get a sense of whether customers were generally happy or unhappy with how well the game met their expectations. To make it easier to tell whether customers overall would recommend purchasing the game, we created a review score. We've intentionally kept this score as transparent as possible, by simply calculating the percentage of positive reviews.

 

We know this review score has become a valuable shortcut for customers to gauge how well the game is matching customer expectations. But the review score has also become a point of fixation for many developers, to the point where some developers are willing to employ deceptive tactics to generate a more positive review score.

 

The majority of review score manipulation we're seeing by developers is through the process of giving out Steam keys to their game, which are then used to generate positive reviews. Some developers organize their own system using Steam keys on alternate accounts. Some organizations even offer paid services to write positive reviews.

 

 

How Steam Keys Impact Review Score

Steam keys have always been free for developers to give out or sell through other online or retail stores. That isn't changing. However, it is too easy for these keys to end up being used in ways that artificially inflate review scores.

 

An analysis of games across Steam shows that at least 160 titles have a substantially greater percentage of positive reviews by users that activated the product with a cd key, compared to customers that purchased the game directly on Steam. There are, of course, legitimate reasons why this could be true for a game: Some games have strong audiences off Steam, and some games have passionate early adopters or Kickstarter backers that are much more invested in the game.

 

But in many cases, the abuse is clear and obvious, such as duplicated and/or generated reviews in large batches, or reviews from accounts linked to the developer. In those cases, we've now taken action by banning the false reviews and will be ending business relationships with developers that continue violating our rules.

 

While helpful users in the community have been valuable in reporting instances of abuse, it's becoming increasingly difficult to detect when this is happening, which reviews from Steam Keys are legitimate, and which are artificially influenced.

 

 

Changes To The Review Score

As a result of this, we are making some changes to how review scores are calculated. As of today, the recent and overall review scores we show at the top of a product page will no longer include reviews written by customers that activated the game through a Steam product key.

 

Customers that received the game from a source outside of Steam (e.g. via a giveaway site, purchased from another digital or retail store, or received for testing purposes from the developer) will still be able to write a review of the game on Steam to share their experience. These reviews will still be visible on the store page, but they will no longer contribute to the score.

 

This does mean that the review score category shown for about 14% of games will change; some up and some down. Most changes in the review score category are a result of games being on the edge of review score cut-offs such as 69% positive or 70% positive. A change of 1% in these cases can mean the difference between a review score category of "Mixed" and "Positive". About 200 titles that only had one or two reviews will no longer have a score at all until a review is written by a customer that purchased that item via Steam. In all of these cases, the written reviews still exist and can easily be found in the review section on that store page.

 

 

Next Steps

We are aware that these changes do not address all of the feedback and suggestions presented by members of the Steam community. We are working to address these other issues, which mainly pertain to the helpfulness of reviews:

There are some titles where the most helpful reviews don't seem to accurately match the general customer sentiment. For example, there are a couple of prominent titles that have review scores of 'positive' but all the reviews marked as helpful are negative. We need to look at this to figure out how to represent cases where the community has highly divergent opinions.

There are some titles where a small group of users are able to consistently mark specific reviews as helpful, and as a result can present a skewed perception of what customers are saying about the game. This is obviously not ideal, so we're looking at ways to ensure that a few users don't have outsized influence over the system.

Some off-topic reviews get marked as 'helpful' simply because they are funny. These don't appear to actually be helpful in determining whether you should buy the game, so we're working on some ways to better detect and filter out these.

 

We know that Steam customer reviews can only be valuable in aiding you as long as you can trust the data we're presenting. The changes made today target the main abuse we're seeing, and give you more control over the information you see when evaluating a game.

 

As always, please let us know what you think.

 

View the full article