Necessary Properties of High-Quality Recommendations in E-commerce

Retail Rocket company has been dealing with recommendation systems in e-commerce for more than 10 years. During all this time we have encountered non-obvious properties of recommendation algorithms, without which the systems will not solve their tasks. We have identified and described in this article such properties as: efficiency, coherence, coverage, context awareness, scope, relevance, and geo-adaptability.If you have a recommendation system and you want to evaluate its quality, or you are looking for criteria for selecting or developing a recommendation system, this article will be useful to you.

A bit of introductory information

Everyone has encountered the work of online store recommendation systems. Recommendations are successfully used by both small companies and industry leaders. According to a Mckinsey report, Amazon generates 35% of its revenue through recommendation systems. There are detailed studies on the business value of recommendations, such as the article “Measuring the Business Value of Recommender Systems”. But the best evidence of their value is their widespread use – it’s hard to find an online store that doesn’t use this tool.

Recommender systems must take into account the problems that arise in the real world. Only then will marketing be able to use such systems to achieve its goals. This can be called the main quality criterion of any marketing tool, in particular, recommendation systems.

We at Retail Rocket have been providing recommendations to the market for more than 10 years, to more than 1000 clients. And all this time we have been researching and improving our system. Back in 2016, we gave a presentation on the hypothesis testing process at the “Recsys” conference, which is still the largest conference dedicated to recommendations. Since then, we have accumulated a lot of experience and expertise. In this article we will talk about the complexities that are hidden in recommendation systems and are not visible at first glance.

This is the first article in the series, where we will consider only recommendation algorithms, without touching upon many complex technical issues, such as data storage, hardware support, monitoring, integration of recommendations into an online store, and much more.

What properties should recommendation algorithms have in the conditions of a real online store?

Making a tool that shows products on a store’s website seems to be a simple task, but as soon as a recommendation system appears, there is a need to solve many different problems. This is similar to the development cycle of any software system – full fulfillment of all requirements and support requires much more resources than the initial development of the basic functionality. Moreover, recommendation systems have their own specifics because of the complexity of the tasks to be solved and the need for marketing to do its job well. 

We have formulated seven properties that are necessary for a recommendation system in a real online store.

Task-Oriented Effectiveness – recommendations should be fully aligned with the buyer’s need

Recommendations have to be designed in a way it helps online retailers’
buyer effectively accomplish their tasks and find relevant items.

Examples of such tasks include:

• familiarizing themselves with the store’s assortment,
• finding the right product, considering alternatives,
• searching for complementary products.

As a result, it ensures that:

• making a purchase decision and placing an order,
• maintaining contact with the customer after the order is made,
• buyer return to the store, regular orders and loyalty.

By the appearance of recommendations you can understand their Task-Oriented Effectiveness in solving the buyer’s tasks. For example, a show of similar products does not solve the task of familiarization with the range of products. And if all previously purchased goods are removed from the recommendations, the algorithms do not solve the buyer’s tasks in relation to the goods of regular demand: when buying pet food, the recommendations will not remind about the toilet filler, and when buying fish, they will not offer the previously purchased suitable spices.

If a buyer is comfortable using an online store, they are more likely to make a purchase and come back for the next one. Improved user experience makes the customer more loyal to the store. All this leads to an increase in LTV – Lifetime value – a measure of the profit a business makes over the entire time it works with a customer. 

It is extremely difficult to quantify the impact of individual marketing tools on LTV, given the length of a customer’s relationship with a store. Therefore, multiple metrics have to be used to increase the likelihood of detecting an impact on LTV. We covered the application of proxy metrics for E-commerce in a separate series of articles.

Logical Clarity – it should be clear what task the algorithm helps the buyer with.

The buyer should have clear picture and understanding on why the widget on the website offers specific products. If recommendations help to find products that complement the order, then it should contain not alternative or the most popular items, but exactly products that complement the order. 

To understand the purpose of the widget, it is important to choose the right header. For example, if the widget is designed to show complementary  products, the headline “This product is bought with this product” instead of the uninformative “We recommend you” would be much more relevant.

Logical Clarity makes the work of recommendations more transparent, predictable. Thanks to this, the buyer will trust the recommendations and the store as a whole. A proper understanding of how the recommendation block works helps the buyer understand how to interact with it to solve his tasks. On the other hand, if the buyer repeatedly encounters illogical behavior of recommendations, he may not only stop noticing recommendations over time, but also lose trust in other tools of the site and the online store itself.

Logical Clarity allows shoppers to distinguish different recommendation tools from each other. This helps to place more than one widget on the page, show more products, increase the impact of recommendations and diversify the interaction with the store. A customer may have a negative attitude to multiple recommendation widgets without clear goals on one page, but will perceive them as useful tools, for example, if one widget helps him find the product he came for, and another shows products that complement the current one.

Coverage – recommendations should be provided for each product

A recommendation system should cover recommendations for the whole product base  in the store. Good coverage makes the store more predictable, does not disrupt the usual user experience and increases the impact of recommendations.

It’s challenging to achieve sufficient coverage level when system provides recommendations only for the most popular products or do not provide recommendations for the new products. It is extremely difficult to get complete coverage when store managers have to manually match connections between categories and products.

If there are no or insufficient recommendations for some products, buyers who are used to recommendations will be forced to find and use other, even if less convenient, tools. It is harder to use several tools than one, so buyers may abandon recommendations.

Context Awareness – recommendations should always take into account the contexts of the page, product, and user

The system should offer different recommendations depending on the current context. The context can be a page type, a product category or brand, or an audience segment that the user relates to.

It is necessary to take into account the context as accurately as possible. In particular, use personalization, not only on the main page of the online store. If the buyer is on the category page, it is important to show the most interesting products of this category. If the buyer is in the product card, it is better to show products with attributes that he prefers. For example, it is better to show products with the appropriate sizes.

Reach – recommendations should cover the entire assortment range of the online store.

The system should be able to recommend as many different products of the store as possible. This quality is important for e-commerce, as online retailers commonly deal with a very wide range of products. They also face the “long tail” effect – when most of the goods may have quite low demand separately, but still make together a significant part of the revenue.

To fulfill product reach, it is also necessary to support the contextual awareness attribute – if there is a need to show as many various products as possible, it’s possible to recommend different items depending on the context. But context awareness does not necessarily mean product scope and vice versa. 

It is important for the store to familiarize the customer with its assortment. Even if he has not bought a product, but knows about its availability in the store, when this product is needed, even after a considerable time, the customer can return to the store and buy it.

We touched on the benefits of assortment familiarity in our research, where we found a positive impact of the average number of product cards viewed by a shopper on LTV.

Actuality should take into account regular changes in the product base and buyers behavior shifts over time

Buyer behavior and the store’s product range constantly change. Recommendation mechanisms should properly react and adapt to these changes and be frequently updated. Ideally – in the real-time mode.

If a new item has been added to the store’s product base, it should soon start to be count in recommendations. When the product is no longer available, it should stop being offered  to buyers as soon as possible.

Recommendations should also take into account the demand variations for the same items at different time periods. For instance, barbeque goods are actively purchased during summer season, while ski equipment is actively being purchased during winter. An outdated smartphone may become less relevant when a new model is released. It’s important to note that demand variations are not taken into account if the recommendation algorithm relies on product characteristics only and does not take into account customer behavior changes.

For buyers, having items in recommendations that they can’t buy leads to violated expectations of the tool. This can lead to a loss of trust in both recommendations and any other store tool. Showing irrelevant products leads to a similar effect.

It is difficult to fulfill the relevance property in practice by manually managing recommendations. This property needs to be maintained continuously, which will require a lot of human effort.

Multiregional Support takes into account the stock availability of products and alignment of their attributes with the specific location

Availability, price and other parameters of online store goods may vary depending on the region and retail point location. There are retailers with tons of huge warehouses and these attributes may significantly vary for the same products available in different regions. Each Geo may have own recommendations logic due to different customer behavior and different product attributes, like availability. 

The recommendation system has to properly take into account these attributes depending on the region of the customer.

In case of showing incorrect price, the problem of discrepancy between the price of the product and the inscription on the price tag arises. The actuality is also violated in case of availability of a product that is unavailable in the buyer’s region.

Joint provision of multiple properties

Providing a single property of a recommender system is not so difficult. The problems begin when trying to provision multiple attributes simultaneously – the need to provide each subsequent property may complicate the implementation of the other properties and, consequently, the system itself. If the recommendation algorithm helps to complement the purchase and uses only ordering information for items that are being often purchased together, then it will be logical clarity, task-oriented effectiveness, and context-awareness.

But to provide comprehensive coverage, you need to show recommendations for new and not very popular products for which there is not enough stats. To fulfill the product reach property, it is necessary to show in recommendations not only popular products. 

Therefore, we have to look for more complex approach.
On the other hand, you can solve the problem of coverage and scope by showing randomly selected products. But then recommendations lose the properties of logical clarity, task-oriented effectiveness, and context-awareness.

If the same items are often presented in recommendations, even if they relate to very popular and widely bought products, system can achieve full coverage. For grocery products, such categories can be bananas, eggs, water, vegetables. For children’s goods, diapers and baby food. For some algorithms, showing such a small group of universal goods allows to achieve high efficiency. However, the spicifisity and product reach are not truly fulfilled.

In algorithms that use user behavior data, the relevance property can be achieved by using history over a small period of time. For example, only the last month can be used to calculate popular items, but in this case it is difficult to ensure the comprehensive coverage and logical clarity

Working together such properties as multiregional support, coverage and logical clarity is also a difficult task. There are stores with multiple warehouses, where for each warehouse it is necessary to provide suitable recommendations for each product, taking into account the product attributes for the corresponding warehouse. The number of different recommendation options in such a case reaches the product of the number of warehouses by the number of products in the store. This requires a large investment in data handling, algorithm efficiency and infrastructure. Further complicating matters is the need to fulfill the actuality. The merchandise base changes very frequently and recommendations need to be constantly calculated and updated. To fulfill the specificity property for personalized recommendations, they need to be updated in real time because the context of the current user changes with each action.

The examples above show that organizing the joint implementation of even two properties is a non-trivial task. And what may be unnoticeable or seem unimportant at the beginning of the recommendation system may later become an insurmountable obstacle. In the next article, we will describe how to ensure joint execution of properties using the example of the related products algorithm.

How to check the quality of a recommendation system

We propose a number of simple checks for violation of recommendation properties:

• Efficiency should be evaluated in relation to the tasks that users perform on the site. For example, the task “familiarization with the assortment” is not fulfilled if the widget with popular products shows only a group of similar products, which does not give an idea of the entire assortment of the store.
• Coherence means that you can visit the store pages and understand what user task each recommendation widget solves. For example, a widget in the product card can help to complete an order with the current product or find a more suitable alternative. The products in the recommendations should correspond to the task the user is solving. If the recommendations complement the order, you can imagine users coming to the store to buy items from the widget together with the current product.
• Coverage is not provided if there are no recommendations for new items. These products can often be found in a block on the homepage or by sorting the category page by new products.
• Context Awareness is not provided if the same recommendations are provided for different products, even similar ones. As a rule, it is enough to view recommendations for several products from the same category.
• Products scope  will not be ensured if only popular products are used in the store recommendations. Less popular but relevant products are also worthy of consideration and can account for a significant portion of revenue. Looking at recommendations for popular items in the same category usually allows you to quickly assess coverage.
• Relevance support can be checked by changing the region and seeing if the recommendations show products available only in that region and that their prices correspond to the prices in that region.

Conclusion

A recommendation system is used to fulfill marketing tasks and is not obviously related to many aspects of the functioning of an online store. For this purpose, the recommendation system must have properties without which it will not be usable in practice. The need to provide these properties makes the implementation of a recommendation system a complex task.

In the following article, we will describe the components that allow us to provide the necessary properties of a quality recommendation system, using the related products algorithm as an example