How AI Is Used To Recommend Products To Customers
Written By: Tad Witkowicz firstname.lastname@example.org
What is it and what it does?
There are two categories of AI Recommenders: Collaborative and Content Based. Collaborative systems make suggestions of the type “Other people also liked/bought/viewed ….”. Content Based system make suggestions: “You may also like ….”.
How it works – Collaborative systems.
We see product recommender systems of the type “customers who bought this product also bought.....” when shopping on Amazon or other on-line stores. Their operation is illustrated in the table below.
This table shows a sample of customers C1, C2, C3.... shown as rows and products A, B, C, …. they bought shown in columns. At first glance there isn't much here, just some data.
The information becomes more interesting if we mark customers who bought multiple products. For example C1 bought product A and product D. Both cells are marked dark green in the table below. We also notice that C4 and C8 who bought A also bought D. There are 5 customers who bought product A and 3 of them also bought D. The recommender system will conclude that it is likely that customers C5 and C7 who bought A may also be interested in D and make a recommendation to C5 and C7 which is marked light green.
The system will also notice that customers C2, C9 and C10 who bought product D (Orange cells below) did not buy product A and therefore will make a product A recommendation to C2, C9 and C10 shown as yellow.
This very simple algorithm scans all the purchases and creates the type of relationships described here. The effectiveness of the recommendations depends on collecting data from many purchases. It would not be very good if all we had was the data contained in this table. The more data the better the chances that the relationships are real.
This approach can be easily extended to on-line store visitors who have not bought anything yet by making recommendations of the type “people who looked at this also looked at …..”. You can imagine how this can quickly get more complex. For example, some people looked at A and did not buy it but they did buy D. Perhaps C2, C9 or C10 did that. Others bought A and looked at D but did not buy it. This is where a lot of examples and computers calculating the probabilities of the various combinations becomes handy in selecting recommendations that have the highest probability of success.
How it works – Content Based System.
Content Based recommender addresses the single user experience/choices and deals with situations of the type “If you liked this, you may also like ….”. Netflix movie recommendation and Spotify music recommendation are examples of Content Based recommenders. Netflix learns from your activities watching movies or listening to songs to find and recommend other similar pieces that you may like.
Let's look at how Netflix does it. Their exact methodology is a trade secret but many articles appeared that describe the Netflix AI recommender system at the basic level which helps us understand their approach and Content Based systems in general.
Step one is to determine if you liked the movie you just watched. Their approach is similar to Facebook's (see tutorial on MachineLearning) , Netflix calculates a score which is assigned to the movie. If you watched the movie to the end in one sitting this may carry a score of 10 points. If you watched it to the end but in pieces it may count as 9 points. If you stopped watching it half way it may be 1 point and if you abandoned it within minutes it may carry 0 or even negative points. To this Netflix may add points for your ranking the movie in terms of number of stars if you did that. Together this establishes your liking score for this type of movie.
The next step is to find movies which are similar to the one you liked, preferably find ones that are very similar to the ones you liked the most. Netflix employs about 40 people called “taggers” who watch every movie and every video offered by Netflix and characterize (lablel) it. They use over 100 different parameters to assign to each movie. Such parameters as genre, director name, key actors, location, ending types (happy or tragic), level of violence & sexual content and even such things as use of curse words or whether characters smoked. Once the movie is so characterized, the computer does the rest. It compares the 100 parameters of the movie you liked with the parameters of the other movies in its library. For example, Netflix counts how many parameters of a movie are the same as the movie you liked the most. If all 100 parameters match then that movie would score 100% and show up on the recommendation list or you get an email with the message “we just added a movie you might like”. If less than 100 match which is most likely the movies will be assigned the appropriate match percentages and will be arranged according from the highest to the lowest. Arranged from left to right on your Netflix screen according to best match.
Making a recommendation on a single movie you watched and liked is not going to necessarily be enough to predict which movies you like in general. Netflix needs to average your watching preferences by factoring the movies you watched and liked the most and those your liked some time and create a profile of a “typical movie “ you normally like. A “typical movie” will contain the values for each of the 100 parameter so it can be compared with movies in the Netflix library to find those that are most similar to your “typical movie”. As you continue watching the various movies and shows Netflix continues to monitor you and adjust their knowledge of your preferences.
There is a problem however. A viewer will watch a variety of movies over a period of time. The movies and shows may range from action movies to comedies to documentaries to family shows. If Netflix continues to adjust your “typical movie” profiles they are likely to end up with an average profile which is neither a comedy nor a drama nor a documentary. It will be an average “typical movie”. Average town, average art, average wine, average sport event and even average people can be dull. It is very likely that an average “typical movie” is also boring and contrary to Netflix goal which is to keep you using their service because they have the most interesting movies.
Netflix solves this by creating movie categories and creating your “typical movie” profile for each category. This averages your viewing history over a much narrower range of movie types you watched and makes recommendations more accurate.
Netflix Recommender screen. Rows represent categries with the highest recommendation at the top. Images within a row represent individual shows with the highest recommendation at left.
While you may see 40 -50 categories on your Netflix screen they keep track of 27,000 (!) categories. As a result every Netflix user is shown a set of categories and movie recommendations within each category that is unique to him/her. Given that Netflix has about 150 Million users (subscriptions actually) World-wide and that each contains 2 – 3 individuals this means that Netflix maintains 300 – 450 Million profiles each with 50 categories means there are close to 2 Billion recommendation lists each with 50 – 100 movies/shows. This type of recommender system could not be possible without Machine Learning (AI).
Examples of Recommender systems from simplest to most advanced
Recommender systems can be used in a variety of ways ranging from recommending products, content such as news stories to services or travel locations. The recommendations algorithms vary in complexity. Here is a sample list starting with the simplest first:
- Product of the day. Minimal or no algorithm is needed, just a decision by a person.
- Latest products, Special deals. Same as above.
- Trending - Based on calculating daily statistics and picking the products whose popularity is growing.
- Best sellers.
- Customers who bought/viewed …. also bought/viewed …
- Complementary - A blouse goes with these skirts, shoes, earings, etc... This can be complicated given that customers may not want to buy the same combination as other customers. Here customer purchase history can be helpful if you have it.
- Accessories. Need to create connections of accessories for each product/service for example: shoes + shoe polish + socks + Belt.
- Shopper's history - This is also complex requiring an approach similar to Netflix although not necessarily as involved. The challenge is dealing with customers who are not very frequent as their preferences are likely to change.
- Similar products -Requires showing products that are similar to what the visitor is viewing. This can be complex because algorithms must find similarities in products for each product viewed by a visitor similar to what Netflix does to categorize movies as similar.
Recommenders by the numbers.
- 35% of Amazon revenue is attributed to their use of product Recommenders.1
- 75% of what Netflix users wach is driven by their Recommender system. 1
- 40% increase of visitor to buyer conversion, 5% revenue/customer incease and 50% overall revenue increase are claimed by Google when using their Recommendation AI service. 2
- Only 7% of on-line store visitors clicked a recommendation but they represent 26% of store revenue.3
- Shoppers that clicked a product rcommendation spent 12.9 minutes on site vs 2.9 minutes for those who did not click a recommendation .4
- $0.35 per 1,000 recommendations is what Google charges for their Recommendation AI Service.
Key considerations if you're thinking of deploying a Recommender System this in your business.
- Product labeling is essential to the more advanced recommendations like accessories, similar products or customer history. Just like Netflix labels its shows with over 100 parameters your recommendation system may require a rich and consistent labeling of parameters. This almost certainly will require doing most if not all of it by hand and can be quite expensive. It is an ongoing process as new products/services are added.
- Dedicated technical staff is likely to be required. A data scientist who designs and improves the recommendation algorithm (complex recommendations mentioned above) and a software coder familiar with the platform you chose are typical.
Key questions you should be asking of vendors.
The list of questions depends on your industry and application of a Recommender System you envision for your business. Here are some questions that apply universally:
- What is the minimum number of transactions and customers does one need for your recommendation system to deliver a good ROI?
- Have you deployed your Recommendation System in my type of (or similar) business?
- How much data is needed, how many parameters per product/customer and what format for a meaningful result? The cost of converting existing data to a format required by a recommender system can be very expensive.
- How much time and cost to deploy a Pilot? Describe a typical deployment timeline.
- What are you providing and what are we responsible for? Buying a bunch of AI tools/software is not likely to do much unless you have access to people with AI experience. AI specialists make between $300,000 - $500,000 in Silicon Valley because there is a shortage of them.
- How much human help is needed for the system to operate. For example, Netflix movie recommender system relies on 40 human experts to categorize each new show. It implements very advanced algorithms but they cannot function without human help.
- How do you measure the accuracy of your recommendation system. Can you provide examples?
- How will I know it is working?
- What specific business improvement can I expect and when?
- How does your system deal with changing customer preferences that are due to changing trends such as fashions, technologies, politics, popularity?
- Will the system require re-training and if so how often? Complex, deep learning AI systems rely on training (see tutorial on Deep Learning) which can get out of date.
https://ignitionone.com - IgnitionOne offers solutions to Retail, Automotive and Travel industries with over 500 customers. Their value proposition reads:
- Engage Customers with Dynamic Content: Deliver compelling, personalized dynamic creative to enhance the customer experience on your website.
- Reduce Cart Abandonment: Present targeted content and offers to engaged customers on your website to capture revenue at key moments of the customer journey.
- Personalize the Omnichannel Experience: With insights into each customer’s products of interest, personalize your messaging across email, display, search, and social to engage your customers no matter where they are.
Overall it lists a lot of benefits that sound similar to over 100 vendors of Recommendation Systems. Other than the marketing text it is impossible to answer any of the questions listed above in order to determine if their solution would be suitable for your business.
How to find a Recommendation System solution for your business
We came up with a better way to find AI solutions than searching the internet, wading through vendors marketing fluff and hype and dealing with persistent sales people. We created a portal (www.180find.com) where you can publish your needs anonymously and let vendors submit their best proposals.
You can still communicate with vendors, ask questions, respond to their comments but stay invisible until you're ready. It is like a dating service, you go off-line once you find something promising.
Our “dating service” works as follows:
- You describe your business needs
- List benefits you expect
- Submit it and we'll publish it anonymously
- We invite all the relevant vendors
- You communicate directly with vendors who respond
- Remain anonymous as long as you wish
- Once you find something take the conversation off -line.
- It is free
We are unbiased, don't feature or promote any vendor. We ask all the relevant vendors to look at your project and come up with a proposal.
We We track news, reports and tutorials on AI deployment in variety of industries. You can visit us to stay informed.