Back to Study Material Artificial Intelligence

Recommendation systems

Recommendation systems are AI systems that predict user preferences and surface items, such as products, movies, music, or articles, that individual users ...

Recommendation systems

Personalising Information at Scale

Recommendation systems are AI systems that predict user preferences and surface items, such as products, movies, music, or articles, that individual users are likely to find relevant or enjoyable. They are commercially critical: Netflix estimates that its recommendation system saves over one billion dollars per year by reducing subscriber churn, and Amazon attributes approximately thirty-five percent of its revenue to recommended products.

Content-based filtering recommends items similar to those a user has previously liked, based on item attributes. If a user watches three science fiction films with space exploration themes, a content-based system recommends other films with similar genre and theme features. The system requires rich item feature representations but does not need data about other users, making it effective for new users or niche items.

Collaborative filtering finds users with similar preferences and recommends items those similar users enjoyed. User-based collaborative filtering identifies a neighbourhood of users most similar to the target user using cosine similarity or Pearson correlation of their rating vectors. Item-based collaborative filtering instead finds items that are frequently rated similarly by users and recommends items similar to what the target user has rated highly. Matrix factorisation methods like Singular Value Decomposition and its variant used in the Netflix Prize, namely the factorised user and item embedding approach, decompose the user-item rating matrix into latent factor vectors that capture underlying preference dimensions such as genre taste or production quality.

The Cold Start Problem

New users and new items present the cold start problem: a new user has no rating history for collaborative filtering to exploit, and a new item has received no ratings. Hybrid systems that combine content-based and collaborative approaches mitigate this by using content features for new items and asking new users a brief preference survey to bootstrap their profile.

Interview Tip

Interviewers at tech companies often ask candidates to design a recommendation system for a specific product. Structure your answer around the data available, the algorithm choice justified by the scale and sparsity of that data, and how you would evaluate the system offline using held-out ratings before A/B testing online. Mentioning precision at k and normalised discounted cumulative gain as offline evaluation metrics for ranked recommendation lists demonstrates knowledge of the specific measurement tools used in industry.