The Netflix Prize was an open competition held by Netflix, a video streaming service, to develop the best collaborative filtering algorithm for predicting user ratings of films. The contest, launched on October 2, 2006, offered a grand prize of US$1,000,000 to any team that could improve Netflix's own recommendation algorithm, Cinematch, by 10% in root mean squared error (RMSE). The competition was open to anyone not connected with Netflix and not a resident of certain blocked countries, such as Cuba or North Korea. On September 21, 2009, the grand prize was awarded to the team BellKor's Pragmatic Chaos, which achieved a 10.06% improvement over Cinematch.
The competition was a landmark event in the field of Machine learning, demonstrating the power of ensemble methods and sparking widespread interest in collaborative-filtering techniques. It provided a unique, large-scale dataset and a clear objective, attracting over 20,000 teams from more than 150 countries.
Problem and data sets
Netflix provided a training data set of 100,480,507 ratings that 480,189 users gave to 17,770 movies. Each training rating was a quadruplet of the form <user, movie, date of grade, grade>, where user and movie were integer IDs and grades were integers from 1 to 5. The qualifying data set contained over 2,817,131 triplets of the form <user, movie, date of grade>, with grades known only to the jury. A participating team's algorithm had to predict grades on the entire qualifying set, but the team was informed of the score for only half of the data: a quiz set of 1,408,342 ratings. The other half, the test set of 1,408,789 ratings, was used by the jury to determine prize winners. Only the judges knew which ratings were in the quiz set and which were in the test set, making it difficult to overfit on the test set.
Submitted predictions were scored against the true grades using RMSE, and the goal was to reduce this error as much as possible. Netflix also identified a probe subset of 1,408,395 ratings within the training data set, chosen to have similar statistical properties to the quiz and test sets. The training set was constructed such that the average user rated over 200 movies, and the average movie was rated by over 5,000 users, but there was wide variance: some movies had as few as 3 ratings, while one user rated over 17,000 movies. To protect customer privacy, some rating data were deliberately perturbed by deleting ratings, inserting alternative ratings and dates, or modifying rating dates.
There was some controversy over the choice of RMSE as the defining metric, as it was claimed that even a 1% improvement in RMSE could significantly affect the ranking of the top-10 recommended movies for a user.
Prizes
Prizes were based on improvement over Netflix's own algorithm, Cinematch, or over the previous year's score if a team had improved beyond a certain threshold. A trivial algorithm that predicted each movie's average grade produced an RMSE of 1.0540 on the quiz set. Cinematch, which used straightforward statistical linear models with a lot of data conditioning, scored an RMSE of 0.9514 on the quiz data, roughly a 10% improvement over the trivial algorithm. To win the grand prize, a team had to achieve an RMSE of 0.8572 on the test set, a 10% improvement over Cinematch.
As long as no team won the grand prize, a progress prize of $50,000 was awarded every year for the best result, provided the algorithm improved the RMSE on the quiz set by at least 1% over the previous progress prize winner (or over Cinematch in the first year). To claim a prize, a participant had to provide source code and a description of the algorithm to the jury within one week, and after verification, provide a non-exclusive license to Netflix. Netflix would publish only the description, not the source code. Teams could submit as many prediction sets as they wished, initially once a week but later once a day. Once a team achieved a 10% improvement, the jury issued a last call, giving all teams 30 days to submit their final entries. The contest would last until the grand prize winner was declared, but would have been terminated at Netflix's discretion after at least five years (until October 2, 2011) if no winner had emerged.
Progress over the years
The competition began on October 2, 2006. By October 8, a team called WXYZConsulting had already beaten Cinematch's results. By October 15, three teams had beaten Cinematch, one by 1.06%, enough to qualify for the annual progress prize. By June 2007, over 20,000 teams had registered from over 150 countries, and 2,000 teams had submitted over 13,000 prediction sets.
Over the first year, several front-runners traded first place, including WXYZConsulting (Wei Xu and Yi Zhang), ML@UToronto A from the University of Toronto led by Prof. Geoffrey Hinton, Gravity from the Budapest University of Technology, and BellKor from AT&T Labs. These teams employed a variety of techniques, including matrix-factorization, Ensemble Learning, and Neural network approaches, often combining multiple models to achieve incremental improvements.
Impact and legacy
The Netflix Prize had a significant impact on the field of recommender-systems and Machine learning. It demonstrated the effectiveness of blending many predictive models to achieve state-of-the-art results, a technique that became widely adopted in industry. The competition also highlighted the importance of data-mining on large-scale, real-world datasets and spurred research into collaborative-filtering algorithms. The winning team, BellKor's Pragmatic Chaos, combined over 100 different models, including restricted-boltzmann-machines and matrix-factorization variants, to achieve the final 10.06% improvement.
The competition also raised awareness of privacy issues in data sharing, as Netflix later faced a lawsuit over the release of the dataset, which could potentially identify users. Despite these concerns, the Netflix Prize remains a benchmark in the history of Artificial intelligence and data-science, inspiring similar competitions and contributing to the development of modern recommendation systems used by streaming services and e-commerce platforms.
See also
- Machine learning
- collaborative-filtering
- recommender-systems
- data-mining