Recommender Systems: Evaluation and Metrics

About this course: In this course you will learn how to evaluate recommender systems. You will gain familiarity with several families of metrics, including ones to measure prediction accuracy, rank accuracy, decision-support, and other factors such as diversity, product coverage, and serendipity. You will learn how different metrics relate to different user goals and business goals. You will also learn how to rigorously conduct offline evaluations (i.e., how to prepare and sample data, and how to aggregate results). And you will learn about online (experimental) evaluation. At the completion of this course you will have the tools you need to compare different recommender system alternatives for a wide variety of uses.

Created by:  University of Minnesota

• Taught by:  Michael D. Ekstrand, Assistant Professor

  Dept. of Computer Science, Boise State University

• Taught by:  Joseph A Konstan, Distinguished McKnight Professor and Distinguished University Teaching Professor

  Computer Science and Engineering

Basic Info Course 3 of 5 in the Recommender Systems Specialization Language English How To Pass Pass all graded assignments to complete the course. User Ratings 4.3 stars Average User Rating 4.3See what learners said Coursework

Each course is like an interactive textbook, featuring pre-recorded videos, quizzes and projects.

Help from your peers

Connect with thousands of other learners and debate ideas, discuss course material, and get help mastering concepts.

Certificates

Earn official recognition for your work, and share your success with friends, colleagues, and employers.

University of Minnesota The University of Minnesota is among the largest public research universities in the country, offering undergraduate, graduate, and professional students a multitude of opportunities for study and research. Located at the heart of one of the nation’s most vibrant, diverse metropolitan communities, students on the campuses in Minneapolis and St. Paul benefit from extensive partnerships with world-renowned health centers, international corporations, government agencies, and arts, nonprofit, and public service organizations.

Syllabus


WEEK 1


Preface



2 videos expand

1. Video: Introduction to Evaluation and Metrics
2. Video: The Goals of Evaluation

Basic Prediction and Recommendation Metrics



5 videos, 1 reading expand

1. Video: Hidden Data Evaluation
2. Video: Prediction Accuracy Metrics
3. Video: Decision Support Metrics
4. Video: Rank-Aware Top-N Metrics
5. Video: Assignment Intro Video
6. Reading: Metric Computation Assignment Instructions

Graded: Basic Prediction and Recommendation Metrics Assignment

WEEK 2


Advanced Metrics and Offline Evaluation



6 videos, 1 reading expand

1. Video: Beyond Basic Evaluation
2. Video: Additional Item and List-Based Metrics
3. Video: Experimental Protocols
4. Video: Unary Data Evaluation
5. Video: Temporal Evaluation of Recommenders (Interview with Neal Lathia)
6. Video: Programming Assignment Introduction
7. Reading: Evaluating Recommenders

Graded: Offline Evaluation and Metrics Quiz
Graded: Programming Assignment Quiz

WEEK 3


Online Evaluation



4 videos expand

1. Video: Introduction to Online Evaluation and User Studies
2. Video: Usage Logs and Analysis
3. Video: A/B Studies (Field Experiments)
4. Video: User-Centered Evaluation (Interview with Bart Knijnenburg)

Graded: Online Evaluation Quiz

WEEK 4


Evaluation Design



3 videos, 2 readings expand

1. Video: Matching Evaluation to the Problem/Challenge
2. Video: Case Examples
3. Video: Assignment Intro Video
4. Reading: Intro to Assignment: Evaluation Design Cases
5. Reading: Quiz Debrief

Graded: Assignment: Evaluation Design Cases