Recommender Systems: Evaluation and Metrics

About this course: In this course you will learn how to evaluate recommender systems. You will gain familiarity with several families of metrics, including ones to measure prediction accuracy, rank accuracy, decision-support, and other factors such as diversity, product coverage, and serendipity. You will learn how different metrics relate to different user goals and business goals. You will also learn how to rigorously conduct offline evaluations (i.e., how to prepare and sample data, and how to aggregate results). And you will learn about online (experimental) evaluation. At the completion of this course you will have the tools you need to compare different recommender system alternatives for a wide variety of uses.

Created by:   University of Minnesota

  • Michael D. Ekstrand
    Taught by:    Michael D. Ekstrand, Assistant Professor
    Dept. of Computer Science, Boise State University

  • Joseph A Konstan
    Taught by:    Joseph A Konstan, Distinguished McKnight Professor and Distinguished University Teaching Professor
    Computer Science and Engineering
Basic Info
Course 3 of 5 in the Recommender Systems Specialization.
How To PassPass all graded assignments to complete the course.
2 videos
  1. Video: Introduction to Evaluation and Metrics
  2. Video: The Goals of Evaluation
Basic Prediction and Recommendation Metrics
5 videos1 reading
  1. Video: Hidden Data Evaluation
  2. Video: Prediction Accuracy Metrics
  3. Video: Decision Support Metrics
  4. Video: Rank-Aware Top-N Metrics
  5. Video: Assignment Intro Video
  6. Reading: Metric Computation Assignment Instructions
Graded: Basic Prediction and Recommendation Metrics Assignment
Advanced Metrics and Offline Evaluation
6 videos1 reading
  1. Video: Beyond Basic Evaluation
  2. Video: Additional Item and List-Based Metrics
  3. Video: Experimental Protocols
  4. Video: Unary Data Evaluation
  5. Video: Temporal Evaluation of Recommenders (Interview with Neal Lathia)
  6. Video: Programming Assignment Introduction
  7. Reading: Evaluating Recommenders
Graded: Offline Evaluation and Metrics Quiz
Graded: Programming Assignment Quiz
Online Evaluation
4 videos
  1. Video: Introduction to Online Evaluation and User Studies
  2. Video: Usage Logs and Analysis
  3. Video: A/B Studies (Field Experiments)
  4. Video: User-Centered Evaluation (Interview with Bart Knijnenburg)
Graded: Online Evaluation Quiz
Evaluation Design
3 videos2 readings
  1. Video: Matching Evaluation to the Problem/Challenge
  2. Video: Case Examples
  3. Video: Assignment Intro Video
  4. Reading: Intro to Assignment: Evaluation Design Cases
  5. Reading: Quiz Debrief
Graded: Assignment: Evaluation Design Cases
How It Works
Each course is like an interactive textbook, featuring pre-recorded videos, quizzes and projects.
Help from Your Peers
Help from Your Peers
Connect with thousands of other learners and debate ideas, discuss course material, and get help mastering concepts.
Earn official recognition for your work, and share your success with friends, colleagues, and employers.
University of Minnesota
The University of Minnesota is among the largest public research universities in the country, offering undergraduate, graduate, and professional students a multitude of opportunities for study and research. Located at the heart of one of the nation’s most vibrant, diverse metropolitan communities, students on the campuses in Minneapolis and St. Paul benefit from extensive partnerships with world-renowned health centers, international corporations, government agencies, and arts, nonprofit, and public service organizations.

No comments:

Post a Comment