Machine Learning System Design: Multi-Model-Based Recommendation & Identification

Abstract

Fanfei Meng

This document presents a comprehensive design framework for two machine learning systems aimed at optimizing recommendation and identification tasks in distinct domains: an Ads Ranking System and a Family-Friendly Listing Ranking System. Both systems leverage multi-modal data, advanced modeling techniques, and robust evaluation methods to achieve high performance and scalability.

The Ads Ranking System prioritizes ads for user engagement and revenue optimization through short-term metrics such as Click-Through Rate (CTR), Conversion Rate (CVR), and Revenue Per Mile (RPM), alongside long-term metrics including user retention and model latency. It integrates diverse data sources, including user behavior, ad content (text, images, tabular data), and contextual information. The system employs feature engineering techniques to generate embeddings for visual, textual, and tabular data and uses models ranging from XGBoost to advanced neural architectures like Deep Interest Networks (DIN). Offline and online evaluation metrics such as AUC, NDCG, and real-time business metrics ensure robust performance monitoring and iterative improvement. The Family-Friendly Listing Ranking System focuses on classifying and ranking listings for family-friendliness, considering features such as amenities, reviews, and location safety. The model strategy incorporates tree-based methods for interpretability and multi-tower neural networks for handling unstructured data. Evaluation involves precision, recall, and ranking metrics alongside A/B testing to align offline improvements with business goals.

Challenges like data distribution shifts and user experience mismatches are addressed through feature refinement and explain ability tools.

This work highlights the integration of machine learning, multi-modal data processing, and systematic evaluation to build scalable and impactful recommendation systems. It also underscores the importance of balancing interpretability, computational efficiency, and long-term user satisfaction.

PDF