Into the Wild: Real-World Testing for ML-Based ABR
Abstract
Machine learning (ML)-based Adaptive Bitrate (ABR) algorithms often struggle to bridge the gap between simulation and reality. Their strong performance in synthetic environments frequently fails to generalize to real-world conditions. Researchers have therefore begun testing these algorithms over the Internet to incorporate real-world feedback into their design. In this paper, we show that since network conditions vary significantly across the globe, testing in individual real-world environments can suffer from the same generalization issues as lab-based testing. Existing testing platforms face (and might even be oblivious to) this limitation because they cover a small geographical region and rely on a narrow set of users affected by survivorship bias. As a result, their insights on an algorithm’s performance generalize poorly to other deployments across the Internet, hindering the widespread adoption of ML-based ABR methods in practice. To address this gap, we present ABR-Arena, a global testing platform that enables researchers to evaluate the performance of ABR algorithms across a diverse set of regions around the globe. As a result of its worldwide coverage, ABR-Arena can reveal the performance shortcomings of several state-of-the-art ML-based approaches. It is extensible and easy to deploy in additional locations. We will make ABR-Arena available to the community to support the development of new ML-based approaches and to facilitate meaningful improvements to existing algorithms.
Research Area: Network Analysis and Reasoning
People
BibTex
@inproceedings{hoffman2025into,
title={Into the Wild: Real-World Testing for ML-Based ABR},
author={Hoffman, Benjamin and Dietm{"u}ller, Alexander and Mishra, Ayush and Vanbever, Laurent},
booktitle={Practical Adoption Challenges of ML for Systems (PACMI 2025) co-located with SOSP 2025},
year={2025}
}Research Collection: 20.500.11850/783319
Slide Sources: https://gitlab.ethz.ch/projects/59362