The DFG-funded STELLA project aims to create an evaluation infrastructure that
allows to evaluate search and recommendation services within productive
web-based search systems with real users.
STELLA provides an integrated
e-Research environment that allows researchers in the field of information
retrieval and recommendation services to conduct studies with real users in
real environments. The experimental set-ups differ considerably from classical
TREC studies, which can only be carried out offline, or also from user studies,
which only allow laboratory experiments, and thus enable researchers to use an
evaluation method that was previously reserved only for industrial research or
the operators of large online platforms.
STELLA is a joint project involving
research groups from Technische Hochschule Köln, ZB MED - Leibniz Information
Centre for Live Sciences, and GESIS - Leibniz Institut for the Social Sciences.
About the infrastructure
STELLA (InfraSTrucutrEs for Living LAbs) offers an Evaluation-as-a-Service platform for living lab experiments with ranking and recommender systems. By using STELLA, researchers can evaluate their experimental systems based on user feedback which stands in contrast to the Cranfield-style approaches with test collections in offline evaluations. STELLA facilitates conventional AB tests but also more data-efficient interleaving experiments in which results lists of two ranking or recommender functions are mixed. A fundamental component of STELLA is the integration of experimental systems as micro-services. While previous living labs restricted the system results to the most popular top-k queries, we allow more comprehensive evaluations by integrating micro-services with entire retrieval and recommender systems. The CLEF lab “Living Labs for Academic Search (LiLAS)” made use of the STELLA infrastructure and served as the first test-bed to evaluate the feasibility of our new infrastructure design. We welcome contributions and look for collaborations with researchers and sites alike. The entire codebase is open-source and hosted on GitHub.