About

STELLA (InfraSTrucutrEs for Living LAbs) offers an Evaluation-as-a-Service platform for living lab experiments with ranking and recommender systems. By using STELLA, researchers can evaluate their experimental systems based on user feedback which stands in contrast to (or complements) the Cranfield-style approaches with test collections in offline evaluations. STELLA facilitates conventional AB tests but also more data-efficient interleaving experiments in which results lists of two ranking or recommender functions are mixed. A fundamental component of STELLA is the integration of experimental systems as micro-services. While previous living labs restricted the system results to the most popular top-k queries, we allow more comprehensive evaluations by integrating micro-services with entire retrieval and recommender systems. The Living Labs for Academic Search (LiLAS) lab at CLEF made use of the STELLA infrastructure and served as the first test-bed to evaluate the feasibility of our new infrastructure design. We welcome contributions and look for collaborations with researchers and sites alike.