Infrastructures for Living Labs
The DFG-funded STELLA project aims to create an evaluation infrastructure that allows to evaluate search and recommendation services within productive web-based search systems with real users.
The 1st STELLA Community Workshop was held on 20 June 2022 at TH Köln. Eleven invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel attended the workshop to learn about the DFG-funded STELLA project’s outcomes and discuss the future directions and research questions regarding the evaluation of academic search systems.
One shortcoming of the previous measures (wins, losses, ties) derived from interleaving experiments is the simplified interpretation of click interactions. By weighting clicks differently, it is possible to account for the meaning of the corresponding elements on the search engine result page (SERP). In this blog post, we share our results on weighting clicks on elements of SERPs differently.
A novelty of our living lab implementation is the use of fully-fledged systems that run within a Docker container. Previous living labs were based on pre-computed results only. In our experiments, we validated both approaches and share the results in this blog post.
In this blog post, we share some general information about the Living Labs for Academic Search (LiLAS) at CLEF2021. More specifically, we give an overview about the schedule, participants, and the volume of the logged user interaction data.