STELLA enables researchers and coders to deploy and evaluate search and recommendation algorithms in a real-world scenario. These so-called “Living Labs” provide a user-focused and thus more realistic evaluation approach. What are the crucial steps when implementing a participant-system in STELLA?
The DFG-funded STELLA project aims to create an evaluation infrastructure that allows to evaluate search and recommendation services within productive web-based search systems with real users.
In academia, we often rely on standard test collections with static relevance judgments that somewhat abstract away the specific information need of single users. Thus, evaluating search engine algorithms with real-world users is a desiderata. Finally, we want to know how users can benefit from innovative search engines. The STELLA project tries to bridge this gap with the help of an innovative infrastructure that makes it possible to evaluate experimental search algorithms in live environments. In this post we want to outline the general idea of our infrastructure.
We invite you to submit a early research of open ideas paper to the LiLAS workshop. The Living Labs for Academic Search (LiLAS) Workshop at CLEF 2020 will be held at Thessaloniki, Greece between 22-25 September 2020. The Call for Contributions is open until 24 May 2020. Please feel free to contact us if you have any questions!