STELLA

Code

Mon, 01 Jan 0001 00:00:00 +0000

All of the project's code is available open source in the STELLA GitHub Project. Below you find a short list of the most important repositories: The STELLA server is the central instance of the infrastructure. It handles the user administration, provides access to a dashboard, it automatically updates the STELLA app, and it hosts the database with user interaction logs. The STELLA app is deployed at sites that want to conduct IR and recommender experiments and evaluate them with user interactions.

About the Project

Mon, 01 Jan 0001 00:00:00 +0000

The DFG-funded STELLA project aims to create an evaluation infrastructure that allows to evaluate search and recommendation services within productive web-based search systems with real users. STELLA provides an integrated e-Research environment that allows researchers in the field of information retrieval and recommendation services to conduct studies with real users in real environments. The experimental set-ups differ considerably from classical TREC studies, which can only be carried out offline, or also from user studies, which only allow laboratory experiments, and thus enable researchers to use an evaluation method that was previously reserved only for industrial research or the operators of large online platforms.

Recommendations for similar documents now in GESIS Search

Mon, 10 Nov 2025 00:00:00 +0000

With GESIS Search, researchers can find social science research datasets, variables from questionnaires, measurement instruments & tools, publications, and documents from the GESIS Library and the GESIS web in one integrated system.

Insights from LongEval at CLEF 2026

Fri, 12 Sep 2025 00:00:00 +0000

Last week in Madrid, we joined the Information Retrieval community for the annual Conference and Labs of the Evaluation Forum (CLEF). As co-organizers of the LongEval shared task, we were deeply involved in building the Scientific Search dataset for the SciRetrieval task. After nearly a year of preparation, it was fantastic to finally meet with participating teams and discuss their ideas and approaches.

Demo Tutorial for the STELLA Evaluation Infrastructure.

Wed, 14 Aug 2024 00:00:00 +0000

In this demo, we’ll explore the STELLA evaluation infrastructure, designed to assess various retrieval systems in a live, online environment. This tutorial provides a technical overview covering the setup, configuration, and operation of the STELLA infrastructure. You’ll learn how to interact with the system components, including the STELLA Server, STELLA App, experimental search systems, and a rudimentary search interface that simulates a search portal for testing.

Insights from the Second Community Workshop

Wed, 14 Aug 2024 00:00:00 +0000

The STELLA project has successfully reached its second phase, and we’re happy to share the highlights of our latest community workshop! Held on June 17th and 18th, 2024, at GESIS in Cologne, this two-day event brought together six invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel with the STELLA team from TH Köln, GESIS and ZB Med. The workshop was designed to explore the future of the DFG-funded STELLA project, discuss various use cases and research questions related to the evaluation of academic search systems, and foster connections within the community.

Report of the 1st STELLA Community Workshop.

Thu, 21 Jul 2022 00:00:00 +0000

The 1st STELLA Community Workshop was held on 20 June 2022 at TH Köln. Eleven invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel attended the workshop to learn about the DFG-funded STELLA project’s outcomes and discuss the future directions and research questions regarding the evaluation of academic search systems.

Evaluating clicks on different SERP elements.

Sat, 02 Oct 2021 00:00:00 +0000

One shortcoming of the previous measures (wins, losses, ties) derived from interleaving experiments is the simplified interpretation of click interactions. By weighting clicks differently, it is possible to account for the meaning of the corresponding elements on the search engine result page (SERP). In this blog post, we share our results on weighting clicks on elements of SERPs differently.

On dockerized systems vs. pre-computed runs.

Fri, 01 Oct 2021 00:00:00 +0000

A novelty of our living lab implementation is the use of fully-fledged systems that run within a Docker container. Previous living labs were based on pre-computed results only. In our experiments, we validated both approaches and share the results in this blog post.

CLEF2021 lab overview

Thu, 30 Sep 2021 00:00:00 +0000

In this blog post, we share some general information about the [Living Labs for Academic Search (LiLAS)](https://clef-lilas.github.io/) at [CLEF2021](clef2021.clef-initiative.eu/). More specifically, we give an overview about the schedule, participants, and the volume of the logged user interaction data.

STELLA Documentation released!

Sat, 01 May 2021 00:00:00 +0000

We have released a public documentation that covers introductory guides for participants, sites & organizers, as wells as REST endpoints and other technical details. The documentation is available at: https://stella-project.org/stella-documentation/

STELLA at ISI2021

Thu, 01 Apr 2021 00:00:00 +0000

We have presented STELLA at [ISI 2021 - 16th International Symposium for Information Science](https://isi2021.net/)! In our talk, we provide an updated overview of the STELLA infrastructure and outline how participants can contribute to the living lab experiments. It was held before our CLEF2021 lab in order to promote the infrastructure and discuss it with an expert community. You will find the corresponding paper here: https://epub.uni-regensburg.de/44953 You will find a record of our talk here: https://av.

New tutorial video available on YouTube!

Fri, 01 Jan 2021 00:00:00 +0000

In one of our [earlier blog posts](../posts/STELLA-participant-systems-in-STELLA), we outlined how to participate. The corresponding repository of the template can be found at [GitHub](https://github.com/stella-project/stella-micro-template).

For those who prefer videos instead of reading guides, we have prepared an introduction that is now available on YouTube!

Integrating Participant Systems in STELLA - An Overview

Mon, 22 Jun 2020 00:00:00 +0000

STELLA enables researchers and coders to deploy and evaluate search and recommendation algorithms in a real-world scenario. These so-called "Living Labs" provide a user-focused and thus more realistic evaluation approach. What are the crucial steps when implementing a participant-system in STELLA?

STELLA - A Primer on our Infrastructure

Fri, 08 May 2020 00:00:00 +0000

In academia, we often rely on standard test collections with static relevance judgments that somewhat abstract away the specific information need of single users. Thus, evaluating search engine algorithms with real-world users is a desiderata. Finally, we want to know how users can benefit from innovative search engines. The STELLA project tries to bridge this gap with the help of an innovative infrastructure that makes it possible to evaluate experimental search algorithms in live environments. In this post we want to outline the general idea of our infrastructure.

LiLAS at CLEF 2020 - Call for Papers

Mon, 06 Apr 2020 00:00:00 +0000

We invite you to submit a early research of open ideas paper to the LiLAS workshop. The [Living Labs for Academic Search](https://clef-lilas.github.io) (LiLAS) Workshop at [CLEF 2020](https://clef2020.clef-initiative.eu) will be held at Thessaloniki, Greece between 22-25 September 2020. The [Call for Contributions](https://clef-lilas.github.io/#call-for-contribution) is open until 24 May 2020. Please feel free to [contact us](mailto:lilas@stella-project.org) if you have any questions!

New Special-Issue on IR Evaluation of Datenbank-Spektrum

Thu, 26 Mar 2020 00:00:00 +0000

The [Datenbank-Spektrum](https://link.springer.com/journal/13222) has a special-issue on *[Trends in Information Retrieval Evaluation](https://link.springer.com/journal/13222/20/1)* that includes an [article on STELLA](https://link.springer.com/article/10.1007/s13222-020-00335-x) and it's overall idea and infrastructure.

Press Release by HN NRW Hochschulnetzwerk

Thu, 23 May 2019 00:00:00 +0000

Follow this [link](https://www.hn-nrw.de/dfg-foerderung-fuer-living-lab-projekt/) to see the official press announcement by HN NRW Hochschulnetzwerk.

Press Release by TH Köln

Thu, 23 May 2019 00:00:00 +0000

Follow this [link](https://www.th-koeln.de/hochschule/dfg-foerderung-fuer-living-lab-projekt---gesis-th-koeln-und-zb-med-entwickeln-virtuelle-infrastruktur_64086.php) to see the official press announcement by the TH Köln.

Project Description by TH Köln

Thu, 23 May 2019 00:00:00 +0000

Follow this [link](https://www.th-koeln.de/en/information-science-and-communication-studies/stella---infrastructures-for-living-labs_68533.php) to see the project description by the TH Köln.

ZB MED Project Page

Thu, 23 May 2019 00:00:00 +0000

Follow this [link](https://www.zbmed.de/forschung-entwicklung/laufende-projekte/stella/) to see the project page by ZB MED. Click [here](https://www.zbmed.de/ueber-uns/presse/neuigkeiten-aus-zb-med/artikel/dfg-foerderung-fuer-living-lab-projekt-gesis-th-koeln-und-zb-med-entwickeln-gemeinsame-virtuelle-inf/) for the official press announcement by ZB MED.

People

Mon, 01 Jan 0001 00:00:00 +0000

TH Köln - University of Applied Sciences Philipp Schaer Jüri Keller GESIS - Leibniz-Institute for the Social Sciences Daniel Hienert Deniz Türkmen ZB MED - Information Centre for Life Sciences Leyla Jael García-Castro Rohitha Ravinder Former Members Timo Breuer (TH Köln) Johann Schaible (GESIS) Narges Tavakolpoursaleh (GESIS) Zeljko Carevic (GESIS) Bernd Müller (ZB MED) Benjamin Wolff (ZB MED)

STELLA

Mon, 01 Jan 0001 00:00:00 +0000

About The DFG-funded STELLA project aims to create an evaluation infrastructure that allows to evaluate search and recommendation services within productive web-based search systems with real users. STELLA provides an integrated e-Research environment that allows researchers in the field of information retrieval and recommendation services to conduct studies with real users in real environments. The experimental set-ups differ considerably from classical TREC studies, which can only be carried out offline, or also from user studies, which only allow laboratory experiments, and thus enable researchers to use an evaluation method that was previously reserved only for industrial research or the operators of large online platforms.

STELLA Publications

Mon, 01 Jan 0001 00:00:00 +0000

2025 Counterfactual Query Rewriting to Use Historical Relevance Feedback. In: ECIR. 2025. Jüri Keller, Maik Fröbe, Gijs Hendriksen, Daria Alexander, Martin Potthast, Matthias Hagen and Philipp Schaer. [doi] [pdf] [BibTeX] 2024 Report on the 1st Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024. ACM SIGIR Forum , 58(2), 2024. Timo Breuer, Christin Katharina Kreutz, Norbert Fuhr, Krisztian Balog, Philipp Schaer, Nolwenn Bernard, Ingo Frommholz, Marcel Gohsen, Kaixin Ji, Gareth J.