MLentory, developed by ZB MED as part of NFDI4DS, brings together machine learning model metadata from platforms such as Hugging Face, OpenML and AI4Life into a unified discovery portal. As a relatively new registry, MLentory faces a familiar challenge: without years of interaction data, it is difficult to know whether search rankings actually reflect what users find relevant. To better understand how search rankings perform in practice, MLentory has now been integrated with the STELLA evaluation framework, enabling live evaluation of lexical and semantic search approaches directly within the portal.
With GESIS Search, researchers can find social science research datasets, variables from questionnaires, measurement instruments & tools, publications, and documents from the GESIS Library and the GESIS web in one integrated system.
Last week in Madrid, we joined the Information Retrieval community for the annual Conference and Labs of the Evaluation Forum (CLEF). As co-organizers of the LongEval shared task, we were deeply involved in building the Scientific Search dataset for the SciRetrieval task. After nearly a year of preparation, it was fantastic to finally meet with participating teams and discuss their ideas and approaches.
In this demo, we’ll explore the STELLA evaluation infrastructure, designed to assess various retrieval systems in a live, online environment. This tutorial provides a technical overview covering the setup, configuration, and operation of the STELLA infrastructure. You’ll learn how to interact with the system components, including the STELLA Server, STELLA App, experimental search systems, and a rudimentary search interface that simulates a search portal for testing.
The STELLA project has successfully reached its second phase, and we’re happy to share the highlights of our latest community workshop! Held on June 17th and 18th, 2024, at GESIS in Cologne, this two-day event brought together six invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel with the STELLA team from TH Köln, GESIS and ZB Med. The workshop was designed to explore the future of the DFG-funded STELLA project, discuss various use cases and research questions related to the evaluation of academic search systems, and foster connections within the community.
The 1st STELLA Community Workshop was held on 20 June 2022 at TH Köln. Eleven invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel attended the workshop to learn about the DFG-funded STELLA project’s outcomes and discuss the future directions and research questions regarding the evaluation of academic search systems.
One shortcoming of the previous measures (wins, losses, ties) derived from interleaving experiments is the simplified interpretation of click interactions. By weighting clicks differently, it is possible to account for the meaning of the corresponding elements on the search engine result page (SERP). In this blog post, we share our results on weighting clicks on elements of SERPs differently.
A novelty of our living lab implementation is the use of fully-fledged systems that run within a Docker container. Previous living labs were based on pre-computed results only. In our experiments, we validated both approaches and share the results in this blog post.
In this blog post, we share some general information about the [Living Labs for Academic Search (LiLAS)](https://clef-lilas.github.io/) at [CLEF2021](clef2021.clef-initiative.eu/). More specifically, we give an overview about the schedule, participants, and the volume of the logged user interaction data.
We have released a public documentation that covers introductory guides for participants, sites & organizers, as wells as REST endpoints and other technical details. The documentation is available at: https://stella-project.org/stella-documentation/