The STELLA project has successfully reached its second phase, and we’re happy to share the highlights of our latest community workshop! Held on June 17th and 18th, 2024, at GESIS in Cologne, this two-day event brought together six invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel with the STELLA team from TH Köln, GESIS and ZB Med. The workshop was designed to explore the future of the DFG-funded STELLA project, discuss various use cases and research questions related to the evaluation of academic search systems, and foster connections within the community.
The workshop was structured as an interactive, collaborative experience, running from lunch on the first day to lunch on the second. The agenda was packed with open discussions aimed at identifying common challenges and needs, as well as focused work sessions where participants engaged directly with the STELLA team on practical implementations.
Day 1 Highlights
- Welcome and Introduction: Daniel Hienert opened the workshop with a warm welcome and an introduction to the STELLA project. He provided a brief recap of the first project phase and outlined the goals for the future, emphasizing the motivation behind the STELLA project and the concept of living lab evaluation.
- STELLA Presentation and Demo: Rohitha Ravinder and Deniz Türkmen delivered an in-depth presentation on living lab evaluations within STELLA, detailing how interleaving techniques can be used to evaluate ranking and recommender systems through real user interactions. They walked us through the STELLA workflow, offering insights into the multi-container application’s infrastructure and a sneak peek at the upcoming continuous evaluation modules.
- Interactive Live Demo: Jüri Keller followed with a practical demonstration, guiding participants through a live demo. Together, we configured a simple experimental system, integrating it into the infrastructure via the STELLA Server, and interacted with the results interleaved by the STELLA App. The live demo is available as a video recording and also as a tutorial for those interested in reproducing the process.
- Partner Presentations and Discussions: Participants from the various institutes shared their research focuses and use cases, leading to engaging discussions that set the stage for the following breakout sessions.
Day 2 Highlights
- Breakout Sessions: The second day was dedicated to tailored breakout sessions, where participants’ specific research questions were further explored and developed. These sessions were instrumental in identifying requirements and feature requests, leading to the creation of initial roadmaps for future collaborations.
- Final Wrap-Up: The workshop concluded with a wrap-up session where key takeaways were summarized, and the next steps for the STELLA project were discussed.
We’re excited to continue improving STELLA in collaboration with our partners and are eager to explore the diverse range of retrieval tasks and research questions that emerged from this workshop. The energy and engagement from all participants have set a strong foundation for the project’s next phase, and we look forward to the continued journey together.
Jüri Keller for the STELLA team, August 14 2024.