The 1st STELLA Community Workshop was held on 20 June 2022 at TH Köln. Eleven invited participants from DIPF Frankfurt, TIB Hannover, and ZBW Kiel attended the workshop to learn about the DFG-funded STELLA project’s outcomes and discuss the future directions and research questions regarding the evaluation of academic search systems.
The workshop was split into one overview talk and three discussion talks that we would like to summarize in the following.
In his introduction presentation, Philipp Schaer gave an overview of the basic ideas and motivation behind the STELLA project. After discussing the special requirements that academic search introduces, he outlined the concept of living labs as an alternative to static TREC-based IR evaluation. From there, he outlined the technical and methodological background of the STELLA framework. Next, the LiLAS workshop series at CLEF was briefly summarized, including the results of the evaluation campaign. The talk ended with an overview of the proposal for STELLA II that is currently under review at DFG. This led to a lively discussion on the future directions of STELLA and living lab-based evaluations as a whole.
In the first discussion talk, Daniel Schiffer from DIPF (Frankfurt) introduced the background of the Fachportal Pädagogik (https://www.fachportal-paedagogik.de/), which bundles the information services of more than 30 different academic partners, including the FIS Bildung literature database with more than 1 million entries. One of the driving issues at DIPF is the ever-growing data collection due to the constant updates and maintenance from the different partners. Next to the continuous growth of the system, questions of inconsistency, missing reproducibility, and a low number of relevant documents for many highly-specific queries were some of the main concerns introduced in his talk. A clear research goal for DIPF would be to extract more and better knowledge about different retrieval approaches and make these open and transparent for end users and researchers.
TIB offers the AV Portal (https://av.tib.eu/), which integrates videos and conference recordings for engineering and natural sciences. According to Ralph Ewert, who introduced TIB and the AV Portal, STELLA could be used to recommend similar videos or evaluate video-based features in the result ranking (automatic voice speed recognition, automatic annotation, overlay text, image recognition). Additionally, different personas and user groups might need different search and ranking models. Search-as-learning also plays a role where users’ knowledge is measured before and after the search task.
ZBW is the maintainer of Econbiz (https://www.econbiz.de), an academic research platform with 13 million publications on economics and related fields. Timo Borst visioned using STELLA to evaluate Bibliometrics-enhanced ranking using popularity data and bibliometric indicators developed in the BibRank project. They want to be able to conduct a day-to-day evaluation. Questions were if STELLA can help to conduct lean and cheap experiments, how the search performance is influenced, and if experiments from other platforms can be transferred.
The workshop was complemented by various talks and discussions that originated from the talks. Common issues from all three partners were:
(1) The costs to integrate STELLA in their platforms for integration and maintenance.
(2) If and how STELLA influences the search performances, especially for the time-sensitive ranking process within the portal.
(3) The existing search indices in the portals are built by time-consuming and complex indexing processes. STELLA, so far, uses its own index in the container, while plans exist to allow partners to reuse existing indexing structures to lower the computational effort.
(4) Another discussion was on personas, target groups, and use cases for each portal and that search and ranking processes should be adapted to it. Again, STELLA could help evaluate these different scenarios.
All participants agreed to further discuss and communicate on the common issues and goals. An outcome of the workshop was the first step to forming a core community gathering around topics and questions regarding academic search and IR evaluation. For summer 2023, the next iteration of this community workshop was announced, hopefully with the ongoing support of DFG in the STELLA II project.
Daniel Hienert and Philipp Schaer for the STELLA team, July 21 2022.
The slides of the talks and presentations can be found here.