STELLA Publications

2024

Evaluation of Temporal Change in IR Test Collections.
In: Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, series ICTIR '24, pages 3–13. Association for Computing Machinery, New York, NY, USA, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[doi] [pdf]  [abstract]  [BibTeX] 
Leveraging Prior Relevance Signals in Web Search.
In: G. Faggioli, N. Ferro, P. Galuscáková and A. G. S. de Herrera, editors, Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024, volume 3740, series CEUR Workshop Proceedings, pages 2396-2406. CEUR-WS.org, 2024.
Jüri Keller, Timo Breuer and Philipp Schaer.
[doi]  [BibTeX] 
SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024).
In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, series SIGIR '24, pages 3058–3061. Association for Computing Machinery, New York, NY, USA, 2024.
Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog, Timo Breuer and Norbert Fuhr.
[doi] [pdf]  [abstract]  [BibTeX] 

2023

Evaluating Temporal Persistence Using Replicability Measures.
In: M. Aliannejadi, G. Faggioli, N. Ferro and M. Vlachos, editors, Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023, volume 3497, series CEUR Workshop Proceedings, pages 2441-2457. CEUR-WS.org, 2023.
Jüri Keller, Timo Breuer and Philipp Schaer.
[doi]  [BibTeX] 
An in-depth investigation on the behavior of measures to quantify reproducibility.
Information Processing and Management, 60(3):103332, 2023.
Maria Maistro, Timo Breuer, Philipp Schaer and Nicola Ferro.
[doi] [pdf]  [BibTeX] 
Preliminary Results of a Scientometric Analysis of the German Information Retrieval Community 2020-2023.
In: M. Leyer and J. Wichmann, editors, Proceedings of the LWDA 2023 Workshops: BIA, DB, IR, KDML and WM. Marburg, Germany, 09.-11. October 2023, volume 3630, series CEUR Workshop Proceedings, pages 222-230. CEUR-WS.org, 2023.
Philipp Schaer, Svetlana Myshkina and Jüri Keller.
[doi] [pdf]  [BibTeX] 

2022

irmetadata: An Extensible Metadata Schema for IR Experiments.
In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3078-3089. ACM, 2022.
Timo Breuer, Jüri Keller and Philipp Schaer.
[doi] [pdf]  [BibTeX] 
Validating Simulations of User Query Variants.
In: M. Hagen, S. Verberne, C. Macdonald, C. Seifert, K. Balog, K. Nørvåg and V. Setty, editors, Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I, volume 13185, series Lecture Notes in Computer Science, pages 80-94. Springer, 2022.
Timo Breuer, Norbert Fuhr and Philipp Schaer.
[doi] [pdf]  [BibTeX] 
Evaluating Research Dataset Recommendations in a Living Lab.
In: A. Barrón-Cedeño, G. Da San Martino, M. Degli Esposti, F. Sebastiani, C. Macdonald, G. Pasi, A. Hanbury, M. Potthast, G. Faggioli and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 135-148. Springer International Publishing, Cham, 2022.
Jüri Keller and Leon Paul Mondrian Munz.
[doi] [pdf]  [abstract]  [BibTeX] 

2021

A Living Lab Architecture for Reproducible Shared Task Experimentation.
In: Information between Data and Knowledge, pages 348-362. Werner Hülsbusch, Glückstadt, 2021. Session 6: Emerging Technologies
Timo Breuer and Philipp Schaer.
[pdf]  [abstract]  [BibTeX] 
Evaluating Elements of Web-based Data Enrichment for Pseudo-Relevance Feedback Retrieval.
In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021), series Lecture Notes in Computer Science. Springer Nature, 2021.
Timo Breuer, Melanie Pest and Philipp Schaer.
[pdf]  [BibTeX] 
reproeval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments.
In: D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast and F. Sebastiani, editors, Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II, volume 12657, series Lecture Notes in Computer Science, pages 481-486. Springer, 2021.
Timo Breuer, Nicola Ferro, Maria Maistro and Philipp Schaer.
[doi] [pdf]  [BibTeX] 
TEKMA at CLEF-2021: BM-25 based rankings for scientific publication retrieval and data set recommendation.
In: G. Faggioli, N. Ferro, A. Joly, M. Maistro and F. Piroi, editors, Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021, volume 2936, series CEUR Workshop Proceedings, pages 1700-1711. CEUR-WS.org, 2021.
Jüri Keller and Leon Paul Mondrian Munz.
[doi] [pdf]  [BibTeX] 
Living Lab Evaluation for Life and Social Sciences Search Platforms - LiLAS at CLEF 2021.
In: D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast and F. Sebastiani, editors, Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II, volume 12657, series Lecture Notes in Computer Science, pages 657-664. Springer, 2021.
Philipp Schaer, Johann Schaible and Leyla Jael Garca Castro.
[doi] [pdf]  [BibTeX] 
Overview of LiLAS 2021 - Living Labs for Academic Search.
In: K. S. Candan, B. Ionescu, L. Goeuriot, B. Larsen, H. Müller, A. Joly, M. Maistro, F. Piroi, G. Faggioli and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021), volume 12880, series Lecture Notes in Computer Science. 2021.
Philipp Schaer, Timo Breuer, Leyla Jael Castro, Benjamin Wolff, Johann Schaible and Narges Tavakolpoursaleh.
[pdf]  [BibTeX] 
Overview of LiLAS 2021 - Living Labs for Academic Search (Extended Overview).
In: G. Faggioli, N. Ferro, A. Joly, M. Maistro and F. Piroi, editors, Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, series CEUR Workshop Proceedings. 2021.
Philipp Schaer, Timo Breuer, Leyla Jael Castro, Benjamin Wolff, Johann Schaible and Narges Tavakolpoursaleh.
[pdf]  [BibTeX] 

2020

How to Measure the Reproducibility of System-oriented IR Experiments.
In: J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J.-R. Wen and Y. Liu, editors, SIGIR, pages 349-358. ACM, 2020.
Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp Schaer and Ian Soboroff.
[doi] [pdf]  [BibTeX] 
Relations Between Relevance Assessments, Bibliometrics and Altmetrics.
In: Proceedings of the 10th International Workshop on Bibliometric-enhanced Information Retrieval co-located with 42nd European Conference on Information Retrieval, BIR@ECIR 2020, Lisbon, Portugal, April 14th, 2020 [online only], pages 101-112. 2020.
Timo Breuer, Philipp Schaer and Dirk Tunger.
[doi] [pdf]  [BibTeX] 
Reproducible Online Search Experiments.
In: Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II, pages 597-601. 2020.
Timo Breuer.
[doi] [pdf]  [BibTeX] 
Editorial.
Datenbank-Spektrum, 20(1):1-3, 2020.
Philipp Schaer, Klaus Berberich and Theo Härder.
[doi] [pdf]  [BibTeX] 
Living Labs for Academic Search at CLEF 2020.
In: J. M. Jose, E. Yilmaz, J. Magalhães, P. Castells, N. Ferro, Má. J. Silva and F. Martins, editors, Advances in Information Retrieval, pages 580-586. Springer International Publishing, Cham, 2020.
Philipp Schaer, Johann Schaible and Bernd Müller.
[pdf]  [abstract]  [BibTeX] 
Overview of LiLAS 2020 - Living Labs for Academic Search.
In: A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, H. Joho, C. Lioma, C. Eickhoff, A. Névéol, L. Cappellato and N. Ferro, editors, CLEF, volume 12260, series Lecture Notes in Computer Science, pages 364-371. Springer, 2020.
Philipp Schaer, Johann Schaible and Leyla Jael García Castro.
[doi] [pdf]  [BibTeX] 
Overview of LiLAS 2020 - Living Labs for Academic Search Workshop Lab (extended abstract).
In: L. Cappellato, C. Eickhoff, N. Ferro and A. Névéol, editors, CLEF (Working Notes), volume 2696, series CEUR Workshop Proceedings. CEUR-WS.org, 2020.
Philipp Schaer, Johann Schaible and Leyla Jael García-Castro.
[doi] [pdf]  [BibTeX] 
Evaluation Infrastructures for Academic Shared Tasks.
Datenbank-Spektrum, 20(1):29-36, 2020.
Johann Schaible, Timo Breuer, Narges Tavakolpoursaleh, Bernd Müller, Benjamin Wolff and Philipp Schaer.
[doi] [pdf]  [abstract]  [BibTeX] 

2019

Dockerizing Automatic Routing Runs for The Open-Source IR Replicability Challenge (OSIRRC 2019).
In: R. Clancy, N. Ferro, C. Hauff, J. Lin, T. Sakai and Z. Z. Wu, editors, OSIRRC@SIGIR, volume 2409, series CEUR Workshop Proceedings, pages 31-35. CEUR-WS.org, 2019.
Timo Breuer and Philipp Schaer.
[pdf]  [BibTeX] 
Replicability and Reproducibility of Automatic Routing Runs.
In: L. Cappellato, N. Ferro, D. E. Losada and H. Müller, editors, CLEF (Working Notes), volume 2380, series CEUR Workshop Proceedings. CEUR-WS.org, 2019.
Timo Breuer and Philipp Schaer.
[pdf]  [BibTeX] 
STELLA: Towards a Framework for the Reproducibility of Online Search Experiments.
In: R. Clancy, N. Ferro, C. Hauff, J. Lin, T. Sakai and Z. Z. Wu, editors, OSIRRC@SIGIR, volume 2409, series CEUR Workshop Proceedings, pages 8-11. CEUR-WS.org, 2019.
Timo Breuer, Philipp Schaer, Narges Tavakolpoursaleh, Johann Schaible, Benjamin Wolff and Bernd Müller.
[pdf]  [BibTeX]