Publications

2026

A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

Raghu Arghal, Fade Chen, Niall Dalton, Evgenii Kortukov, Calum McNamara, Angelos Nalmpantis, Moksh Nirvaan, Gabriele Sarti, Mario Giulianelli

Preprint

Reasoning aligns language models to human cognition

Goncalo Guiomar, Elia Torre, Pehuen Moure, Victoria Shavina, Mario Giulianelli, Shih-Chii Liu, Valerio Mante

Preprint

Incremental Alternative Sampling as a Lens into the Temporal and Representational Resolution of Linguistic Prediction

Mario Giulianelli, Sarenne Wallbridge, Ryan Cotterell, Raquel Fernandez

Journal of Memory and Language

Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?

Matteo Gay, Coleman Haley, Mario Giulianelli, Edoardo Ponti

Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026)

Diachronic Lexical Semantics with Computational Models

Nina Tahmasebi, Andrey Kutuzov, Haim Dubossarsky, Mario Giulianelli

The Wiley Blackwell Companion to Diachronic Linguistics. John Wiley & Sons

2025

Skewed Score: A statistical framework to assess autograders

Magda Dubois, Harry Coppock, Mario Giulianelli, Timo Flesch, Lennart Luettgau, Cozmin Ududec

Preprint

Evaluating LLMs in Games with Strategic Behavior Discrepancies

Tianyu Liu, Francesco Ignazio Re, Mario Giulianelli, Alessandro Torcinovich

The 5th Wordplay Workshop: When Language Meets Games @ EMNLP 2025

Structure-Conditional Minimum Bayes Risk Decoding

Bryan Eikema, Anna Rutkiewicz, Mario Giulianelli

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)

Lessons from a Chimp: AI "Scheming" and the Quest for Ape Language

Christopher Summerfield, Lennart Luettgau, Magda Dubois, Hannah Rose Kirk, Kobi Hackenburg, Catherine Fist, Katarina Slama, Nicola Ding, Rebecca Anselmetti, Andrew Strait, Mario Giulianelli, Cozmin Ududec

Preprint

Establishing Best Practices for Building Rigorous Agentic Benchmarks

Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellerman, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, Daniel Kang

NeurIPS 2025 Datasets and Benchmarks

Playpen: An Environment for Exploring Learning Through Conversational Interaction

Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momente, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernandez, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)

Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests

Filippo Momente, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernandez, Raffaella Bernardi

Findings of the Association for Computational Linguistics: EMNLP 2025

The Harmonic Structure of Information Contours

Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Information Locality as an Inductive Bias for Neural Language Models

Taiga Someya, Anej Svete, Brian DuSell, Timothy J. O'Donnell, Mario Giulianelli, Ryan Cotterell

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

🏆 Panel (top 0.8%)

A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior

Francesco Ignazio Re, Andreas Opedal, Glib Manaiev, Mario Giulianelli, Ryan Cotterell

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernandez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre FT Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

From Language Models over Tokens to Language Models over Characters

Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, Ryan Cotterell

Proceedings of the 42nd International Conference on Machine Learning (ICML 2025)

🏆 Spotlight (top 2.6%)

Language Models over Canonical Byte-Pair Encodings

Tim Vieira, Tianyu Liu, Clemente Pasti, Yahya Emara, Brian DuSell, Benjamin LeBrun, Mario Giulianelli, Juan Luis Gastaldi, John Terilla, Timothy O'Donnell, Ryan Cotterell

Proceedings of the 42nd International Conference on Machine Learning (ICML 2025)

2024

On the Proper Treatment of Tokenization in Psycholinguistics

Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, Ryan Cotterell

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024)

Generalized Measures of Anticipation and Responsivity in Online Language Processing

Mario Giulianelli, Andreas Opedal, Ryan Cotterell

Findings of the Association for Computational Linguistics: EMNLP 2024

Clara Meister, Mario Giulianelli, Tiago Pimentel

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024)

Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse

Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, Alex Warstadt

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024)

Efficiency and Effectiveness in Task-Oriented Dialogue: On Construction Repetition, Information Rate, and Task Success

Jun Sen Yee, Mario Giulianelli, Arabella Sinclair

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models

Ivar Frisch and Mario Giulianelli

Proceedings of the 1st Personalization of Generative AI Workshop (EACL)

Beyond Perplexity: Examining Temporal Generalization in Large Language Models via Definition Generation

Iris Luden, Mario Giulianelli, and Raquel Fernandez

Computational Linguistics in the Netherlands Journal 13 (CLIN)

2023

A taxonomy and review of generalization research in NLP

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, et al.

Nature Machine Intelligence 5, 1161-1174

Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives

Mario Giulianelli, Sarenne Wallbridge, Raquel Fernandez

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernandez, Barbara Plank

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)

Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue

Aron Molnar, Jaap Jumelet, Mario Giulianelli, Arabella Sinclair

Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL 2023)

Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis

Mario Giulianelli, Iris Luden, Raquel Fernandez, Andrey Kutuzov

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)

Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind

Ece Takmaz, Nicolo Brandizzi, Mario Giulianelli, Sandro Pezzelle, Raquel Fernandez

Findings of the 61st Annual Meeting of the Association for Computational Linguistics (Findings of ACL 2023)

2022

Towards Pragmatic Production Strategies for Natural Language Generation

Mario Giulianelli

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)

Construction Repetition Reduces Information Rate in Dialogue

Mario Giulianelli, Arabella Sinclair, Raquel Fernandez

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP 2022)

Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change

Mario Giulianelli, Andrey Kutuzov, Lidia Pivovarova

Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change

AnaLog: Testing Analytical and Deductive Logic Learnability in Language Models

Samuel Ryb, Mario Giulianelli, Arabella Sinclair, Raquel Fernandez

Proceedings of *SEM 2022: The 11th Joint Conference on Lexical and Computational Semantics

Dataset

Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models

With many collaborators :)

Transactions of Machine Learning Research

2021

Is Information Density Uniform in Task-Oriented Dialogues?

Mario Giulianelli, Arabella Sinclair, Raquel Fernandez

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)

Code

Analysing Human Strategies of Information Transmission as a Function of Discourse Context

Mario Giulianelli and Raquel Fernandez

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL 2021)

Code

Grammatical Profiling for Semantic Change Detection

Mario Giulianelli, Andrey Kutuzov, Lidia Pivovarova

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL 2021)

Code

2020

Analysing Lexical Semantic Change with Contextualised Word Representations

Mario Giulianelli, Marco Del Tredici, Raquel Fernandez

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)

Code

UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection

Andrey Kutuzov and Mario Giulianelli

Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval 2020)

Code

Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts

Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, Raquel Fernandez

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)

Code

2018

Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information

Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, Willem Zuidema

1st Workshop on Analyzing and Interpreting Neural Networks for NLP (EMNLP 2018)

🏆 Best Paper Award

Code

Semi-supervised emotion lexicon expansion with label propagation

Mario Giulianelli and Daniel de Kok

Computational Linguistics in the Netherlands Journal 8 (CLIN)

Code

Dissertations

Neural Models of Language Use

Mario Giulianelli

PhD thesis · 2023

Lexical Semantic Change Analysis with Contextualised Word Representations

Mario Giulianelli

Master's thesis · 2019

Semi-supervised emotion lexicon expansion with label propagation and specialized word embeddings

Mario Giulianelli

Bachelor's thesis · 2017