I am an Associate Professor of Computational Linguistics at UCL. Prior to this, I was a senior research scientist at the UK AI Security Institute, working on the (nascent) science of AI evaluation, and a postdoctoral fellow at ETH Zurich, working with Ryan Cotterell in the Institute for Machine Learning, Department of Computer Science. I did my PhD at the Institute for Logic, Language and Computation of the University of Amsterdam, where I was advised by Raquel Fernández. I am an associated researcher at the ETH AI Center and a member of the ELLIS Society.

My research explores the computational principles underlying the ability to understand, produce, learn, and use language in interaction—both in natural and in artificial cognitive systems. I’m also deeply committed to improving evaluation standards for AI systems, in the domain of language and across the broader AI landscape. In 2018, I co-authored a paper that, according to the very kind Aaron Mueller, introduced the first mechanistic interpretability method for language models—apparently before it was cool.

News

• I received a Cohere Labs Catalyst Grant to support our SPAR experiments on measuring goal-directedness in LLM agents.
• I received a Cosmos Grant to study emergent self-interest in LLM agents using non-cooperative dialogue games.
• I will be a SPAR mentor this Fall. I am very thankful for the many applications I received to work with me on modelling, measuring, and intervening on goal-directed behaviour. Applications are now closed.
• Check out these AISI blogposts on capability elicitation, LLM judges, and agentic testing methodology (I had the privilege of leading the analysis for this international joint testing exercise, which we conducted in collaboration with other AISIs around the world)
• In September 2025, I will join UCL as an Associate Professor of Computational Linguistics.
• Our paper with exact and (fast) approximate algorithms for converting token-level language models to character-level ones is an ICML Spotlight (top 2.6%)!
• Our paper on inductive biases in human and neural language models was selected for an ACL panel discussion (top 0.8%)!
• Four papers accepted at ACL and two at ICML! More info in the publication section below.
• I’ve joined the UK AI Security Institute in London! I work across risk domains and closely with the Science of Evaluations team.
• I’m an organizer of the 2nd edition of the GenBench Workshop. See you in Miami on 16th November!
• Very excited to have been awarded a competitive postdoctoral fellowship at ETH Zürich, where I will be hosted by the Rycolab!
• I am honored to be nominated as a Member of the ELLIS Society.
• I kind of started a blog and my first blog post is out :)
• I am giving a keynote at the EMNLP Workshop on Computational Approaches to Historical Language Change (LChange’23)
• Our GenBench taxonomy is published in Nature Machine Intelligence: A taxonomy and review of generalization research in NLP

Working with me

Before e-mailing me, please read the following!

• I am hiring a PhD student! This is a fully funded 4-year PhD position in the UCL Division of Psychology and Language Sciences. Only candidates eligible for UK fee status can apply. More info: tinyurl.com/phd-ucl-clnlpai
• Below is a list of alternative funding schemes and programmes. If you intend to apply for one of these opportunities, for another programme, or if you have secured your own funding, you are welcome to get in touch. I am happy to discuss and support applications that align with my research. Please indicate clearly in your message the area you would like to work on, and include a brief research proposal). General inquiries are extremely unlikely to receive a response.
• UCL Research Excellence Scholarship
• UCL Research Opportunity Scholarship
• UBEL DTP’s ESRC Studentship
• LAHP Open Studentship
• UCL Language and Cognition MPhil/PhD
• UCL Linguistics MPhil/PhD
• UCL Foundational Artificial Intelligence MPhil/PhD
• Gatsby Unit PhD Programme
• Information page on studentships

Invited talks and guest lectures

• Linguistics Research Seminar, University of Geneva. 21 October 2025.
• Human-centred AI Research Network, University of Aberdeen. 12 August 2025.
• Deep Linguistic Modeling Colloquium, Heinrich-Heine Universität Düsseldorf. 17 January 2025.
• Language Processing Group, UC Irvine Department of Language Science. 3 December 2024.
• CLColloquium, UT Austin. 18 November 2024.
• Workshop on NLP and Multi-Modality, University of Gothenburg. 10 June 2024.
• Guest lecture, Seminar on Detection of Semantic Shift, University of Zürich. 21 March 2024.
• Text Technology & Digital Linguistics colloquium, University of Zürich. 5 March 2024.
• Department of Computer Science and Computational Linguistics, Saarland University, Germany. 26 February 2024.
• Keynote at the EMNLP Workshop on Computational Approaches to Historical Language Change (LChange’23), Singapore. 6 December 2023.
• Reatch nanoTalks. Zürich, Switzerland. 26 November 2023.
• Natural Language Processing Seminar. University of Groningen. 3 November.
• Explainable Machine Learning Lab, University of Tübingen. 26 July 2023.
• Cognitive Lexicon Laboratory, University of Toronto. Canada. 13 July 2023.
• COLT Seminar, Universitat Pompeu Fabra. Barcelona, Spain. 29 June 2023.
• ILCC Seminar, School of Informatics, University of Edinburgh. UK. 31 May 2023.
• Rycolab, ETH Zürich. Switzerland. 3 May 2023.
• Inria & Loria. 27 October 2022. Nancy, France. [Abstract]
• School of Natural and Computing Sciences, University of Aberdeen. UK. 12 October 2022. [Abstract]
• Guest lecture, Advanced Information Retrieval (MSc AI and Informatics). University of Amsterdam. Netherlands. 21 September 2022. [Slides]
• Language Technology Group Seminar. Language Technology Group. University of Oslo, Norway. 14 March 2022.
• NLPitch. Institute for Logic, Language and Computation. University of Amsterdam. Netherlands. 26 October 2020.
• AILC Lectures on Computational Linguistics. Università Tor Vergata. Rome, Italy. 17 June 2021.
• Cognitive Machine Learning Lab. Ecole Normale Supérieure. Paris, France. 29 June 2021.
• Symposium on Meaning Variation in Social Contexts. Institute for Logic, Language and Computation. Amsterdam, NL. 5 November 2020.
• Computational Linguistic Seminar. Institute for Logic, Language and Computation. University of Amsterdam, Netherlands. 5 October 2021.
• Guest lecture, NLP 1 (MSc AI course). University of Amsterdam, Netherlands. Online. 18 November 2020.
• Cognitive Science & Artificial Intelligence PhD Group. Tilburg, Netherlands. 1 May 2020.
• Cool Logic Seminar. Amsterdam, Netherlands. 1 February 2019.

Work in progress

• [PDF] Christopher Summerfield, Lennart Luettgau, Magda Dubois, Hannah Rose Kirk, Kobi Hackenburg, Catherine Fist, Katarina Slama, Nicola Ding, Rebecca Anselmetti, Andrew Strait, Mario Giulianelli, Cozmin Ududec. Lessons from a Chimp: AI “Scheming” and the Quest for Ape Language. Preprint.
• [PDF] Magda Dubois, Harry Coppock, Mario Giulianelli, Timo Flesch, Lennart Luettgau, Cozmin Ududec. Skewed Score: A statistical framework to assess autograders. Preprint.
• [PDF] Mario Giulianelli, Sarenne Wallbridge, Ryan Cotterell, Raquel Fernández. Incremental Alternative Sampling as a Lens into the Temporal and Representational Resolution of Linguistic Prediction. Preprint.
• Tianyu Liu, Francesco Ignazio Re, Mario Giulianelli, Alessandro Torcinovich. Evaluating LLMs in Games with Strategic Behavior Discrepancies. The 5th Wordplay Workshop: When Language Meets Games @ EMNLP 2025.

Publications

• [PDF] Bryan Eikema, Anna Rutkiewicz, Mario Giulianelli. Structure-Conditional Minimum Bayes Risk Decoding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025).
• [PDF] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellerman, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, Daniel Kang. Establishing Best Practices for Building Rigorous Agentic Benchmarks NeurIPS 2025 Datasets and Benchmarks.
• [PDF] Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia. Playpen: An Environment for Exploring Learning Through Conversational Interaction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025).
• [PDF] Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, Raffaella Bernardi. Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests. In Findings of the Association for Computational Linguistics: EMNLP 2025.
• [PDF] Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli. The Harmonic Structure of Information Contours. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025).
• [PDF] Taiga Someya, Anej Svete, Brian DuSell, Timothy J. O’Donnell, Mario Giulianelli, Ryan Cotterell. Information Locality as an Inductive Bias for Neural Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). Panel (top 0.8%).
• [PDF] Francesco Ignazio Re, Andreas Opedal, Glib Manaiev, Mario Giulianelli, Ryan Cotterell. A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025).
• [PDF] Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André FT Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni. 2025. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025).
• [PDF] Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O’Donnell, Ryan Cotterell. From Language Models over Tokens to Language Models over Characters. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). Spotlight (top 2.6%).
• [PDF] Tim Vieira, Tianyu Liu, Clemente Pasti, Yahya Emara, Brian DuSell, Benjamin LeBrun, Mario Giulianelli, Juan Luis Gastaldi, John Terilla, Timothy O’Donnell, Ryan Cotterell. Language Models over Canonical Byte-Pair Encodings. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).
• [PDF] Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, Ryan Cotterell. 2024. On the Proper Treatment of Tokenization in Psycholinguistics. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).
• [PDF] Mario Giulianelli, Andreas Opedal, Ryan Cotterell. 2024. Generalized Measures of Anticipation and Responsivity in Online Language Processing. In Findings of the Association for Computational Linguistics: EMNLP 2024.
• [PDF] Clara Meister, Mario Giulianelli, Tiago Pimentel. 2024. Towards a Similarity-aware Surprisal Theory. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).
• [PDF] Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, Alex Warstadt. Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).
• [PDF] Jun Sen Yee, Mario Giulianelli, and Arabella Sinclair. 2024. Efficiency and Effectiveness in Task‐Oriented Dialogue: On Construction Repetition, Information Rate, and Task Success. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC‐COLING 2024).
• [PDF] Ivar Frisch and Mario Giulianelli. 2024. LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models In Proceedings of the 1st Personalization of Generative AI Workshop (EACL).
• [PDF] Iris Luden, Mario Giulianelli, and Raquel Fernández. 2024. Beyond Perplexity: Examining Temporal Generalization in Large Language Models via Definition Generation. Computational Linguistics in the Netherlands Journal 13 (CLIN).
• [PDF] Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, et al. 2023. A taxonomy and review of generalization research in NLP. Nature Machine Intelligence 5, 1161–1174.
• [PDF] Mario Giulianelli, Sarenne Wallbridge, Raquel Fernández. 2023. Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023).
• [PDF] Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, Barbara Plank. 2023. What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023).
• [PDF] Aron Molnar, Jaap Jumelet, Mario Giulianelli, Arabella Sinclair. 2023. Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue. In Proceedings of the 27th Conference on Computational Natural Language Learning (CONLL 2023).
• [PDF] Mario Giulianelli, Iris Luden, Raquel Fernández, Andrey Kutuzov. 2023. Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).
• [PDF] Ece Takmaz, Nicolò Brandizzi, Mario Giulianelli, Sandro Pezzelle, Raquel Fernández. 2023. Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind. In Findings of the 61st Annual Meeting of the Association for Computational Linguistics (Findings of ACL 2023).
• [PDF] Mario Giulianelli. 2022. Towards Pragmatic Production Strategies for Natural Language Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)
• [PDF] Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artexte, Tiago Pimentel, et al. 2022. State-of-the-art generalisation research in NLP: A taxonomy and review. Preprint.
• [PDF] Mario Giulianelli, Arabella Sinclair, Raquel Fernández. 2022. Construction Repetition Reduces Information Rate in Dialogue. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP 2022).
• [PDF] Mario Giulianelli, Andrey Kutuzov, Lidia Pivovarova. 2022. Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change.
• [PDF][Dataset] Samuel Ryb, Mario Giulianelli, Arabella Sinclair, Raquel Fernández. 2022. AnaLog: Testing Analytical and Deductive Logic Learnability in Language Models. In Proceedings of *SEM 2022: The 11th Joint Conference on Lexical and Computational Semantics.
• [PDF] With many collaborators :). 2022. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. In Transactions of Machine Learning Research.
• [PDF][Code] Mario Giulianelli, Arabella Sinclair, Raquel Fernández. 2021. Is Information Density Uniform in Task-Oriented Dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021).
• [PDF][Code] Mario Giulianelli and Raquel Fernández. 2021. Analysing Human Strategies of Information Transmission as a Function of Discourse Context. In Proceedings of the 25th Conference on Computational Natural Language Learning (CONLL 2021).
• [PDF][Code] Mario Giulianelli, Andrey Kutuzov, Lidia Pivovarova. 2021. Grammatical Profiling for Semantic Change Detection. In Proceedings of the 25th Conference on Computational Natural Language Learning (CONLL 2021).
• [PDF][Code] Mario Giulianelli, Marco Del Tredici, Raquel Fernández. 2020. Analysing Lexical Semantic Change with Contextualised Word Representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020).
• [PDF][Code] Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. In the Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval 2020).
• [PDF][Code] Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, Raquel Fernández. 2020. Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020).
• [PDF][Code] Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, Willem Zuidema. 2018. Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information. Best Paper Award at 1st Workshop on Analyzing and Interpreting Neural Networks for NLP (EMNLP 2018).
• [PDF][Code] Mario Giulianelli and Daniel de Kok. 2018. Semi-supervised emotion lexicon expansion with label propagation. Computational Linguistics in the Netherlands Journal 8 (CLIN).

Theses

• [PDF] Neural Models of Language Use. 2023. PhD thesis.
• [PDF] Lexical Semantic Change Analysis with Contextualised Word Representations. 2019. Master’s thesis.
• [PDF] Semi-supervised emotion lexicon expansion with label propagation and specialized word embeddings. 2017. Bachelor’s thesis.

---