Xiangming Gu

I am a final-year Ph.D. candidate from National University of Singapore. I received my bachelor's degrees from Tsinghua University in 2021. I was a student researcher at Google Deepmind and a research intern at Sea AI Lab.

My most representative work is to thoroughly demystify attention sink phenomenon [1, 2] and promote the architecture modification, such as attention biases, to facilitate long-context, and training stability [post1, 2, 3].

I have research experiences on different modality data, including text, vision, and audio. My current research interests include pre-training, architecture design, and reasoning for generative models from first principles.

Email  /  Google Scholar  /  Openreview  /  Linkedin  /  Twitter  /  Github

• [2026.03]: I was invited to give two talks about Demystifying Attention Sink in LLMs and its Applications to Architecture Design by National University of Singapore SIAM, NextGen Data 2026.
• [2026.01]: I released gemma_penzai, a JAX package to look into LLM internals and debug LLMs with multi-modal support. This work was one of my projects at Google Deepmind. See the tutorials and post.
• [2026.01]: I was invited to give a talk about Demystifying Attention Sink in LLMs and its Applications to Architecture Design by AER Labs at Network School.
• [2025.11]: I was invited to give a talk about Attention Sink in LLMs and its Applications by Department of Electronic Engineering, Tsinghua University and Tencent Hunyuan.
• [2025.10]: I was glad to give a final presentation to wrap up my student researcher at Google Deepmind: Looking into LLMs: From Tokens to Solutions.
• [2025.10]: We released a technical report titled Extracting Alignment Data in Open Models!
• [2025.09]: One paper got accepted to Advances in Neural Information Processing Systems (NeurIPS'2025)!
• [2025.07]: One paper got accepted to Conference on Language Modeling (COLM'2025)!
• [2025.06]: I was glad to give a talk about Understanding Attention Sink in (Large) Language Models in the team of Deep Learning: Agent Frontier at Google Deepmind.
• [2025.05]: I was invited to give a talk on When Attention Sink Emerges in Language Models: An Empirical View by ASAP Seminar Series!
• [2025.05]: I joined Google Deepmind (GDM) as a student researcher in London, United Kingdom!
• [2025.02]: I was invited to give a talk titled On the Interpretability and Safety of Generative Models by research week open house of NUS!
• [2025.01]: Two papers with one spotlight and one poster got accepted to International Conference on Learning Representations (ICLR'2025), one paper got accepted to Transactions on Machine Learning Research (TMLR'2025)!
• [2025.01]: I gave a poster presentation about Agent Smith during Global Young Scientists Summit 2025!

You've probably seen this website template before, thanks to Jon Barron.