Towards a Grammatical Description Native to LLMs

Jingcheng Niu

Large language models (LLMs) now serve as the de facto interface to language technology, and, admittedly, are the closest thing to a coherent linguistic system we have ever built. Yet we still understand remarkably little about where this competence comes from: neither how it is implemented mechanistically, nor where in the pre-training process it arises. The dominant response has been to reach for the structures we already know: to take the categories developed for human language and cognition and go fishing for them in a model's outputs and internal activations. This rarely works well, and a growing body of evidence, I will argue, points to a deeper reason: LLMs appear to implement a fundamentally different system, one that need not organise language the way we do. The categories we already have therefore have little reason to fit it, no matter how hard we look. Instead, I propose that we should create a grammar native to the model: an LLM-Native Grammar, a descriptive framework grounded in what the model actually does, rather than in the pseudo-linguistic or cognitive scaffolding inherited from the study of human language.In this talk, I will walk through several directions that work towards such an LLM native grammar. I start with representational geometry: what LLMs' activation spaces look like, and how information is organised within them to compute the model's outputs. I then move to component analysis, working out the role each neuron, attention head, and MLP plays. Circuit and sheaf discovery then shows how these components connect and interact. Together, these investigations can give us a much better understanding of LLMs, one that lets us more directly control and steer them. And studying this second kind of linguistic system may, in turn, teach us something about our own.