Do German-trained LLMs track grammatical relationships?

Gert Webelhuth

Do LLMs trained on German build genuine hierarchical structure, or do they rely on local statistical shortcuts between nearby tokens? We test this across six phenomena — three morphosyntactic (subject-verb agreement, dative case marking, movement/gap-tracking) and three semantic (collectivity, animacy selectional restrictions, negative-polarity licensing)— measuring surprisal on confound-controlled minimal pairs while independently manipulating structural embedding depth and linear distance. Each phenomenon's stimuli were iteratively refined against lemma-frequency imbalances, filler repetition, and subword-tokenization artifacts, and evaluated across seven German-capable models spanning 66M to 8B parameters, covering both masked and causal architectures. Manipulating depth and distance independently within the same syntactic frame lets us ask whether any observed sensitivity tracks genuine hierarchical embedding or merely surface proximity between the two related expressions. Results are analyzed within a Bayesian mixed-effects framework and stress-tested with a battery of robustness checks — multiple-comparisons correction, a sensitivity analysis for the linearity assumption, expanded posterior predictive checks, and a systematic tokenization audit — before we draw any conclusions about what these models do and do not track.