Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese

2026-08-02

AI Summary

This study addresses the potential for large language models (LLMs) to reproduce or even distort human biases when evaluating non-native Japanese writing in high-stakes contexts. Extending language attitude theory beyond English-dominant settings for the first time, the research systematically audits LLM bias by comparing human raters’ assessments with those of six LLMs across three dimensions—fluency, status, and solidarity—using parallel Japanese email corpora. Findings reveal that LLMs generally replicate the direction and ranking of human evaluators’ negative judgments toward non-native speakers, yet attenuate perceived solidarity gaps and erroneously introduce distinctions based on learners’ native-language backgrounds that human raters do not exhibit. These results highlight critical limitations and socially biased tendencies in LLMs’ sociolinguistic evaluations.

Abstract

Large language models (LLMs) increasingly evaluate human writing in high-stakes domains such as hiring and academic assessment, putting non-native speakers at particular risk. Drawing on the language attitudes framework, we compared human and LLM evaluations of parallel L1- and L2-written Japanese emails on three dimensions: fluency, status, and solidarity. Japanese raters rated L2 texts significantly lower on all three dimensions, with a fluency gap roughly twice the size of the status and solidarity gaps. Six LLM judges reproduced the direction of this bias, and five reproduced its ordering across dimensions. The models diverged from humans in two ways: all understated the solidarity gap, the most socially grounded dimension, and all differentiated among learner L1 backgrounds where humans did not. LLM judges thus reproduce native speakers' language attitudes in a structured yet attenuated form, and the language attitudes framework offers a ready-made yardstick for auditing them beyond English.

Problem

• Human-LLM alignment
• language attitudes
• non-native speakers
• bias
• Japanese

Innovation

• language attitudes
• human-LLM alignment
• non-native Japanese
• bias auditing
• cross-lingual evaluation