One Confidence Score, Two Languages

A thing that is easy to miss until it bites you: a model's confidence is not calibrated the same way across languages.

You tune a threshold on your primary language, watch it behave, and ship it as a global setting. It looks safe because the language you tested it in is safe. But the same score does not carry the same meaning in your second language, where the model has usually seen less data, handles dialect and script variation worse, and is more willing to sound fluent while being wrong. The number reads identical. The real risk behind it does not.

I work on bilingual English and Arabic systems, and this shows up constantly. A confidence cutoff that is genuinely conservative in English can be optimistic in Arabic, which means the language you understand least well is the one where the model is allowed to act most freely. That is the opposite of what you want.

The fix is not a cleverer global number. It is to stop treating confidence as one currency. Calibrate per language, per channel, per task, from real evaluation data, and let each context carry its own threshold. Until you have that data, assume your second language needs a tighter gate, not the same one. A single threshold is a convenient fiction, and it fails quietest exactly where you can check it least.