ITALERT: Assessing the quality of LLMs and NMT in translating Italian emergency response text

This paper presents the outcomes of an initial investigation into the performance of Large Language Models (LLMs) and Neural Machine Translation (NMT) systems in translating high-stakes messages. The research employed a novel bilingual corpus, ITALERT (Italian Emergency Response Text) and applied a human-centric post-editing based metric (HOPE) to assess translation quality systematically. The initial dataset contains eleven texts in Italian and their corresponding English translations, both extracted from the national communication campaign website of the Italian Civil Protection Department. The texts deal with eight crisis scenarios: flooding, earthquake, forest fire, volcanic eruption, tsunami, industrial accident, nuclear risk, and dam failure. The dataset has been carefully compiled to ensure usability and clarity for evaluating machine translation (MT) systems in crisis settings. 

The findings show that current LLMs and NMT models, such as ChatGPT (OpenAI’s GPT-4o model) and Google MT, face limitations in translating emergency texts, particularly in maintaining the appropriate register, resolving context ambiguities, and managing domain-specific terminology. Errors from each system were categorised using the default 7 error types (merged from 8) from the HOPE metric, with a revised severity mapping, adjusted to account for the sensitivity of the crisis domain. The findings reveal that both systems share common error types but differ in their rankings. ChatGPT showed a high incidence of Style and Terminology errors, while Google MT was characterised by a greater presence of Mistranslation, Impact, Terminology, and Style issues. Importantly, both systems produced a non-negligible amount of severe and major errors, despite the predominance of minor and medium-level issues.