Informed Prompts and Improving ChatGPT English to Arabic Translation

Abstract

The aim of the study is to investigate the quality of ChatGPT translation and the effectiveness of using informed prompts to improve it. The research team built a dataset composed of various English complex sentence types (150complex sentences) that are selected from various news sites. The sentences were translated into Arabic using a default ChatGPT translation prompt (Translate the following sentences into Arabic). The translated sentences were annotated by three professional annotators. An error taxonomy was performed based on the Multidimensional Quality Metrics (MQM). The results of the error taxonomy shows a high error frequency that amounts to 2.73 errors per sentence which indicates that ChatGPT falls short when translating English complex sentences into Arabic and that it still needs to be trained effectively. The sentences whose translation outputs have the most errors are translated again using informed prompts that require the model to correct the original translation. Both the original and the new translation outputs are evaluated manually by the professional annotators and automatically using the BLEU metric. The study, therefore, identifies the effectiveness of the adopted prompt strategies in improving translation quality and will recommend further research in the area of informed prompts.