Intersemiotic mismatch in memes: a study of machine translation output from English into Portuguese
DOI :
https://doi.org/10.5281/zenodo.8364507Résumé
This study presents findings on the use of Google Translator output for multimodal contexts. Development and evaluation of machine translation tend to focus on the linguistic component, while manual exploration of text-image relations in multimodal documents remains scarce. Therefore, this article aims at describing some text-image relationships in memes automatically translated from English into Portuguese. The methodology involves the selection and analysis of 100 memes found on Instagram and Facebook pages and their intersemiotic relationships both in English (as a source text) and in Portuguese (as a target text). Among the memes analyzed, 73% resulted in correct translations, 17% had errors with no intersemiotic mismatches, and only 10% showed linguistic deviations that altered text-image relationships for a meme. From these 10% of mismatches, patterns were identified, such as i) misspelled words with additive relations; and ii) unknown words with homospatiality. Finally, the results show that the automatic translation of some memes, whose semantic text-image relations share greater congruence, introduce more mismatches compared to those in which this does not happen.
Téléchargements
Références
BANITZ, B. Tradução automática: uma análise crítica do desempenho da tradução automática estatística e baseada em regras. Cadernos de Tradução, v. 40, n. 1, p. 54–71, 22 jan. 2020.
BATEMAN, J. A. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. New York: Palgrave MacMillan, 2008.
CAGLAYAN, O. et al. Does Multimodality Help Human and Machine for Translation and Image Captioning? Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, p. 627–633, 2016.
CAGLAYAN, O. Multimodal Machine Translation. Computation and Language [cs.CL]—[s.l.] Université du Maine, 2019.
CALIXTO, I.; LIU, Q. An error analysis for image-based multi-modal neural machine translation. Machine Translation, v. 33, n. 1, p. 155–177, 2019.
DORR, B. Solving thematic divergences in machine translation. Proceedings of the 28th Annual Meeting of the Association for Computational Linguistics. Pittsburgh, Pennsylvania, USA: Association for Computational Linguistics, jun. 1990. Disponível em: http://www.aclweb.org/anthology/P90-1017. Acesso em: 26 out. 2021.
DRIP. In: DICIONÁRIO Informal. [S.n.: S.l.], 2021. Disponível em: https://www.dicionarioinformal.com.br/drip/ . Acesso em: 25 maio 2021.
FARRÚS, M.; COSTA-JUSSÀ, M. R.; MORSE, M. P. Study and correlation analysis of linguistic, perceptual, and automatic machine translation evaluations. Journal of the American Society for Information Science and Technology, v. 63, n. 1, p. 174–184, jan. 2012.
HEO, Y.; KANG, S.; YOO, D. Multimodal Neural Machine Translation With Weakly Labeled Images. IEEE Access, v. 7, p. 54042–54053, 2019.
HIRASAWA, T. et al. Multimodal Machine Translation with Embedding Prediction. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. Anais...Minneapolis, Minnesota: Association for Computational Linguistics, jun. 2019. Disponível em: <https://www.aclweb.org/anthology/N19-3012>. Acesso em: 13 abr. 2020
JURAFSKY, D.; MARTIN, J. H. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. 3rd. ed. Upper Saddle River, N.J.: Prentice Hall., 2021.
KAMEYAMA, M.; OCHITANI, R.; PETERS, S. Resolving Translation Mismatches With Information Flow. Proceedings of the Annual Meeting of the Association for Computational Linguistics ACL91. 1991.
KRESS, G.; VAN LEEUWEN, T. Reading images: the grammar of visual design. 2. ed. London: Routledge, 2006.
LIU, Y.; O’HALLORAN, K. L. Intersemiotic texture: analyzing cohesive devices between language and images. Social Semiotics, Abingdon, v. 19, n. 4, p. 1-40, 2009.
PAPADOPOULOU, A. Automatic Error Detection and Correction in Neural Machine Translation : A comparative study of Swedish to English and Greek to English. Master’s Thesis—[s.l.] Uppsala University, Department of Linguistics and Philology, 2019.
PIRES, T. B.; ESPINDOLA, A. V S. Aproximando resultados de tradução automática e imagens em documentos multimodais. Cadernos de Tradução, Florianópolis, v. 41, n.2, p.85-108, 2021. Disponível em: https://periodicos.ufsc.br/index.php/traducao/article/view/75483/46513. Acesso em: 24 ago. 2021.
PIRES, T. B. Ampliando olhares sobre a tradução automática online: um estudo exploratório de categorias de erros de máquina de tradução gerados em documentos multimodais. 2017. Tese (Doutorado em Ciência da Informação) – Universidade de Brasília, Brasília.
QUAH, C. K. Translation and technology. Houndmills [England] ; New York: Palgrave Macmillan, 2006.
Téléchargements
Publiée
Comment citer
Rubrique
Licence
(c) Tous droits réservés Revista Letras Raras 2023
Ce travail est disponible sous licence Creative Commons Attribution - Pas d’Utilisation Commerciale 4.0 International.
