The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Электронный научный архив УРФУ
Информация об архиве | Просмотр оригинала| Поле | Значение | |
| Заглавие |
The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
|
|
| Автор |
Efimov, P.
Boytsov, L. Arslanova, E. Braslavski, P. |
|
| Тематика |
CROSS-LINGUAL TRANSFER
MULTILINGUAL EMBEDDINGS NATURAL LANGUAGE PROCESSING SYSTEMS ZERO-SHOT LEARNING CONTEXTUAL WORDS CONTINUAL LEARNING CROSS-LINGUAL CROSS-LINGUAL TRANSFER EMBEDDINGS LANGUAGE MODEL MULTILINGUAL EMBEDDING PARALLEL CORPORA PERFORMANCE WORD REPRESENTATIONS EMBEDDINGS |
|
| Описание |
Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. [8] proposed a data- and compute-efficient method for cross-lingual adjustment of mBERT that uses a small parallel corpus to make embeddings of related words across languages similar to each other. They showed it to be effective in NLI for five European languages. In contrast we experiment with a topologically diverse set of languages (Spanish, Russian, Vietnamese, and Hindi) and extend their original implementations to new tasks (XSR, NER, and QA) and an additional training regime (continual learning). Our study reproduced gains in NLI for four languages, showed improved NER, XSR, and cross-lingual QA results in three languages (though some cross-lingual QA gains were not statistically significant), while mono-lingual QA performance never improved and sometimes degraded. Analysis of distances between contextualized embeddings of related and unrelated words (across languages) showed that fine-tuning leads to “forgetting” some of the cross-lingual alignment information. Based on this observation, we further improved NLI performance using continual learning. Our software is publicly available https://github.com/pefimov/cross-lingual-adjustment. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.
Russian Science Foundation, RSF: 20-11-20166 Acknowledgment. This research was supported in part through computational resources of HPC facilities at HSE University [27]. PE is grateful to Yandex Cloud for their grant toward computing resources of Yandex DataSphere. PB acknowledges support by the Russian Science Foundation, grant No 20-11-20166. |
|
| Дата |
2024-04-05T16:18:01Z
2024-04-05T16:18:01Z 2023 |
|
| Тип |
Conference Paper
Conference object (info:eu-repo/semantics/conferenceObject) info:eu-repo/semantics/submittedVersion |
|
| Идентификатор |
Efimov, P, Boytsov, L, Arslanova, E & Braslavski, P 2023, The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer: book chapter. в J Kamps & L Goeuriot (ред.), Advances in Information Retrieval: 45th European Conference on Information Retrieval: book. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Том. 13982, Springer Cham, стр. 51-67. https://doi.org/10.1007/978-3-031-28241-6_4
Efimov, P., Boytsov, L., Arslanova, E., & Braslavski, P. (2023). The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer: book chapter. в J. Kamps, & L. Goeuriot (Ред.), Advances in Information Retrieval: 45th European Conference on Information Retrieval: book (стр. 51-67). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Том 13982). Springer Cham. https://doi.org/10.1007/978-3-031-28241-6_4 9783031282409 0302-9743 Final All Open Access, Green https://www.scopus.com/inward/record.uri?eid=2-s2.0-85151051828&doi=10.1007%2f978-3-031-28241-6_4&partnerID=40&md5=6cf730ccd4a5490b5051ddd822241e89 https://arxiv.org/pdf/2204.06457 http://elar.urfu.ru/handle/10995/130299 10.1007/978-3-031-28241-6_4 85151051828 000995495200004 |
|
| Язык |
en
|
|
| Связанные ресурсы |
info:eu-repo/grantAgreement/RSF//20-11-20166
|
|
| Права |
Open access (info:eu-repo/semantics/openAccess)
|
|
| Формат |
application/pdf
|
|
| Издатель |
Springer Science and Business Media Deutschland GmbH
|
|
| Источник |
Advances in Information Retrieval
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|