mALBERT: Is a Compact Multilingual BERT Model Still Worth It? - Laboratoire Interdisciplinaire des Sciences du Numérique Access content directly
Conference Papers Year : 2024

mALBERT: Is a Compact Multilingual BERT Model Still Worth It?


Within the current trend of Pretained Language Models (PLM), emerge more and more criticisms about the ethical and ecological impact of such models. In this article, considering these critical remarks, we propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLM. However, PLMs enable huge breakthroughs in Natural Language Processing tasks, such as Spoken and Natural Language Understanding, classification, Question–Answering tasks. PLMs also have the advantage of being multilingual, and, as far as we know, a multilingual version of compact ALBERT models does not exist. Considering these facts, we propose the free release of the first version of a multilingual compact ALBERT model, pre-trained using Wikipedia data, which complies with the ethical aspect of such a language model. We also evaluate the model against classical multilingual PLMs in classical NLP tasks. Finally, this paper proposes a rare study on the subword tokenization impact on language performances.
Fichier principal
Vignette du fichier
mALBERT_final.pdf (231.91 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04520797 , version 1 (26-03-2024)


Public Domain


  • HAL Id : hal-04520797 , version 1


Christophe Servan, Sahar Ghannay, Sophie Rosset. mALBERT: Is a Compact Multilingual BERT Model Still Worth It?. The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, May 2024, Torino, Italy. ⟨hal-04520797⟩
17 View
8 Download


Gmail Facebook X LinkedIn More