How Many Layers and Why? An Analysis of the Model Depth in Transformers

TitreHow Many Layers and Why? An Analysis of the Model Depth in Transformers
Publication TypeArticle dans des actes
Année de la conférence2021
AuthorsSimoulin, Antoine, and Benoît Crabbé
Nom de la conférenceProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
Pagination221–228
PublisherAssociation for Computational Linguistics
Conference LocationOnline
Abstract

In this study, we investigate the role of the multiple layers in deep transformer models. We design a variant of Albert that dynamically adapts the number of layers for each token of the input. The key specificity of Albert is that weights are tied across layers. Therefore, the stack of encoder layers iteratively repeats the application of the same transformation function on the input. We interpret the repetition of this application as an iterative process where the token contextualized representations are progressively refined. We analyze this process at the token level during pre-training, fine-tuning, and inference. We show that tokens do not require the same amount of iterations and that difficult or crucial tokens for the task are subject to more iterations.