Titre | How Many Layers and Why? An Analysis of the Model Depth in Transformers |
Publication Type | Article dans des actes |
Année de la conférence | 2021 |
Authors | Simoulin, Antoine, and Benoît Crabbé |
Nom de la conférence | Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop |
Pagination | 221–228 |
Publisher | Association for Computational Linguistics |
Conference Location | Online |
Abstract | In this study, we investigate the role of the multiple layers in deep transformer models. We design a variant of Albert that dynamically adapts the number of layers for each token of the input. The key specificity of Albert is that weights are tied across layers. Therefore, the stack of encoder layers iteratively repeats the application of the same transformation function on the input. We interpret the repetition of this application as an iterative process where the token contextualized representations are progressively refined. We analyze this process at the token level during pre-training, fine-tuning, and inference. We show that tokens do not require the same amount of iterations and that difficult or crucial tokens for the task are subject to more iterations. |