Vision Transformers: From Theory to Implementation

Abdelhafid Berroukham; Mohammed Lahraichi

doi:10.64060/ANMIV1i31

Authors

Abdelhafid Berroukham Faculty of Science, Ibne Tofail University, Kenitra, Morocco Author
Mohammed Lahraichi Faculty of Science, Ibne Tofail University, Kenitra, Morocco Author

DOI:

https://doi.org/10.64060/ANMIV1i31

Keywords:

Computer Vision, Deep Learning, Vision Transformers, Convolutional Neural Networks

Abstract

The field of computer vision has passed through a paradigm shift with the appearance of Vision Transformers (ViTs), an revolutionary technique inspired by natural language processing (NLP). ViTs represents a paradigm shift, transferring far from traditional convolutional neural networks (CNNs) by the usage of the strength of self-attention mechanisms for comprehensive image understanding. This paper offers a comprehensive explanation of ViTs, starting with an exploration of their theoretical foundations and concepts, along with the mechanisms of self-attention and multi-headed attention, to their implementation. It presents practical insights for researchers and practitioners. Furthermore, we highlighted the actual-world applications of ViTs, which cover a number of domains from healthcare to autonomous systems, in order to reveal their adaptability and potential.

References

[1] A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” arXiv, Jun. 03, 2021. Accessed: May 22, 2022. [Online]. Available: http://arxiv.org/abs/2010.11929

[2] A. Vaswani et al., “Attention Is All You Need.” arXiv, Dec. 05, 2017. Accessed: May 22, 2022. [Online]. Available: http://arxiv.org/abs/1706.03762

[3] K. O’Shea and R. Nash, “An Introduction to Convolutional Neural Networks.” arXiv, Dec. 02, 2015. Accessed: May 22, 2022. [Online]. Available: http://arxiv.org/abs/1511.08458

[4] J. Y.-Y. Lin, S.-M. Liao, H.-J. Huang, W.-T. Kuo, and O. H.-M. Ou, “Galaxy Morphological Classification with Efficient Vision Transformer.” arXiv, Feb. 03, 2022. Accessed: Jun. 30, 2022. [Online]. Available: http://arxiv.org/abs/2110.01024

[5] H. Chen et al., “GasHis-Transformer: A Multi-scale Visual Transformer Approach for Gastric Histopathology Image Classification.” arXiv, Feb. 17, 2022. Accessed: Jun. 01, 2022. [Online]. Available: http://arxiv.org/abs/2104.14528

[6] Y. Lee and P. Kang, “AnoViT: Unsupervised Anomaly Detection and Localization With Vision Transformer-Based Encoder-Decoder,” IEEE Access, vol. 10, pp. 46717–46724, 2022, doi: 10.1109/ACCESS.2022.3171559.

[7] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-End Object Detection with Transformers.” arXiv, May 28, 2020. Accessed: May 18, 2022. [Online]. Available: http://arxiv.org/abs/2005.12872

[8] S. Zheng et al., “Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.” arXiv, Jul. 25, 2021. Accessed: May 18, 2022. [Online]. Available: http://arxiv.org/abs/2012.15840

[9] R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for Semantic Segmentation.” arXiv, Sep. 02, 2021. Accessed: May 05, 2023. [Online]. Available: http://arxiv.org/abs/2105.05633

[10] D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs).” arXiv, Jul. 08, 2020. Accessed: Feb. 19, 2023. [Online]. Available: http://arxiv.org/abs/1606.08415