arXiv:2111.06377 · cs.CV · 2021
Masked Autoencoders Are Scalable Vision Learners
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick
13,216 citations · 2,164 influential citations— Semantic Scholar
TL;DR · Semantic Scholar
This paper develops an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens.
Saved summary
AI-generated while a reader studied this paper. Check the original source for accuracy.
문제: 컴퓨터 비전은 이미지의 높은 spatial redundancy와 픽셀 복원의 낮은 의미론적 수준으로 인해 self-supervised learning 적용에 어려움이 있었습니다. 기존 방식은 큰 모델을 효율적으로 학습시키지 못했습니다.
방법: 이미지 패치의 75%를 마스킹하고 빠진 픽셀을 복원하는 Masked Autoencoders (MAE)를 제안합니다. 마스킹 안 된 패치만 처리하는 encoder와 mask tokens를 함께 처리하는 가벼운 decoder의 asymmetric 구조를 가집니다.
결과: pre-training 속도를 3배 이상 높였고 vanilla ViT-Huge 모델로 ImageNet-1K에서 87.8%의 정확도를 기록했습니다. COCO, ADE20K 등 downstream tasks에서도 뛰어난 transfer learning 성능을 보였습니다.
한계: 학습 데이터의 편향을 반영하여 부정적인 사회적 영향을 줄 수 있고 존재하지 않는 내용을 임의로 생성할 위험이 있습니다. 또한 representation의 linear separability가 상대적으로 낮아 linear probing 성능은 약세입니다.
Abstract
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.
Source: https://arxiv.org/abs/2111.06377 — Thank you to arXiv for use of its open access interoperability.
Papers citing this work · Semantic Scholar
- Towards elastic vision transformers via contrastive masked distillation (2027) · 0 citations
- PGP-TAK: A self-supervised pretraining-driven framework for superheat degree monitoring in aluminum electrolysis (2026) · 0 citations
- DFX-Net: dual-domain frequency networks with cross-scale exchange for image manipulation localization (2026) · 0 citations
- DSEL: Deep-layer selective learning of spatiotemporal context for object tracking (2026) · 0 citations
- Impact damage localization in woven CFRP laminates using laser-ultrasonic wavefields and a masked spatiotemporal reconstruction network (2026) · 0 citations
Read the original paper, explore key passages and ask questions in AewolFlow.