Overcoming catastrophic forgetting with unlabeled data in the wild

Kibok Lee, Kimin Lee, Jinwoo Shin, Honglak Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

78 Citations (Scopus)


Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: The performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a novel class-incremental learning scheme with (a) a new distillation loss, termed global distillation, (b) a learning strategy to avoid overfitting to the most recent task, and (c) a confidence-based sampling method to effectively leverage unlabeled external data. Our experimental results on various datasets, including CIFAR and ImageNet, demonstrate the superiority of the proposed methods over prior methods, particularly when a stream of unlabeled data is accessible: Our method shows up to 15.8% higher accuracy and 46.5% less forgetting compared to the state-of-the-art method. The code is available at https://github.com/kibok90/iccv2019-inc.

Original languageEnglish
Title of host publicationProceedings - 2019 International Conference on Computer Vision, ICCV 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages10
ISBN (Electronic)9781728148038
Publication statusPublished - 2019 Oct
Event17th IEEE/CVF International Conference on Computer Vision, ICCV 2019 - Seoul, Korea, Republic of
Duration: 2019 Oct 272019 Nov 2

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
ISSN (Print)1550-5499


Conference17th IEEE/CVF International Conference on Computer Vision, ICCV 2019
Country/TerritoryKorea, Republic of

Bibliographical note

Funding Information:
This work was supported in part by Kwanjeong Educational Foundation Scholarship, NSF CAREER IIS-1453651, and Sloan Research Fellowship. We also thank Lajanugen Logeswaran, Jongwook Choi, Yijie Guo, Wilka Carvalho, and Yunseok Jang for helpful discussions.

Publisher Copyright:
© 2019 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Overcoming catastrophic forgetting with unlabeled data in the wild'. Together they form a unique fingerprint.

Cite this