Continual Variational Autoencoder Learning via Online Cooperative Memorization

Fei Ye; Adrian Gheorghe Bors

doi:10.1007/978-3-031-20050-2_31

Continual Variational Autoencoder Learning via Online Cooperative Memorization

Fei Ye, Adrian Gheorghe Bors

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Due to their inference, data representation and reconstruction properties, Variational Autoencoders (VAE) have been successfully used in continual learning classification tasks. However, their ability to generate images with specifications corresponding to the classes and databases learned during Continual Learning (CL) is not well understood and catastrophic forgetting remains a significant challenge. In this paper, we firstly analyze the forgetting behaviour of VAEs by developing a new theoretical framework that formulates CL as a dynamic optimal transport problem. This framework proves approximate bounds to the data likelihood without requiring the task information and explains how the prior knowledge is lost during the training process. We then propose a novel memory buffering approach, namely the Online Cooperative Memorization (OCM) framework, which consists of a Short-Term Memory (STM) that continually stores recent samples to provide future information for the model, and a Long-Term Memory (LTM) aiming to preserve a wide diversity of samples. The proposed OCM transfers certain samples from STM to LTM according to the information diversity selection criterion without requiring any supervised signals. The OCM framework is then combined with a dynamic VAE expansion mixture network for further enhancing its performance.

Original language	English
Title of host publication	Proceedings of the European Conference on Computer Vision (ECCV)
Publisher	Springer
Pages	531–549
Number of pages	18
Volume	LNCS 13683
ISBN (Print)	9783031200496
DOIs	https://doi.org/10.1007/978-3-031-20050-2_31
Publication status	Published - 28 Oct 2022

Publication series

Name	Lecture Notes in Computer Science (LNCS)
Publisher	Springer

Bibliographical note

This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

Access to Document

10.1007/978-3-031-20050-2_31

OnlineCooperativeMemory_ECCV2022
This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details
Accepted author manuscript, 1.7 MB

Cite this

@inproceedings{1a6491dbba0a4403811641bfedc44575,

title = "Continual Variational Autoencoder Learning via Online Cooperative Memorization",

abstract = "Due to their inference, data representation and reconstruction properties, Variational Autoencoders (VAE) have been successfully used in continual learning classification tasks. However, their ability to generate images with specifications corresponding to the classes and databases learned during Continual Learning (CL) is not well understood and catastrophic forgetting remains a significant challenge. In this paper, we firstly analyze the forgetting behaviour of VAEs by developing a new theoretical framework that formulates CL as a dynamic optimal transport problem. This framework proves approximate bounds to the data likelihood without requiring the task information and explains how the prior knowledge is lost during the training process. We then propose a novel memory buffering approach, namely the Online Cooperative Memorization (OCM) framework, which consists of a Short-Term Memory (STM) that continually stores recent samples to provide future information for the model, and a Long-Term Memory (LTM) aiming to preserve a wide diversity of samples. The proposed OCM transfers certain samples from STM to LTM according to the information diversity selection criterion without requiring any supervised signals. The OCM framework is then combined with a dynamic VAE expansion mixture network for further enhancing its performance.",

author = "Fei Ye and Bors, {Adrian Gheorghe}",

note = "This is an author-produced version of the published paper. Uploaded in accordance with the publisher{\textquoteright}s self-archiving policy. Further copying may not be permitted; contact the publisher for details ",

year = "2022",

month = oct,

day = "28",

doi = "10.1007/978-3-031-20050-2_31",

language = "English",

isbn = "9783031200496",

volume = "LNCS 13683",

series = "Lecture Notes in Computer Science (LNCS)",

publisher = "Springer",

pages = "531–549",

booktitle = "Proceedings of the European Conference on Computer Vision (ECCV)",

address = "Germany",

}

TY - GEN

T1 - Continual Variational Autoencoder Learning via Online Cooperative Memorization

AU - Ye, Fei

AU - Bors, Adrian Gheorghe

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

PY - 2022/10/28

Y1 - 2022/10/28

N2 - Due to their inference, data representation and reconstruction properties, Variational Autoencoders (VAE) have been successfully used in continual learning classification tasks. However, their ability to generate images with specifications corresponding to the classes and databases learned during Continual Learning (CL) is not well understood and catastrophic forgetting remains a significant challenge. In this paper, we firstly analyze the forgetting behaviour of VAEs by developing a new theoretical framework that formulates CL as a dynamic optimal transport problem. This framework proves approximate bounds to the data likelihood without requiring the task information and explains how the prior knowledge is lost during the training process. We then propose a novel memory buffering approach, namely the Online Cooperative Memorization (OCM) framework, which consists of a Short-Term Memory (STM) that continually stores recent samples to provide future information for the model, and a Long-Term Memory (LTM) aiming to preserve a wide diversity of samples. The proposed OCM transfers certain samples from STM to LTM according to the information diversity selection criterion without requiring any supervised signals. The OCM framework is then combined with a dynamic VAE expansion mixture network for further enhancing its performance.

AB - Due to their inference, data representation and reconstruction properties, Variational Autoencoders (VAE) have been successfully used in continual learning classification tasks. However, their ability to generate images with specifications corresponding to the classes and databases learned during Continual Learning (CL) is not well understood and catastrophic forgetting remains a significant challenge. In this paper, we firstly analyze the forgetting behaviour of VAEs by developing a new theoretical framework that formulates CL as a dynamic optimal transport problem. This framework proves approximate bounds to the data likelihood without requiring the task information and explains how the prior knowledge is lost during the training process. We then propose a novel memory buffering approach, namely the Online Cooperative Memorization (OCM) framework, which consists of a Short-Term Memory (STM) that continually stores recent samples to provide future information for the model, and a Long-Term Memory (LTM) aiming to preserve a wide diversity of samples. The proposed OCM transfers certain samples from STM to LTM according to the information diversity selection criterion without requiring any supervised signals. The OCM framework is then combined with a dynamic VAE expansion mixture network for further enhancing its performance.

U2 - 10.1007/978-3-031-20050-2_31

DO - 10.1007/978-3-031-20050-2_31

M3 - Conference contribution

SN - 9783031200496

VL - LNCS 13683

T3 - Lecture Notes in Computer Science (LNCS)

SP - 531

EP - 549

BT - Proceedings of the European Conference on Computer Vision (ECCV)

PB - Springer

ER -