Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process

Fei Ye; Adrian Gheorghe Bors

doi:10.1109/ICCV48922.2021.01052

Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process

Fei Ye, Adrian Gheorghe Bors

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Recent research efforts in lifelong learning propose to grow a mixture of models to adapt to an increasing number of tasks. The proposed methodology shows promising results in overcoming catastrophic forgetting. However, the theory behind these successful models is still not well understood. In this paper, we perform the theoretical analysis for lifelong learning models by deriving the risk bounds based on the discrepancy distance between the probabilistic representation of data generated by the model and that corresponding to the target dataset. Inspired by the theoretical analysis, we introduce a new lifelong learning approach , namely the Lifelong Infinite Mixture (LIMix) model, which can automatically expand its network architectures or choose an appropriate component to adapt its parameters for learning a new task, while preserving its previously learnt information. We propose to incorporate the knowledge by means of Dirichlet processes by using a gat-ing mechanism which computes the dependence between the knowledge learnt previously and stored in each component , and a new set of data. Besides, we train a compact Student model which can accumulate cross-domain representations over time and make quick inferences. The code is available at https://github.com/dtuzi123/ Lifelong-infinite-mixture-model.

Original language	English
Title of host publication	IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher	IEEE
Pages	10695-10704
Number of pages	10
DOIs	https://doi.org/10.1109/ICCV48922.2021.01052
Publication status	Published - 28 Feb 2022

Bibliographical note

© IEEE, 2022. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

Access to Document

10.1109/ICCV48922.2021.01052

Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process
© IEEE, 2022. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details
Accepted author manuscript, 655 KB

Cite this

@inproceedings{5ef298e13a7d4063b55ef3e3eddd32b8,

title = "Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process",

abstract = "Recent research efforts in lifelong learning propose to grow a mixture of models to adapt to an increasing number of tasks. The proposed methodology shows promising results in overcoming catastrophic forgetting. However, the theory behind these successful models is still not well understood. In this paper, we perform the theoretical analysis for lifelong learning models by deriving the risk bounds based on the discrepancy distance between the probabilistic representation of data generated by the model and that corresponding to the target dataset. Inspired by the theoretical analysis, we introduce a new lifelong learning approach , namely the Lifelong Infinite Mixture (LIMix) model, which can automatically expand its network architectures or choose an appropriate component to adapt its parameters for learning a new task, while preserving its previously learnt information. We propose to incorporate the knowledge by means of Dirichlet processes by using a gat-ing mechanism which computes the dependence between the knowledge learnt previously and stored in each component , and a new set of data. Besides, we train a compact Student model which can accumulate cross-domain representations over time and make quick inferences. The code is available at https://github.com/dtuzi123/ Lifelong-infinite-mixture-model.",

author = "Fei Ye and Bors, {Adrian Gheorghe}",

note = "{\textcopyright} IEEE, 2022. This is an author-produced version of the published paper. Uploaded in accordance with the publisher{\textquoteright}s self-archiving policy. Further copying may not be permitted; contact the publisher for details",

year = "2022",

month = feb,

day = "28",

doi = "10.1109/ICCV48922.2021.01052",

language = "English",

pages = "10695--10704",

booktitle = "IEEE/CVF International Conference on Computer Vision (ICCV)",

publisher = "IEEE",

address = "United States",

}

TY - GEN

T1 - Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process

AU - Ye, Fei

AU - Bors, Adrian Gheorghe

N1 - © IEEE, 2022. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

PY - 2022/2/28

Y1 - 2022/2/28

N2 - Recent research efforts in lifelong learning propose to grow a mixture of models to adapt to an increasing number of tasks. The proposed methodology shows promising results in overcoming catastrophic forgetting. However, the theory behind these successful models is still not well understood. In this paper, we perform the theoretical analysis for lifelong learning models by deriving the risk bounds based on the discrepancy distance between the probabilistic representation of data generated by the model and that corresponding to the target dataset. Inspired by the theoretical analysis, we introduce a new lifelong learning approach , namely the Lifelong Infinite Mixture (LIMix) model, which can automatically expand its network architectures or choose an appropriate component to adapt its parameters for learning a new task, while preserving its previously learnt information. We propose to incorporate the knowledge by means of Dirichlet processes by using a gat-ing mechanism which computes the dependence between the knowledge learnt previously and stored in each component , and a new set of data. Besides, we train a compact Student model which can accumulate cross-domain representations over time and make quick inferences. The code is available at https://github.com/dtuzi123/ Lifelong-infinite-mixture-model.

AB - Recent research efforts in lifelong learning propose to grow a mixture of models to adapt to an increasing number of tasks. The proposed methodology shows promising results in overcoming catastrophic forgetting. However, the theory behind these successful models is still not well understood. In this paper, we perform the theoretical analysis for lifelong learning models by deriving the risk bounds based on the discrepancy distance between the probabilistic representation of data generated by the model and that corresponding to the target dataset. Inspired by the theoretical analysis, we introduce a new lifelong learning approach , namely the Lifelong Infinite Mixture (LIMix) model, which can automatically expand its network architectures or choose an appropriate component to adapt its parameters for learning a new task, while preserving its previously learnt information. We propose to incorporate the knowledge by means of Dirichlet processes by using a gat-ing mechanism which computes the dependence between the knowledge learnt previously and stored in each component , and a new set of data. Besides, we train a compact Student model which can accumulate cross-domain representations over time and make quick inferences. The code is available at https://github.com/dtuzi123/ Lifelong-infinite-mixture-model.

U2 - 10.1109/ICCV48922.2021.01052

DO - 10.1109/ICCV48922.2021.01052

M3 - Conference contribution

SP - 10695

EP - 10704

BT - IEEE/CVF International Conference on Computer Vision (ICCV)

PB - IEEE

ER -