Lifelong Mixture of Variational Autoencoders

TY - JOUR

T1 - Lifelong Mixture of Variational Autoencoders

AU - Ye, Fei

AU - Bors, Adrian Gheorghe

N1 - © IEEE 2021. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

PY - 2023/1/1

Y1 - 2023/1/1

N2 - In this paper, we propose an end-to-end lifelong learning mixture of experts. Each expert is implemented by a Variational Autoencoder (VAE). The experts in the mixture system are jointly trained by maximizing a mixture of individualcomponent evidence lower bounds (MELBO) on the loglikelihood of the given training samples. The mixing coefficients in the mixture model, control the contributions of each expert in the global representation. These are sampled from a Dirichlet distribution whose parameters are determined through nonparametric estimation during the lifelong learning. The model can learn new tasks fast when these are similar to those previously learnt. The proposed Lifelong mixture of VAE (LMVAE) expands its architecture with new components when learning a completely new task. After the training, our model can automatically determine the relevant expert to be used whenfed with new data samples. This mechanism benefits both the memory efficiency and the required computational cost as only one expert model is used during the inference. The L-MVAE inference model is able to perform interpolations in the jointlatent space across the data domains associated with different tasks and is shown to be efficient for disentangled learning representation.

AB - In this paper, we propose an end-to-end lifelong learning mixture of experts. Each expert is implemented by a Variational Autoencoder (VAE). The experts in the mixture system are jointly trained by maximizing a mixture of individualcomponent evidence lower bounds (MELBO) on the loglikelihood of the given training samples. The mixing coefficients in the mixture model, control the contributions of each expert in the global representation. These are sampled from a Dirichlet distribution whose parameters are determined through nonparametric estimation during the lifelong learning. The model can learn new tasks fast when these are similar to those previously learnt. The proposed Lifelong mixture of VAE (LMVAE) expands its architecture with new components when learning a completely new task. After the training, our model can automatically determine the relevant expert to be used whenfed with new data samples. This mechanism benefits both the memory efficiency and the required computational cost as only one expert model is used during the inference. The L-MVAE inference model is able to perform interpolations in the jointlatent space across the data domains associated with different tasks and is shown to be efficient for disentangled learning representation.

U2 - 10.1109/TNNLS.2021.3096457

DO - 10.1109/TNNLS.2021.3096457

M3 - Article

VL - 34

SP - 461

EP - 474

JO - IEEE Transactions on Neural Networks and Learning Systems

JF - IEEE Transactions on Neural Networks and Learning Systems

SN - 2162-237X

IS - 1

ER -