We are very excited to join forces with MLCommons and OctoML.ai! Contact Grigori Fursin for more details!

Class-Agnostic Continual Learning of Alternating Languages and Domains

lib:c5f99b36f312a903 (v1.0.0)

Authors: Germán Kruszewski,Ionut-Teodor Sorodoc,Tomas Mikolov
ArXiv: 2004.03340
Document:  PDF  DOI 
Abstract URL: https://arxiv.org/abs/2004.03340v1


Continual Learning has been often framed as the problem of training a model in a sequence of tasks. In this regard, Neural Networks have been attested to forget the solutions to previous task as they learn new ones. Yet, modelling human life-long learning does not necessarily require any crisp notion of tasks. In this work, we propose a benchmark based on language modelling in a multilingual and multidomain setting that prescinds of any explicit delimitation of training examples into distinct tasks, and propose metrics to study continual learning and catastrophic forgetting in this setting. Then, we introduce a simple Product of Experts learning system that performs strongly on this problem while displaying interesting properties, and investigate its merits for avoiding forgetting.

Relevant initiatives  

Related knowledge about this paper Reproduced results (crowd-benchmarking and competitions) Artifact and reproducibility checklists Common formats for research projects and shared artifacts Reproducibility initiatives

Comments  

Please log in to add your comments!
If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!