This portal has been archived. Explore the next generation of this technology.
Lightweight Adaptive Mixture of Neural and N-gram Language Models
lib:78e4373a718aff78 (v1.0.0)
Authors: Anton Bakhtin,Arthur Szlam,Marc'Aurelio Ranzato,Edouard Grave
ArXiv: 1804.07705
Document: PDF DOI
Abstract URL: http://arxiv.org/abs/1804.07705v2
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams. In this work, we propose a method to improve how these two models are combined. By using a small network which predicts the mixture weight between the two models, we adapt their relative importance at each time step. Because the gating network is small, it trains quickly on small amounts of held out data, and does not add overhead at scoring time. Our experiments carried out on the One Billion Word benchmark show a significant improvement over the state of the art ensemble without retraining of the basic modules.
ArXiv: 1804.07705
Document: PDF DOI
Abstract URL: http://arxiv.org/abs/1804.07705v2
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams. In this work, we propose a method to improve how these two models are combined. By using a small network which predicts the mixture weight between the two models, we adapt their relative importance at each time step. Because the gating network is small, it trains quickly on small amounts of held out data, and does not add overhead at scoring time. Our experiments carried out on the One Billion Word benchmark show a significant improvement over the state of the art ensemble without retraining of the basic modules.
Relevant initiatives
Related knowledge about this paper
Reproduced results (crowd-benchmarking and competitions)
Artifact and reproducibility checklists
- ACM/cTuning artifact appendix and reproducibility checklist
- NeurIPS reproducibility checklist
- SIGPLAN's checklist for empirical evaluation
- Collective Knowledge (organizing research projects based on FAIR principles)