We are very excited to join forces with MLCommons and OctoML.ai! Contact Grigori Fursin for more details!

Context Vectors are Reflections of Word Vectors in Half the Dimensions

lib:38cbfc11449489dc (v1.0.0)

Authors: Zhenisbek Assylbekov,Rustem Takhanov
ArXiv: 1902.09859
Document:  PDF  DOI 
Abstract URL: http://arxiv.org/abs/1902.09859v1


This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are well supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.

Relevant initiatives  

Related knowledge about this paper Reproduced results (crowd-benchmarking and competitions) Artifact and reproducibility checklists Common formats for research projects and shared artifacts Reproducibility initiatives

Comments  

Please log in to add your comments!
If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!