Imported from https://github.com/pytorch/fairseq/blob/master/fairseq/models/wav2vec.py