Self-supervised learning (SSL) has shown to be vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art for various tasks with minimal adaptation. However, existing works of SSL explored a limited number of speech processing tasks, blurring the generalizability and re-usability of SSL models across speech processing tasks. This talk will first introduce Speech processing Universal PERformance Benchmark (SUPERB), which is a leaderboard to benchmark the performance of SSL model across a wide range of speech processing tasks. The results on SUPERB demonstrates that SSL representations show competitive generalizability across speech processing tasks. I will also share some ongoing research based on SUPERB.

Self-supervised speech representation learning has attracted a lot of interest in both industry and academia. It is a challenge to leverage self-supervision on unlabeled data to improve the performance of streaming ASR models optimized with large-scale labeled data. In this talk, we will introduce our recent work on building streamable Transformer models with supervised and semi-supervised learning. In the first part, we will discuss how to balance accuracy, latency, and runtime cost for streamable Transformer model in real scenarios, and elaborate our streamable method which outperforms traditional RNN-T significantly on the accuracy-wise evaluation with large-scale training data. In the second part, we introduce UniSpeech which combines supervised loss and contrastive loss in a manner of multitask learning. UniSpeech outperforms supervised transfer-learning and unsupervised pre-training on both low-resource and high-resource scenarios. We further extend the idea on a reconstruction-loss-based model, showing better results on a public evaluation set and significantly decrease the GPU cost in training.

Learning representations of data in an unsupervised way is an open but actively researched problem of machine learning. We consider representations of speech and handwriting learned using autoencoders and contrastive self-supervised methods. These representations capture high level semantic content from the signal, e.g. phoneme or character identities, while being invariant to confounding low level details in the signal such as the underlying pitch contour or background noise. We investigate how to enforce desired properties of the latent representation, concentrating on enforcing temporal segmentation of the latent representation. Finally, we demonstrate the usefulness of such representations in unsupervised acoustic unit discovery on the ZeroSpeech task.

There are around 7000 spoken languages in the world. For most of these languages, not enough transcribed speech data is available to train automatic speech recognition (ASR) systems. This leads to the current and unwanted situation that high-performance ASR systems are only available for a small number of major languages,. To facilitate ASR technology for low-resource languages, investigation of unsupervised acoustic modeling (UAM) methods is necessary, which aims to find and model a set of basic speech units that represents all the sounds in the language of interest, i.e., the low-resource, target language.

This talk addresses the topic of unsupervised subword modelling, i.e., learning acoustic feature representations that can distinguish subword (phoneme) units of a target language, and is robust to linguistically-irrelevant factors, such as speaker information. Unsupervised subword modeling thus focuses on learning an intermediate representation towards the ultimate goal of UAM. I will present a recently proposed, state-of-the-art method for unsupervised subword modelling which consists of a two-stage learning framework that combines self-supervised learning and cross-lingual knowledge transfer, and several recent experiments that investigate the effectiveness and robustness of this method. To this end, I will present experiments investigating the robustness of the approach’s effectiveness to different amounts of training data, and several comprehensive and systematic analyses at the phoneme- and articulatory feature (AF)-level to investigate the method’s ability to capture specific linguistic information.

Despite rapid progress in the recent past, current speech recognition systems rely heavily on labeled training data. This limits the technology to a small fraction of languages and accents spoken around the globe. In this talk, I will give an overview of the wav2vec project which recently enabled speech recognition models with no labeled data. The key ingredient is self-supervised pre-training to learn powerful representations of speech audio solely from unlabeled data. Unsupervised speech recognition rivals some of the best published systems trained on 960 hours of labeled data from only two years ago while using no labeled data. This is an important step towards systems which can learn to solve tasks without explicit supervision.

Recent success in speech representation learning enables a new way to leverage unlabeled data to train speech recognition models. The essence of speech representation is to exploit acoustically meaningful representation on large quantities of unlabeled data via self-supervised learning. There are many successful examples on speech representation learning, including but not limited to wav2vec/wav2vec 2.0, APC/vq-APC, Mockingjay, and more recently, HuBERT. In this talk, I am going to describe a collection of our work and their applications to speech recognition and other speech processing tasks. First, I will describe the original DeCoAR (Deep Contextualized Acoustic Representation) work, which encodes input audio into a contextualized representation from bidirectional sequence models; followed by that, I will discuss a few modifications to improve DeCoAR, based on inspirations from other speech representation learning work. I will demonstrate that DeCoAR and its improved version can attain competitive ASR performance using limited amount of labeled data. Finally, I will present BERTPhone, a variation of DeCoAR that combines self-supervision objective with weak supervision, and its application to speaker and language identification tasks.

Semi-supervised learning, which uses unlabeled data to enhance the performance of labeled tasks, has recently played a crucial part in improving pubic automatic speech recognition (ASR) benchmarks. In this talk, we summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining state-of-the-art (SoTA) performance on many public benchmarks. We also utilize the learned representation of pre-trained networks for achieving SoTA results on non-ASR tasks. In addition to scaling up, we proposed w2v-BERT, a framework that combines contrastive learning and MLM, where the former trains the model to discretize input continuous speech signals into a finite set of discriminative speech tokens, and the latter trains the model to learn contextualized speech representations via solving a masked prediction task consuming the discretized tokens.