Cognitive Science Program

Deep learning for monaural speech separation (2014)
Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson and Paris Smaragdis
1562 -1566

Abstract

Monaural source separation is useful for many real-world applications though it is a challenging problem. In this paper, we study deep learning for monaural speech separation. We propose the joint optimization of the deep learning models (deep neural networks and recurrent neural networks) with an extra masking layer, which enforces a reconstruction constraint. Moreover, we explore a discriminative training criterion for the neural networks to further enhance the separation performance. We evaluate our approaches using the TIMIT speech corpus for a monaural speech separation task. Our proposed models achieve about 3.8~4.9 dB SIR gain compared to NMF models, while maintaining better SDRs and SARs.

Joint optimization of masks and deep recurrent neural networks for monaural source separation (2015)
Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson and Paris Smaragdis
IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23 (12), 2136-2147

Abstract

Monaural source separation is important for many real world applications. It is challenging because, with only a single channel of information available, without any constraints, an infinite number of solutions are possible. In this paper, we explore joint optimization of masking functions and deep recurrent neural networks for monaural source separation tasks, including speech separation, singing voice separation, and speech denoising. The joint optimization of the deep recurrent neural networks with an extra masking layer enforces a reconstruction constraint. Moreover, we explore a discriminative criterion for training neural networks to further enhance the separation performance. We evaluate the proposed system on the TSP, MIR-1K, and TIMIT datasets for speech separation, singing voice separation, and speech denoising tasks, respectively. Our approaches achieve 2.30-4.98 dB SDR gain compared to NMF …

Bitwise Neural Networks (2015)
Minje Kim and Paris Smaragdis

Abstract

Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.

SINGING-VOICE SEPARATION USING DEEP RECURRENT NEURAL NETWORKS (2014)
Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson and Paris Smaragdis

SINGING-VOICE SEPARATION FROM MONAURAL RECORDINGS USING DEEP RECURRENT NEURAL NETWORKS (2014)
Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson and Paris Smaragdis

Abstract

Monaural source separation is important for many real world applications. It is challenging since only single channel information is available. In this paper, we explore using deep recurrent neural networks for singing voice separation from monaural recordings in a supervised setting. Deep recurrent neural networks with different temporal connections are explored. We propose jointly optimizing the networks for multiple source signals by including the separation step as a nonlinear operation in the last layer. Different discriminative training objectives are further explored to enhance the source to interference ratio. Our proposed system achieves the state-of-the-art performance, 2.30⇠ 2.48 dB GNSDR gain and 4.32⇠ 5.42 dB GSIR gain compared to previous models, on the MIR-1K dataset.

Experiments on Deep Learning for Speech Denoising (2014)
Ding Liu, Paris Smaragdis and Minje Kim

Abstract

In this paper we present some experiments using a deep learning model for speech denoising. We propose a very lightweight procedure that can predict clean speech spectra when presented with noisy speech inputs, and we show how various parameter choices impact the quality of the denoised signal. Through our experiments we conclude that such a structure can perform better than some comparable single-channel approaches and that it is able to generalize well across various speakers, noise types and signal-to-noise ratios.

Nonnegative matrix partial co-factorization for drum source separation (2010)
Jiho Yoo, Minje Kim, Kyeongok Kang and Seungjin Choi
IEEE. 1942-1945

Abstract

We address a problem of separating drums from polyphonic music containing various pitched instruments as well as drums. Nonnegative matrix factorization (NMF) was successfully applied to spectrograms of music to learn basis vectors, followed by support vector machine (SVM) to classify basis vectors into ones associated with drums (rhythmic source) only and pitched instruments (harmonic sources). Basis vectors associated with pitched instruments are used to reconstruct drum-eliminated music. However, it is cumbersome to construct a training set for pitched instruments since various instruments are involved. In this paper, we propose a method which only incorporates prior knowledge on drums, not requiring such training sets of pitched instruments. To this end, we present nonnegative matrix partial co-factorization (NMPCF) where the target matrix (spectrograms of music) and drum-only-matrix (collected …

Nonnegative Matrix Partial Co-Factorization for Spectral and Temporal Drum Source Separation (2011)
Minje Kim, Jiho Yoo, Kyeongok Kang and Seungjin Choi
Selected Topics in Signal Processing, IEEE Journal of, 5 (6), 1192-1204

Abstract

We address a problem of separating drum sources from monaural mixtures of polyphonic music containing various pitched instruments as well as drums. We consider a spectrogram of music, described by a matrix where each row is associated with intensities of a frequency over time. We employ a joint decomposition to several spectrogram matrices that include two or more column-blocks of the mixture spectrograms (columns of mixture spectrograms are partitioned into 2 or more blocks) and a drum-only (drum solo playing) matrix constructed from various drums a priori. To this end, we apply nonnegative matrix partial co-factorization (NMPCF) to these target matrices, in which column-blocks of mixture spectrograms and the drum-only matrix are jointly decomposed, sharing a factor matrix partially, in order to determine common basis vectors that capture the spectral and temporal characteristics of drum sources …

Mixtures of Local Dictionaries for Unsupervised Speech Enhancement (2015)
Minje Kim and Paris Smaragdis
IEEE Signal Processing Letters, 22 (3), 288 - 292

Abstract

We propose a novel extension of Nonnegative Matrix Factorization (NMF) that models a signal with multiple local dictionaries activated sparsely. This set of local dictionaries for a source, e.g., speech, disjointly constitute a superset that is more discriminative than an ordinary NMF dictionary, because its local structures represent the source's manifold better. A block sparsity constraint is used to regularize the NMF solutions so that only one or a small number of blocks are active at a given time. Moreover, a concentrationz prior further regularizes each block of bases to be close to each other for better locality preservation. We test the proposed Mixture of Local Dictionaries (MLD) on single-channel speech enhancement tasks and show that it outperforms the state of the art technology by up to 2 dB in signal-to-distortion ratio, especially in the unsupervised environment where neither the speaker identity nor the type of …

Monaural music source separation: Nonnegativity, sparseness, and shift-invariance (2006)
Minje Kim and Seungjin Choi
Springer, Berlin, Heidelberg. 617-624

Abstract

In this paper we present a method for polyphonic music source separation from their monaural mixture, where the underlying assumption is that the harmonic structure of a musical instrument remains roughly the same even if it is played at various pitches and is recorded in various mixing environments. We incorporate with nonnegativity, shift-invariance, and sparseness to select representative spectral basis vectors that are used to restore music sources from their monaural mixture. Experimental results with monaural instantaneous mixture of voice/cello and monaural convolutive mixture of saxophone/viola, are shown to confirm the validity of our proposed method.

Single Channel Source Separation Using Smooth Nonnegative Matrix Factorization with Markov Random Fields (2013)
Minje Kim and Paris Smaragdis

Abstract

This paper presents a single channel source separation method based on an extension of Nonnegative Matrix Factorization (NMF) algorithm by smoothing the original posterior probabilities with an additional Markov Random Fields (MRF) structure. Our method is based on the alternative interpretation of NMF with β-divergence as latent variable models. By doing so, we can redefine NMF-based separation procedure as a Bayesian labeling problem where each label stands for the mask for a specific source. This understanding leads us to intervene in the calculation of posterior probabilities, so that the priors from MRF's neighboring structure can smooth out isolated masking values that have different labeling results from their neighbors. Experiments on several dictionary-based source separation tasks show sensible performance gains.

XNOR-POP: A processing-in-memory architecture for binary Convolutional Neural Networks in Wide-IO2 DRAMs (2017)
Lei Jiang, Minje Kim, Wujie Wen and Danghui Wang
IEEE. 6-Jan

Abstract

It is challenging to adopt computing-intensive and parameter-rich Convolutional Neural Networks (CNNs) in mobile devices due to limited hardware resources and low power budgets. To support multiple concurrently running applications, one mobile device needs to perform multiple CNN tests simultaneously in real-time. Previous solutions cannot guarantee a high enough frame rate when serving multiple applications with reasonable hardware and power cost. In this paper, we present a novel process-in-memory architecture to process emerging binary CNN tests in Wide-IO2 DRAMs. Compared to state-of-the-art accelerators, our design improves CNN test performance by 4× ~ 11× with small hardware and power overhead.

Adaptive Denoising Autoencoders: A Fine-tuning Scheme to Learn from Test Mixtures (2015)
Minje Kim and Paris Smaragdis

Abstract

This work aims at a test-time fine-tune scheme to further improve the performance of an already-trained Denoising AutoEncoder (DAE) in the context of semi-supervised audio source separation. Although the state-of-the-art deep learning-based DAEs show sensible denoising performance when the nature of artifacts is known in advance, the scalability of an already-trained network to an unseen signal with an unknown characteristic of deformation is not well studied. To handle this problem, we propose an adaptive fine-tuning scheme where we define a test-time target variables so that a DAE can learn from the newly available sources and the mixing environments in the test mixtures. In the proposed network topology, we stack an AutoEncoder (AE) trained from clean source spectra of interest on top of a DAE trained from a variety of available mixture spectra. Hence, the bottom DAE outputs are used as the …

LPC residual signal encoding/decoding apparatus of modified discrete cosine transform (MDCT)-based unified voice/audio encoding device (2014)
Seung Kwon Beack, Tae Jin Lee, Min Je Kim, Kyeongok Kang, Dae Young Jang, Jin Woo Hong, SEO Jeongil, Chieteuk Ahn, Hochong Park, Young-Cheol Park

Abstract

Disclosed is an LPC residual signal encoding/decoding apparatus of an MDCT based unified voice and audio encoding device. The LPC residual signal encoding apparatus analyzes a property of an input signal, selects an encoding method of an LPC filtered signal, and encode the LPC residual signal based on one of a real filterbank, a complex filterbank, and an algebraic code excited linear prediction (ACELP).

Breakdown of the chiral anomaly in Weyl semimetals in a strong magnetic field (2017)
Pilkwang Kim, Ji Hoon Ryoo and Cheol-Hwan Park
Physical review letters, 119 (26), 266401

Abstract

The low-energy quasiparticles of Weyl semimetals are a condensed-matter realization of the Weyl fermions introduced in relativistic field theory. Chiral anomaly, the nonconservation of the chiral charge under parallel electric and magnetic fields, is arguably the most important phenomenon of Weyl semimetals and has been explained as an imbalance between the occupancies of the gapless, zeroth Landau levels with opposite chiralities. This widely accepted picture has served as the basis for subsequent studies. Here we report the breakdown of the chiral anomaly in Weyl semimetals in a strong magnetic field based on ab initio calculations. A sizable energy gap that depends sensitively on the direction of the magnetic field may open up due to the mixing of the zeroth Landau levels associated with the opposite-chirality Weyl points that are away from each other in the Brillouin zone. Our study provides a theoretical …