Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations

Author
Affiliations

Yoto Fujita

Simon Leglaive

Laurent Girin

CentraleSupélec, IETR (UMR CNRS 6164)

Grenoble INP, GIPSA-Lab (UMR CNRS 5216)

Published

May 19, 2026

Abstract

Previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained using neural audio codecs (NACs). However, a recent study has shown that continuous latent representations of NACs can be advantageous for SE in terms of speech quality and intelligibility. In this work, we propose masked autoregressive SE (MARSE), a unified probabilistic framework for SE based on iterative decoding of masked clean speech frames using continuous NAC representations of speech. In particular, we investigate a set of different decoding policies, ceteris paribus, that is, using the same DNN (a Conformer model), the same NAC (the DAC codec) and the same training setup. The results show that the proposed framework enables a flexible trade-off between SE performance and computational cost.

Note: The source code repository is still a work in progress.

Method

MARSE illustration
Overview of the proposed MARSE framework applied on a continuous NAC representation (left: inference; right: training).

Experiment

OVRL comparison
DNSMOS OVRL score obtained by the proposed MARSE model (for the three decoding policies) and the C-AR and C-NAR baselines on a subset of 300 samples from Libri1Mix test, as a function of the number of decoding iterations.

Audio Examples

The audio samples below were randomly selected.

The "Clean" entry contains the ground-truth clean speech signal, while the "Clean (DAC)" entry contains the same signal encoded and decoded by the DAC neural audio codec, isolating the degradation introduced by the codec itself from that introduced by the enhancement models.

SNR values (in dB) were computed as the difference between the ITU-R BS.1770-4 integrated loudness (LUFS) of the clean signal and that of the noise, following the LibriMix dataset creation protocol.


Libri2Mix example 1

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 2

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 3

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 4

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 5

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 6

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 7

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 8

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 9

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

Libri2Mix example 10

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 1

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 2

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 3

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 4

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 5

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 6

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 7

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 8

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 9

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50

LibriDEMAND example 10

Noisy Clean Clean (DAC) C-NAR C-AR ConvTasNet DPTNet
MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal MARSE-causal
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random MARSE-NC-random
N=1 N=5 N=10 N=20 N=30 N=40 N=50
MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle MARSE-NC-oracle
N=1 N=5 N=10 N=20 N=30 N=40 N=50