AuK: An Open-Source Foundational Model for Speech Generation and Editing

Shanghai Jiao Tong University Tencent Hunyuan Shanghai Innovation Institution

One model for every speech task

AuK is a 1.5B-parameter foundational model that answers a natural-language instruction with edited or generated audio. A multimodal language model reads the instruction alongside optional reference audio; a 50 Hz VAE supplies acoustic latents; a hybrid transformer fuses both streams under a flow-matching objective.

Abstract

We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an audio VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation–editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 times wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.

Performance comparison with SOTA models across speech generation, editing, enhancement and separation.
Performance comparison with SOTA models across speech generation, editing, enhancement and separation. (a) Seed-TTS-Eval WER and SIM, averaged over test-en/test-zh/test-zh-hard, measure intelligibility and speaker similarity; InstructTTSEval DSD (mean of DSD-ZH/EN) measures how faithfully a model realises a free-form timbre instruction. (b) MMAE-Speech EMR measures exact-match success across all editing rubrics; SpeechEditBench averages five edit types, Ming-Freeform-Audio-Edit four semantic-editing splits. (c) Per model, DNSMOS-OVRL (darker, left) and UTMOS (lighter, right) rate perceptual quality; DNS Challenge and CHiME-4 are enhancement, Libri2Mix separation. Higher is better everywhere except WER, where shorter bars indicate fewer errors.

Demos

Citation

If you use AuK in your research, please cite:

@misc{ma2026auktechnicalreportopensource,
  title         = {AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing},
  author        = {Ziyang Ma and Zhikang Niu and Wenming Tu and Tianrui Wang and Ruiqi Yan and Junxi Liu and Yanru Huo and Nickk Huang and Yang Liu and Qicong Xie and Zeyu Xie and Hui Wang and Haitao Li and Zixuan Jiang and Yalin Li and Jie Fang and Yifan Duan and Zeyue Tian and Guangzheng Li and Haina Zhu and Shuyi Wang and Jinwen Wang and Mingyu Cui and Tian Tan and Auden and Sen Liang and Steve Yves and Shan Yang and Liefeng Bo and Zilong Zheng and Kai Yu and Eng-Siong Chng and Xie Chen},
  year          = {2026},
  eprint        = {2609.08936},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2609.08936}
}