Yuxuan Lou

DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion

Yuxuan Lou*, Ziming Wu*, Yaochen Wang, Yong Liu, Yingxuan Ren, Fuming Lai, Shaobing Lian, Jie Tang, Yang You

MoST: Modality-Aware Mixture of Experts for Efficient Speech-Text Foundation Model

Yuxuan Lou, Kai Yang, Yang You

MoRS: Distill Large Language Model into Mixture of Reasoning Students In Submission

Yuxuan Lou, Yang You

EnvBridge: Bridging Diverse Environments with Cross-Environment Knowledge Transfer for Embodied AI

Tomoyuki Kagaya*, Yuxuan Lou*, Thong Jing Yuan*, Subramanian Lakshmi*, Jayashree Karlekar, Sugiri Pranata, Natsuki Murakami, Akira Kinose, Koki Oguri, Felix Wick, Yang You

RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents

Tomoyuki Kagaya*, Yuxuan Lou*, Thong Jing Yuan*, Subramanian Lakshmi*, Jayashree Karlekar, Sugiri Pranata, Natsuki Murakami, Akira Kinose, Koki Oguri, Felix Wick, Yang You

Cross-token Modeling with Conditional Computation

Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, Yang You

Developed DiffuSpeech, the first diffusion-based speech-text language model supporting both understanding and generation, introducing a "Silent Thought, Spoken Answer" paradigm where internal text reasoning informs spoken responses. Unified discrete text and tokenized speech under a single masked diffusion framework with modality-specific masking schedules. Constructed ThinkingTalk, the first speech QA dataset with paired text reasoning traces (26K samples, 319 hours), achieving state-of-the-art speech-to-speech QA accuracy (+9 points over best baseline) and best TTS quality among generative models (6.2% WER).

Developed MoST, a novel speech-text foundation model featuring a Modality-Aware Mixture of Experts (MAMOE) architecture; achieved competitive performance across multiple speech-text benchmarks using exclusively open-source data. Developed MoRS, a four-stage distillation method that compresses large language models (70B) into efficient MoE architectures (12B, 3B activated) while preserving reasoning capabilities, achieving up to +14.5% on benchmarks. Created the first framework to distill dense LMs into MoE without relying on pre-existing small models.

Developed RAP, a multimodal planning agent leveraging past successful experiences to enhance decision-making. Developed EnvBridge, a multimodal embodied agent transferring knowledge across diverse environments. Achieved SOTA on text-only environments (ALFWorld, Webshop) and significant improvements on multimodal robotics benchmarks (Franka Kitchen, Meta-World, RLBench).

Developed large-scale vision models: Sparse-MLP, Widenet based on Mixture of Experts. Proposed a fully-MLP architecture with conditional computation in two directions and extended MoE to the spatial dimension of image representation.

Making large AI models cheaper, faster, and more accessible. A collection of parallel components for distributed training. Managed and contributed to Colossal-AI examples.

A curated collection of Mixture of Experts papers and projects.

MoST

Official implementation of MoST, a speech-text foundation model with Modality-Aware Mixture of Experts (MAMOE) architecture for enhanced cross-modal understanding.

RAP

Official implementation of RAP, a multimodal planning agent leveraging past successful experiences to enhance decision-making.

Developed DiffuSpeech, the first diffusion-based speech-text language model supporting both understanding and generation. Collaborated on building the ThinkingTalk dataset and speech-text diffusion framework.

Colossal-AI and Colossal-AI Examples co-developer. Built an open domain dialog system with internet knowledge augmentation.

Data mining and cleaning of users' comments. Built ML models for emotion classification and deep learning abstractive text summarization.