Hello, I'm
RL Algorithms & RL-Infra · Agentic-RL
Apodex
I work at Apodex on RL algorithms and Agentic-RL, backed by the RL-Infra I build. I develop the miles-based RL training framework behind apodex-v1.0 and v1.1 — a fully-asynchronous stack that decouples training from rollout, evaluates without pausing training, and reclaims idle GPUs for rollout via hybrid-colocate — plus miles-values, a dedicated SAO (Async Single-rollout PPO) framework. Earlier I built async_openrlhf, a fully-async RLHF pipeline extending OpenRLHF. My research spans LLM reasoning, MCTS, agentic scaling laws, and reasoning compression.
RL Algorithms & RL-Infra (Agentic-RL)
Math-Reasoning & Agentic-RL
AIGC Safety
Content Safety, Public Opinion Analysis, Personal Privacy, LLM Safety
Department of Electronic Engineering
Bachelor's Degree
apodex-v1.0 and apodex-v1.1, Apodex's flagship agentic models, are trained with a fully-asynchronous, high-throughput Agentic-RL framework I develop on top of miles. On the same stack I also built miles-values, a dedicated SAO (Async Single-rollout PPO) framework.
Training workers and rollout engines run on independent loops, communicating only through a shared data buffer and output queue, so neither ever blocks the other. Off-policy staleness is bounded by two-level (cross-group + per-slot) filtering and a configurable staleness threshold, with cooperative-abort weight sync over NCCL between iterations.
A dedicated eval pool hot-reloads the rolling training checkpoints and evaluates in the background, so evaluation never interrupts the training loop (“training-data roll”). Eval failures are isolated and can never abort training, keeping throughput steady while metrics stream continuously.
When the buffer has no consumable data, colocated training GPUs onload as generation engines and join the rollout routing pool — then offload again for the optimizer step. Idle training cards become extra rollout throughput, driving near-full cluster utilization with dedicated remote engines handling steady-state generation.
Eval accuracy over RL steps — HL-Gauss value loss + SAO on an agentic task. Raw (light) vs. EMA-smoothed (bold); dataset names withheld.
Top-venue publications accepted in 2025–2026
A family of open research agents (MiroThinker-1.7 & h1) that scale long-horizon, verification-centric reasoning, trained with our fully-asynchronous RL stack.
An evidence-assembly framework that lets deep-research agents accumulate and verify evidence at scale for reliable long-horizon research.
Discovering scaling laws for agentic reinforcement learning where LLMs spontaneously learn to execute code for mathematical reasoning.
Rethinking how to scale agent interactions for long-horizon research tasks.
A comprehensive gym environment for training and evaluating generalist large language models.
A novel self-play framework that advances agent capabilities through search without external supervision.
A dual-perspective dynamic preference learning framework that adaptively reweights samples by jointly considering data quality and model performance evolution during training.
A comprehensive survey of reasoning LLMs, tracing the evolution from fast intuitive (System 1) to slow deliberate (System 2) reasoning paradigms.
A Ray-based open-source framework enabling scalable RLHF training, widely adopted by the research community.
Showing that LLMs can bootstrap inductive reasoning capability through deductive self-teaching.
An efficient method to compress long chain-of-thought reasoning while preserving accuracy.
Full list on Google Scholar
Technical writing on LLM reasoning, MCTS, and reinforcement learning
Research interests: Probabilistic Graphical Models, Reinforcement Learning, NLP
SAO: Async Single-rollout PPO — part 1
SAO: Async Single-rollout PPO — part 2
Pretraining the value model to warm-start RL
System 1 vs System 2 thinking paradigm with MCTS for LLM reasoning
Insights on reproducing OpenAI o1 with long-CoT distillation and DPO
Reward shaping, domain randomization, and what zero-RL really optimizes
Deep dive into RL implementation details: loss, gradient accumulation, GRPO variance
Simplifying LLM-RL: start minimal, add complexity only when needed
Zero-shot tool-integrated reasoning with RL — small models, big results
Async rollout + pipeline: 4x faster training built on OpenRLHF
Training-inference consistency for MoE models under RL
Stabilizing off-policy RL training under high staleness