[📜 Paper] • [🐱 GitHub] • [🐦 Twitter] • [📕 Rednote]

Figure 1: An overview of the DAC-style inference and reward assignments in training, illustrated with a case study.

What this repo does: it trains LLMs to think in a divide-and-conquer (DAC) way via an end-to-end RL pipeline.

Core idea: instead of only learning sequential chain-of-thought (CoT), the policy learns to:

1. Divide a hard problem into structured subproblems, then
2. Conquer: solve subproblems and finally solve the original problem conditioned on those solutions.

Why it matters: CoT is strictly sequential and can hit a ceiling on very hard problems. DAC offers stronger test-time scalability by enabling structured exploration through decomposition.

• [2026/02/02] DAC-RL paper and repo are released.

At a high level, DAC-RL alternates between two roles (not necessarily two separate models):

• Divide stage: generate subproblems with a strict tag format.
• Conquer stage: solve subproblems sequentially and then solve the original problem, outputting a final boxed answer.

The repo implements this as a single training loop that creates two rollout batches per iteration (divide and conquer) and assigns rewards to both.

flowchart TD
  Q[Original question] --> D[Divide prompt: propose subproblems]
  D --> S[Parse & validate subproblems]
  S --> C[Conquer prompt: solve subproblems then original]
  C --> R[Rule-based verifier: answer correctness]
  R --> RC[Conquer reward]
  RC --> RD[Divide reward (derived from conquer outcomes)]
  RC --> PPO[PPO/GRPO-style policy update]
  RD --> PPO

We recommend using Conda to manage your environment. We use vLLM (0.10.1.1) to accelerate inference. Run the following commands to setup your environment:

conda create -n svs python=3.10.16
conda activate svs
pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu128 # CUDA 12.6 for example
pip install -r requirements.txt

The provided benchmark is data/dac-rl-benchmarks.jsonl.

Each line is a JSON object with keys:

• question: the problem statement
• ref_answer: the reference final answer
• data_source: benchmark name (e.g., AIME-2024-30, AIME-2025-30, ...)

The training script expects a parquet like data/DAPO-Math-17k.parquet. One simple way to download & save as parquet:

from datasets import load_dataset
import pandas as pd

ds = load_dataset('BytedTsinghua-SIA/DAPO-Math-17k')
# Adjust split names if needed.
train = ds['train'].to_pandas()
train.to_parquet('data/DAPO-Math-17k.parquet', index=False)
print('saved:', 'data/DAPO-Math-17k.parquet', 'rows:', len(train))

We provide evaluation scripts for both CoT and DAC inference. To use them, simply configure the model_name_or_path (default: Qwen/Qwen3-4B-Instruct-2507) and the data_path (by default, AIME 24, AIME 25, Beyond-AIME, and HMMT-25 are used for evaluation, as described in the paper) in scripts/eval_cot.sh and scripts/eval_dac.sh, and then run the following command:

bash scripts/eval_cot.sh # Evaluate model performance using chain-of-thought prompting
bash scripts/eval_dac.sh # Evaluate model performance using divide-and-conquer style reasoning

DAC evaluation does two-stage generation:

1. divide prompt → generate subproblems
2. conquer prompt → solve and answer original

We also present our complete training scripts, where the core implementation is the RayDACTrainer class in the verl/trainer/ppo/ray_trainer.py file. We provide the training data used in our paper in data. For example, to train the Qwen3-4B-Instruct-2507 model, run the following command:

bash scripts/run_dac_training.sh

Which file implements DAC training? - verl/trainer/ppo/ray_trainer.py → RayDACTrainer.

DAC-specific reward controls: These parameters directly encode the paper’s training-time alignment to DAC inference.

If you find this repository helpful, please consider citing our paper:

@misc{liang2026trainingllmsdivideandconquerreasoning,
      title={Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability}, 
      author={Xiao Liang and Zhong-Zhi Li and Zhenghao Lin and Eric Hancheng Jiang and Hengyuan Zhang and Yelong Shen and Kai-Wei Chang and Ying Nian Wu and Yeyun Gong and Weizhu Chen},
      year={2026},
      eprint={2602.02477},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.02477}, 
}

This project is released under the Apache License 2.0. See LICENSE.