Hugging Face's logo
Back to Articles

Agent-Factory-3: Training GPT-OSS-120B with Agentic RL on 4×H100

Community Article
Published May 12, 2026
Yi-Chia Chen's avatar
Yi-Chia Chen
ycchen

We are open-sourcing agent-factory-3, an agentic RL framework built specifically for GPT-OSS.

The goal of this release is straightforward: to make agentic RL for GPT-OSS-120B actually runnable on a single node with limited GPU resources. By combining MXFP4 MoE backward, attention LoRA, Pipeline RL, DFlash rollout acceleration, MoE routing replay, a Harmony token loop, and MCP tools, we completed agentic RL training for GPT-OSS-120B on 4×H100, reaching end-to-end throughput of approximately 6000 tokens/sec.

As far as we know, this is one of the first open-source stacks to publicly demonstrate end-to-end single-node agentic RL for GPT-OSS-120B.

Repo: https://github.com/ycchen-tw/agent-factory-3

The repo currently provides two training examples:

  • Wordle: improves the gpt-oss-120b win rate from 65% to 90% in 4 hours (W&B).
  • Minesweeper (8×8, 8 mines): improves the gpt-oss-120b win rate from 20% to 45% in 10 hours (W&B).

Training curves on Wordle and Minesweeper

Figure: Training curves on the two demo tasks. Left: Wordle solve rate vs. training step. Right: Minesweeper solve rate vs. wall-clock time (minutes). Both runs are on 4×H100.

These two tasks are still toy environments, but they exercise several core problems in agentic RL: multi-turn tool calls, long trajectories, sparse rewards, rollout tail latency, token/logprob alignment, MoE train/infer mismatch, and rollout-level debugging.

Table of contents

TL;DR

agent-factory-3 currently focuses on agentic RL training for GPT-OSS. Its main design points include:

  • MXFP4 MoE backward: GPT-OSS experts remain frozen in MXFP4, but training still needs activation gradients to pass through MoE layers so that attention LoRA receives correct gradients.
  • MXFP4-aware FSDP2: MXFP4 blocks / scales are packed into bf16 Parameters in a bit-preserving way, allowing FSDP2 to shard and all-gather them. During computation, they are unpacked and dynamically dequantized to bf16 or fp8.
  • Attention LoRA: the current recipe updates only the q_proj / k_proj / v_proj / o_proj LoRA adapters, while MoE experts and the router remain frozen.
  • Pipeline RL: rollout and training run in separate process groups and exchange samples through a queue, reducing trainer stalls caused by long-tail rollouts. It also supports prefix cache isolation to prevent new rollouts from hitting KV cache produced by old weights.
  • DDIS + staleness filter: token-level logprobs are saved during rollout; Direct Double-sided Importance Sampling is used during training for off-policy correction, and overly stale samples are discarded.
  • MoE routing replay: expert top-k indices are recorded during rollout and replayed during training, reducing inference / training routing mismatch.
  • DFlash rollout acceleration: diffusion-based speculative decoding is integrated into the SGLang rollout pipeline, with target logprobs, routing capture, and long-context draft-model adjustments required by RL.
  • Harmony token-in-token-out: tokens generated during rollout go directly into the training loss, avoiding action-level mismatch caused by parsing, serialization, and retokenization.
  • MCP environments: each rollout can launch its own MCP server, making it easy to plug in stateful tools such as Wordle, Minesweeper, Python, and web search.
  • Ray-free single-node stack: the system uses multiprocessing, Hugging Face Accelerate, PyTorch FSDP2, and SGLang HTTP servers to reduce single-node deployment and debugging cost.

Introduction to GPT-OSS

GPT-OSS is OpenAI’s open-weight model family, including gpt-oss-120b and gpt-oss-20b. In OpenAI’s gpt-oss announcement, gpt-oss-120b has 117B total parameters, 5.1B active parameters per token, and a 128k context length; gpt-oss-20b has 21B total parameters, 3.6B active parameters per token, and a 128k context length. Both are MoE models, and each token activates 4 experts.

GPT-OSS uses the Harmony response format to represent conversation structure, reasoning output, function calls, and tool namespaces. The OpenAI Harmony Cookbook and openai/harmony both make it clear that GPT-OSS will not work correctly if used directly without applying the Harmony format.

Another important property is MXFP4 quantization. The Hugging Face Transformers MXFP4 docs describe MXFP4 as the 4-bit floating-point format used by GPT-OSS 120B / 20B. This is also one of the key reasons why gpt-oss-120b can fit on a single 80GB GPU and gpt-oss-20b can run within 16GB of memory.

These properties make GPT-OSS well suited for agentic RL, but they also make the training system more complex. We need to handle MXFP4 MoE, Harmony tokens, tool calls, long contexts, multi-turn environment interaction, MoE routing mismatch, and rollout tail effects at the same time.

Why build a new RL framework?

Development of agent-factory started in February 2025. The original goal was tool-integrated RL, which is now more commonly referred to as agentic RL.

At first, I prototyped on top of TRL and tried to add tool-use training to an existing RLHF pipeline. After implementing it, I found that the main bottleneck in agentic RL was not the algorithm alone, but the entire system stack.

A single agentic trajectory can include multiple rounds of model outputs, tool calls, tool responses, environment state updates, timeouts, error recovery, and the final reward. Trajectory lengths also vary greatly: some end after a few hundred tokens, while others approach 100k tokens.

This directly affects several system-design choices: rollout must not be tied to the slowest trajectories; training must see the exact tokens sampled at rollout time; metadata such as logprobs, routing indices, and entropy needs to be collected during rollout; tool environments need isolation; and debugging cannot rely only on scalar metrics.

Many existing RL frameworks primarily target multi-node, large-scale training, with typical setups based on Ray resource management and tens to thousands of GPUs. agent-factory-3 chooses a narrower direction: single-node first, QLoRA first, and agentic-environment first.

MXFP4 MoE backward

One important advantage of GPT-OSS is that its MoE weights use MXFP4 quantization. Since most parameters in an MoE model live in the experts, keeping expert weights in MXFP4 greatly reduces the memory footprint. This is also one of the key reasons why gpt-oss-120b can be deployed on a single 80GB GPU.

On the inference side, this design is very effective. GPT-OSS’s efficient inference path can directly use MXFP4 MoE kernels, fusing routing, gather, expert matmul, scatter, and related operations, without keeping all experts dequantized into bf16 resident memory.

However, these kernels are mainly designed for inference forward. For RL training, forward alone is not enough. Even if we do not update the MXFP4 expert weights, the gradient of the training loss still needs to pass through the MoE layers and return to the hidden states. Otherwise, the trainable modules before and after the MoE cannot receive correct gradients.

What agent-factory-3 adds is this MXFP4 MoE training path: experts stay frozen in MXFP4, while MoE layers support activation backward. In other words, we do not dequantize GPT-OSS back into bf16 for full fine-tuning. Instead, we keep the MXFP4 memory advantage while allowing frozen MoE experts to participate in the training computation graph.

Dynamic dequant + custom autograd

GPT-OSS expert weights are stored as MXFP4 blocks / scales. During forward, agent-factory-3 unpacks the MXFP4 weights inside each MoE layer, dynamically dequantizes them into bf16 or fp8, and then calls MoE matmul. MoE routing, gather, scatter, and top-k reduction follow the inference-aligned kernel path as much as possible.

The key point in backward is to avoid expanding the entire model’s experts into bf16 for a long time. DequantMoEMatmul only dequantizes the corresponding expert weights when the current layer needs computation, and releases them immediately after computation finishes. The autograd context stores lightweight information such as the module reference, projection name, and routing metadata, rather than the full dequantized weights.

During backward, FSDP2 all-gathers the packed MXFP4 parameters for the current layer again. DequantMoEMatmul then retrieves blocks / scales from the packed parameters, dequantizes them into bf16 or fp8, computes activation gradients, and returns None gradients for expert weights. Experts therefore remain frozen, while gradients can still pass correctly through the MoE layer back to preceding hidden states.

With this design, peak memory mainly comes from the dequantized bf16 expert weights for the layer currently being computed, rather than bf16 experts for the entire model. In other words, we pay the memory cost of one-layer dequantization, not full-model dequantization.

Dynamic dequantization adds extra compute cost, but in RL training this can be amortized with a larger packed batch size. MoE weight dequantization is a fixed per-layer cost. When the batch contains enough effective tokens, most time is still spent on expert matmul, attention, and logprob / entropy computation. This is also one reason agent-factory-3 implements sequence packing and a large token capacity.

FSDP2-compatible MXFP4 storage

FSDP2 natively manages floating-point nn.Parameters. In MXFP4 checkpoints, expert weights are uint8 blocks and uint8 scales, which cannot be used directly as ordinary FSDP2 parameters.

agent-factory-3 follows an approach similar to bitsandbytes: uint8 bytes are packed into bf16 Parameters in a bit-preserving way. FSDP2 sees bf16 parameters, so it can shard, all-gather, and reshard them. The MoE forward path sees the unpacked uint8 blocks / scales.

Here, bf16 is only a container, not the numerical representation. Before computation, the bytes are reinterpreted back into uint8 and then MXFP4-dequantized.

This design also exposes a problem that does not appear in ordinary bf16 training: packed bytes may happen to form bf16 NaN bit patterns. If a copy path canonicalizes bf16 NaNs, the underlying bytes will be rewritten, immediately corrupting the expert weights. agent-factory-3 protects the bf16 copy path in FSDP2 to ensure packed bit patterns remain unchanged during shard / all-gather.

Hugging Face quantizer integration

To integrate with Transformers, we implemented the mxfp4_bf16_dequant quantizer through the Hugging Face HfQuantizer registration mechanism. When the model is loaded, the quantizer replaces GPT-OSS expert modules with MXFP4 dynamic-dequant expert modules and converts the _blocks / _scales in the checkpoint into FSDP2-manageable packed parameters.

The usage remains close to a standard Transformers workflow:

config = AutoConfig.from_pretrained(MODEL_PATH)
config.quantization_config = Mxfp4Bf16DequantConfig(
    dequant_dtype="bf16",
).to_dict()

model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    config=config,
    dtype=torch.bfloat16,
)

On GPUs that support FP8, dequant_dtype="fp8" can also be used. The current Wordle / Minesweeper recipes use bf16 dequantization by default; the FP8 path still needs more ablation.

Pipeline RL

The wall-clock time of agentic rollouts is highly unstable. Wordle may end in a few rounds; Minesweeper may require many steps; more complex agent tasks may also involve web requests, Python execution, timeouts, and error retries.

Synchronous RL waits for the entire rollout batch to finish before training starts. This is wasteful for agentic tasks because GPUs often end up waiting for the slowest few trajectories.

agent-factory-3 adopts a concurrent rollout / training design similar to PipelineRL. PipelineRL describes the problem as a trade-off between hardware efficiency and data on-policyness, and improves accelerator utilization through asynchronous data generation and model training.

The agent-factory-3 flow is:

SGLang rollout servers
    │  trajectories + logprobs + routing
    ▼
Orchestrator / SampleProcessor
    │  training samples
    ▼
FSDP2 Trainer
    │  weight sync
    ▼
SGLang rollout servers

The orchestrator continuously sends rollout jobs to SGLang servers, collects results, computes rewards, advantages, and filtering decisions, then pushes samples into sample_queue. The trainer pulls samples from the queue, applies the staleness filter, performs sequence packing and an FSDP2 train step, and syncs new weights back to SGLang according to the configuration.

The queue size is intentionally kept small to create backpressure. If rollout is too fast, the orchestrator blocks on the queue instead of accumulating unlimited stale samples. If training is too fast, the trainer naturally waits for new samples.

Prefix cache also requires special handling in Pipeline RL. Agentic rollout prompts often share a long common prefix, especially system prompts, developer instructions, tool schemas, and MCP descriptions. If rollouts from different training steps directly share prefix cache, a new rollout may hit KV cache produced by old weights. On the surface, the sample comes from the new weight version; in reality, the hidden states in the first part of the sequence are mixed with the old policy, making the off-policy degree worse than what the metadata suggests.

agent-factory-3 therefore supports prefix cache isolation. Through cache_salt_mode, prefix cache hits can be restricted to the same rollout, or to the same group. per_rollout is the most conservative setting and almost avoids cross-rollout stale KV entirely. per_group preserves cache reuse within the same prompt group while avoiding hits across steps and groups. This sacrifices part of the prefix-cache hit rate, but reduces hidden off-policy bias under asynchronous weight updates.

The GPU layout is adjusted by task. In the released recipes, the Wordle recipe uses 2 GPUs for SGLang TP=2 and 2 GPUs for FSDP2 training; the Minesweeper recipe uses three SGLang TP=1 servers plus 1 GPU for training. In other words, the main result in this post is a 4×H100 single-node end-to-end layout; the trainer itself and the rollout/training GPU ratio can be adjusted by task.

DDIS and staleness filter

Pipeline RL improves hardware utilization, but it also means samples are no longer perfectly on-policy. A rollout may be generated by weight version v, while training may already have advanced to v+k.

agent-factory-3 uses Direct Double-sided Importance Sampling (DDIS). The GLM-5 technical report uses DDIS in an asynchronous RL setting, applying token-level clipping / masking to control the gap between the rollout policy and the current policy.

During rollout, we save the logprob for each generated token:

log μ(a_t | s_t)

During training, the current policy forwards the same token sequence again and obtains:

log π_θ(a_t | s_t)

The importance ratio for each token is:

rt(θ)=exp⁡(log⁡πθ(at∣st)−log⁡μ(at∣st)) r_t(\theta) = \exp\left(\log \pi_\theta(a_t \mid s_t) - \log \mu(a_t \mid s_t)\right)

DDIS uses a double-sided trust region:

mt=1[1−ϵlow≤rt(θ)≤1+ϵhigh] m_t = \mathbf{1}\left[1 - \epsilon_{\text{low}} \le r_t(\theta) \le 1 + \epsilon_{\text{high}}\right]

The token-level objective can be written as:

L(θ)=−1Z∑tmtsg(rt(θ))Atlog⁡πθ(at∣st) \mathcal{L}(\theta) = -\frac{1}{Z}\sum_t m_t\mathrm{sg}\left(r_t(\theta)\right)A_t\log \pi_\theta(a_t \mid s_t)

where sg denotes stop-gradient, A_t is the advantage, and Z is the normalization divisor.

This has two practical advantages. First, it does not require an additional old-policy forward pass; the denominator has already been recorded during rollout. Second, the correction is token-level, so tokens whose ratios remain reasonable can still be kept within the same trajectory.

Before DDIS, there is also a sample-level staleness filter. Every sample carries a weight version. If it lags behind the current training step by more than max_staleness, it is dropped directly and never enters forward. The staleness filter blocks obviously outdated trajectories, while DDIS handles token-level drift among the remaining samples.

MoE routing replay

DDIS controls logprob ratio drift. MoE models have another source of drift: expert routing.

During rollout, the SGLang inference engine selects top-k experts based on router logits. During training, even with the same input tokens, the router may select different experts because of weight updates, kernel paths, or numerical boundary effects. Top-k routing is a discrete operation; replacing one expert changes subsequent hidden states and logprobs.

R3: Rollout Routing Replay proposes routing replay for MoE RL: record routing during rollout / inference and replay it during training to reduce the train / infer router discrepancy. agent-factory-3 adopts R3-style expert-identity replay: it captures top-k expert indices during rollout and replays the same expert identities during training, while gate weights are still recomputed by the current router.

agent-factory-3 implements routing replay for GPT-OSS. During rollout, SGLang returns the expert top-k indices for each generated token at every layer. For gpt-oss-120b, the shape is:

[num_generated_tokens, 36 layers, 4 experts]

During training, these indices are packed into [B, S, L, K] and passed into model forward. Each layer receives its own [B, S, K] routing indices, and the MoE MLP uses the captured indices to construct gather / scatter routing data. Router logits are still computed by the current model and used for the gate weights of the selected experts; expert identity is aligned with the rollout path.

This reduces unnecessary differences between train-time logprobs and rollout-time logprobs, especially under asynchronous weight updates.

DFlash rollout acceleration

The bottleneck in agentic RL is often rollout. For GPT-OSS-120B, each rollout requires long-context generation, multi-turn tool calls, logprob collection, and routing capture. After Pipeline RL separates rollout and training, the bottleneck becomes even more obvious: if the trainer is already waiting for samples, accelerating rollout directly improves end-to-end training speed.

The basic idea of speculative decoding is draft-and-verify: a smaller or faster draft path first generates multiple candidate tokens, and the target model verifies them in parallel. Two common foundational references in this direction are Leviathan et al.’s Fast Inference from Transformers via Speculative Decoding and Chen et al.’s Accelerating Large Language Model Decoding with Speculative Sampling. As long as the verification step preserves the target distribution, accepted tokens can still be treated as samples from the target policy, while the number of decoding steps is reduced.

This direction has already begun to be used in RL rollout. The slime speculative decoding docs describe how to use SGLang speculative decoding to accelerate rollout, and also support online training of MTP layers during RL to prevent the draft / target mismatch from quickly lowering the acceptance rate after policy updates. NeMo-RL speculative decoding also integrates EAGLE-3, MTP, external draft models, and other methods into rollout generation; the training path still computes logprobs, KL, and loss on the verifier policy.

DFlash takes a different route. EAGLE-3 / MTP are still closer to autoregressive drafting; the DFlash paper uses a lightweight block-diffusion draft model to generate a block of tokens in parallel, then passes it to the target LLM for verification. The vLLM speculators DFlash docs also describe DFlash as using a small diffusion-LLM draft model to predict a token block in a single forward pass, conditioned on target-model hidden states.

The main challenge in connecting DFlash to RL rollout is not speculative decoding itself, but rollout metadata. agent-factory-3 training samples require per-token target logprobs, MoE routing indices, entropy, and end reason. The DFlash decode path is not identical to the ordinary decoding path, so all this metadata must be aligned with the final accepted tokens before it can be used by DDIS, routing replay, and visualization.

Another challenge is long-context agentic rollout. Tasks such as Wordle and Minesweeper repeatedly insert tool observations, so the context grows round by round. If the draft model only has high acceptance length on short contexts, the acceleration effect will quickly degrade after several tool calls. The draft model’s own KV cache also cannot be too large, otherwise it erases the acceleration benefit under long contexts.

Therefore, we made several GPT-OSS agentic-rollout-specific adjustments: the DFlash path supports target logprobs, routing capture, and entropy; the draft model uses a hybrid SWA / full-attention design to reduce long-context KV cache; the attention-sink design draws on StreamingLLM, allowing SWA to retain early-context anchors such as system prompts and tool schemas under long contexts; and the draft model is fine-tuned on long-context agentic trajectories.

In Minesweeper training, rollout is a clear bottleneck. Enabling DFlash accelerated rollout by about 50%, and also accelerated end-to-end training by about 50%. Here, “lossless” means speculative decoding preserves the target-model sampling distribution; it does not mean the entire system has no numerical differences. No reward / win-rate regression was observed in these runs.

Harmony token loop and MCP environments

GPT-OSS uses the Harmony response format. The OpenAI Harmony Cookbook defines Harmony as token-level structure for roles, channels, recipients, tool calls, structured outputs, and more.

In agentic RL, action logprob must align with the action actually sampled during rollout. If rollout output is first parsed into a tool action and then serialized back into text for training-time retokenization, mismatches can easily occur.

agent-factory-3 therefore uses a token-in-token-out agent loop. The token ids generated during rollout go directly into the training sample. The parser is still responsible for detecting tool calls, dispatching MCP servers, identifying final answers, and determining end reasons; loss computation uses the original generated tokens.

Tool environments are integrated through the Model Context Protocol. Each dataset item can carry its own MCP config, and rollout launches a local or remote MCP server accordingly. In Wordle, each target word corresponds to an independent server. In Minesweeper, each seed corresponds to an independent board server. This per-rollout server pattern avoids environment-state leakage.

The agent loop also provides several budget controls: max_rounds, max_total_tokens, max_round_tokens, max_context_tokens, tool_call_timeout, and max_total_tool_time. These settings directly affect training-data quality. Timeouts, context overflow, and parse mismatches can all contaminate the reward signal.

Single-node system design

agent-factory-3 does not use Ray. For multi-node RL systems with many workers, Ray is very valuable; for a 4×H100 or 8×H100 single-node research workflow, deployment, versioning, logging, and debugging costs deserve higher priority.

agent-factory-3 uses a relatively direct combination:

  • Python multiprocessing manages the orchestrator and workers;
  • Hugging Face Accelerate manages trainer launch and FSDP2;
  • mp.Queue passes samples and control signals;
  • SGLang runs as independent HTTP servers;
  • the training environment and SGLang environment are separated to avoid version conflicts between inference kernels and training dependencies.

Sequence packing

Agentic trajectory lengths vary greatly. Naive padding wastes a large number of tokens. agent-factory-3 uses packed batching: multiple sequences are placed into the same token bucket, then fed to the attention kernel through cu_seqlens. The sampler balances token load across ranks to avoid one rank constantly receiving the longest sequences.

We have also tested longer-context settings. On 4×H100, the current pipeline setup supports high-token-capacity packed training, and we have also run stress tests close to 120k context. The public Wordle / Minesweeper recipes use more conservative context budgets. This matters for agentic RL because the context budget affects not only whether long trajectories can finish, but also the real behavior of rollout filtering, tool-observation retention, prefix-cache isolation, and training packing.

Fused logprobs / entropy

Another memory hot spot is logits. GPT-OSS has a large vocabulary; directly materializing [B, T, V] logits for packed long sequences quickly exhausts memory.

agent-factory-3 supports fused linear cross-entropy / logprob computation, conceptually similar to Liger Kernel: it fuses the lm_head matmul with logprob / entropy / loss computation in blocks, avoiding materialization of the full logits tensor.

RL loss requires per-token outputs: completion mask, advantage, importance ratio, DDIS mask, and entropy metrics. This differs from the scalar mean loss in ordinary SFT, so the loss backend must support per-token reduction and custom weighting.

Visualization and debugging

Agentic RL is difficult to debug using only scalar metrics. A reward drop may come from policy degradation, but it may also come from tool-format drift, context overflow, timeouts, reward-parser failure, an excessively high DDIS mask rate, or routing replay not being aligned.

agent-factory-3 renders the latest rollouts into self-contained HTML and uploads them to W&B in real time. Opening it shows each trajectory’s system / developer / user messages, assistant reasoning, tool calls, tool outputs, final answer, end reason, reward, advantage, rounds, tokens, runtime, weight version, and filter status.

Rollout Visualizer

Rollout Visualizer

Figure: Rollout visualizer in W&B. The left side shows the group / rollout list with reward and advantage; the right side shows the full trajectory, including assistant reasoning, tool calls, tool outputs, rounds, tokens, runtime, and the weight version used by the rollout.

It also records several metrics for asynchronous agentic RL: DDIS mask rate, importance-ratio distribution, train/gen logprob KL, sample staleness, end-reason distribution, filtered all-failed / all-solved groups, rollout tokens/sec, and training tokens/sec.

Together with the rollout viewer, these metrics usually make it much faster to locate problems than by looking at the reward curve alone.

Results

Wordle

The Wordle example uses 2,314 English 5-letter answers. Each target word generates one group, and multiple rollouts within the group use different random seeds. The reward is binary: if any tool output shows won=True, the reward is 1.0. Groups where all rollouts succeed or all fail are filtered out by default, because group-relative advantage provides no useful signal there.

On 4×H100, the win rate of gpt-oss-120b on Wordle improved from 65% to 90%. The full run is available on W&B.

Minesweeper

The Minesweeper example uses 8×8 boards with 8 mines and no-guess solvable boards. The model observes local information through a reveal(x, y) tool and decides the next move based on the board state. The reward is also binary: if the tool output contains state: won, reward=1.0.

This task requires more long-horizon reasoning and decision-making under partial observation than Wordle. Trajectories are longer, tool interactions are more frequent, full context length can exceed 30k tokens, and it is easier to expose problems in rollout tail latency, context budget, routing replay, and DFlash acceleration.

On 4×H100, the win rate of gpt-oss-120b on Minesweeper improved from 20% to 45%. In this run, DFlash accelerated rollout by about 50% and end-to-end training by about 50%. The full run is available on W&B.

Limitations

agent-factory-3 is still an experimental framework. This version mainly targets GPT-OSS; MXFP4 MoE backward, the Harmony token loop, routing replay, and the DFlash draft model are all highly tied to GPT-OSS’s architecture and format. Other models can theoretically be integrated, but they are not the main support target at this stage.

The public results currently come mainly from the attention LoRA recipe. We update only the q_proj / k_proj / v_proj / o_proj LoRA adapters, while keeping MoE experts and the router frozen. This setting allows GPT-OSS-120B to be trained on a single node, but we have not yet run full ablations against full fine-tuning, MoE LoRA, router tuning, or other LoRA target modules.

The public examples are still controlled environments. Wordle and Minesweeper are useful for validating the pipeline, but they do not represent software engineering, web agents, research agents, or longer-horizon real-world tasks. DDIS thresholds, max staleness, routing replay, the DFlash draft model, and the rollout/training GPU ratio also still need more systematic comparison.

Finally, the codebase and documentation still need cleanup. agent-factory-3 was built rapidly under competition and experimental pressure. Parts of the code and documentation also used AI-assisted development, so there may be less-tested code paths, outdated comments, inconsistent naming, or places where the documentation does not fully match implementation details. At this stage, we recommend starting from the provided examples and carefully checking any training path you modify.

Conclusion

agent-factory-3 solves a concrete problem: how to make GPT-OSS-120B agentic RL actually run on a single node with limited GPU resources.

To do that, we integrated MXFP4 MoE backward, FSDP2, attention LoRA, Pipeline RL, DDIS, routing replay, DFlash speculative decoding, the Harmony token loop, MCP environments, sequence packing, fused logprob computation, and W&B rollout visualization into a single stack.

Current results show that GPT-OSS-120B can perform agentic RL on 4×H100, and GPT-OSS-20B can also be trained on a single RTX 4090. We hope this makes it possible for more researchers without large GPU clusters to explore large-model agentic RL.

Repo: https://github.com/ycchen-tw/agent-factory-3

Acknowledgements

Special thanks to the Kaggle AI Mathematical Olympiad - Progress Prize 3. The goal of single-node GPT-OSS-120B agentic RL was largely shaped by this competition: adapting an open-weight reasoning model quickly through tools and RL under constrained hardware, limited time, and a strict submission environment. The final competition result was not ideal, but the agent-factory-3 system that emerged from the process is valuable to us, and we hope it will also help future open agentic RL research.

Special thanks as well to Professor Chu-Song Chen of AI² Lab and Professor Daisuke Kawahara of Waseda University Kawahara Lab for their guidance on the research direction and support with compute resources.

agent-factory-3 builds on many open-source efforts. We thank GPT-OSS, Harmony, SGLang, DFlash, Liger Kernel, Hugging Face Transformers / Accelerate / PEFT, FastMCP, and the open-source ML systems community.

We also thank open-source RL frameworks such as TRL, slime, verl, and ROLL. agent-factory-3 takes a single-node-first, Ray-free, GPT-OSS-specific path, but these projects provided many references for post-training recipes, rollout / training system design, engine integration, and large-model RL engineering.

Special thanks also to zhaochenyang20/Awesome-ML-SYS-Tutorial. The ML systems blogs collected there were very helpful for both the development and writing of agent-factory.

Further reading and related links

Community

Sign up or log in to comment