OpenClaw RLC — Conversational AI Agent Training Extension

OpenClaw-RL — What It Is

• Official Repo: https://github.com/Gen-Verse/OpenClaw-RL
• Authors: Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang (Princeton), Ling Yang — from the Gen-Verse research group
• License: Not explicitly stated in search results (repo is open-source; the parent OpenClaw project is MIT-licensed)
• Last Major Update: March 11, 2026 (Track 2 released with terminal, GUI, SWE, and tool-call agent support)
• Initial Release: February 26, 2026 (v1)
• arXiv Paper: https://arxiv.org/abs/2603.10165 — "OpenClaw-RL: Train Any Agent Simply by Talking" (published ~March 10, 2026)
• Parent Project (OpenClaw): https://github.com/openclaw/openclaw — 280,000+ GitHub stars as of early March 2026, MIT-licensed, created by Peter Steinberger
• GitHub Stars for OpenClaw-RL: Not independently confirmed at time of research; the parent OpenClaw repo has 280,000+ stars

OpenClaw-RL is a fully asynchronous reinforcement learning framework that turns everyday conversations into training signals for personalized AI agents, and supports training general agents with large-scale environment parallelization.

OpenClaw-RL v1 was released on February 26, 2026 as a fully asynchronous RL framework for training personalized AI agents from natural conversation feedback. A major update on March 11, 2026 released a new combination method and Track 2, featuring scalable RL implementations for general agent settings across terminal, GUI, SWE, and tool-call scenarios.

How It Works — Technical Architecture

OpenClaw-RL is not a simple memory or prompt-injection system. It actually modifies the weights of a locally-hosted LLM in real time. Here is the full pipeline:

The Core Insight: Every agent interaction generates a next-state signal — the user reply, tool output, terminal or GUI state change that follows each action — yet no existing agentic RL system recovers it as a live, online learning source. OpenClaw-RL is built on the observation that next-state signals are universal, and policy can learn from all of them simultaneously.

Async Decoupled Architecture: OpenClaw-RL decouples agent serving, rollout collection, PRM judging, and policy training into independent async loops. None of them block one another — the model serves requests while training runs in the background, and PRM evaluation happens concurrently with new conversations.

The Four-Stage Pipeline:
1. Intercept: Most RL-for-LLM systems assume centralized, batch-mode training with pre-collected datasets. OpenClaw-RL takes a fundamentally different approach: it wraps your self-hosted model in OpenClaw as an OpenAI-compatible API, intercepts live multi-turn conversations, and continuously optimizes the policy in the background — all without interrupting your usage.
2. Score: When the next turn arrives, its user/environment message serves as the "next state" for the previous turn. A Process Reward Model (PRM) judges the previous response quality given the next state. It produces m independent evaluations via majority vote, scoring each turn as +1 (good), -1 (bad), or 0 (neutral). The majority-voted score becomes the scalar reward for that turn.
3. Train (Binary RL / GRPO): Binary RL converts evaluative signals into scalar process rewards, while OPD converts directive signals into token-level advantage supervision. Combining the two yields significant optimization gains.
4. Distill (OPD): Hindsight-Guided OPD converts directive signals into token-level advantage supervision by extracting textual hints from the next state and constructing an enhanced teacher context, where rich textual feedback provides directional guidance for improvement.

Privacy: The entire stack (model, PRM, training) runs on your own infrastructure. Conversation data never leaves your system. No external API keys required.

Default Model: OpenClaw-RL wraps Qwen3-4B by default with a 32K context window, though the architecture is model-agnostic.

Hardware Requirements: The default configuration requires 8× GPUs, configurable via NUM_GPUS, ACTOR_GPUS, ROLLOUT_GPUS, PRM_GPUS environment variables.

Empirical Results: The combined method (Binary RL + OPD) achieves the strongest optimization performance. On-policy distillation shows delayed gains due to sparse training samples, while binary RL alone provides only marginal improvement. After 36 problem-solving interactions in the student setting, the agent learns to avoid obviously AI-like phrasing, such as using words like "bold" or producing overly structured, step-by-step responses.

Codebase Structure:

openclaw-rl/
├── README.md
├── run_qwen3_4b_openclaw_rl.sh       # Launch script
├── openclaw_api_server.py            # FastAPI proxy + PRM scoring + sample submission
├── openclaw_rollout.py               # Async rollout worker (bridges API server ↔ SLIME trainer)
└── results/                          # Runtime records (auto-created)

Dependencies: Built on top of Slime (Tsinghua's THUDM training framework), SGLang (model serving), and OpenClaw itself. The RL backbone comes from Slime, Tsinghua's THUDM lab framework that already powers training for GLM-family models and has built up over 4,400 GitHub stars.

Try It Yourself

⚠️ Hardware Warning: The default config requires 8× H100-class GPUs. This is a research-grade setup. Consumer GPU paths are not yet documented. Cloud GPU rental (e.g., Lambda Labs, RunPod) is the realistic path for most users.

Step 1: Clone the repo and set up the environment

git clone https://github.com/Gen-Verse/OpenClaw-RL.git
cd OpenClaw-RL
# Follow environment setup in ./instructions/README.md
# Requires: Python 3.10+, CUDA, SGLang, Slime

Step 2: Launch the RL server (Binary RL — recommended combo method)

cd slime
bash ../openclaw-combine/run_qwen3_4b_openclaw_combine.sh

This starts the Qwen3-4B model behind an OpenAI-compatible API at http://<HOST_IP>:30000/v1.

Step 3: Configure OpenClaw to route to your RL server

Open your openclaw.json and add under "models" → "providers":

{
  "models": {
    "providers": {
      "qwen": {
        "baseUrl": "http://<HOST_IP>:30000/v1",
        "apiKey": "your-SGLANG_API_KEY",
        "api": "openai-completions",
        "models": [
          {
            "id": "qwen3-4b",
            "name": "Qwen3 4B",
            "reasoning": true,
            "input": ["text"],
            "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
            "contextWindow": 32768,
            "maxTokens": 8192
          }
        ]
      }
    }
  }
}

Replace <HOST_IP> with your RL server's IP address.

Step 4: Install OpenClaw (bundled version from the repo)

# Use the OpenClaw version bundled in the OpenClaw-RL repo
# (not the latest from npm — the bundled version is patched for RL integration)
npm install -g openclaw@latest   # or use the bundled version per repo instructions
openclaw onboard --install-daemon

Step 5: Start chatting — training happens automatically

# Just talk to your agent normally via Telegram, WhatsApp, or WebChat
# The RL server will automatically:
# - Collect conversation trajectories
# - Compute rewards via PRM
# - Train the model in the background

Pro tip: Provide frequent feedback (e.g., 👍/👎) to help the model optimize effectively. For OPD mode, provide concrete feedback such as "you should have checked the file first" or "don't use that library".

What The Creator Didn't Mention

1. The hardware barrier is severe. The elephant in the room is the hardware barrier. Eight H100-class GPUs puts this firmly in "well-funded research lab or startup" territory. Individual developers and hobbyists — the exact people who might want a personalized agent — are priced out. There's no quantized mode, no CPU fallback, no guidance on whether consumer GPUs work at all.

2. This is actual weight modification, not just memory/prompting. OpenClaw's existing skill ecosystem handles a wide range of use cases without touching model weights at all. If the agent keeps forgetting preferences, that's a memory problem. If it doesn't know how to handle a specific workflow, that's a skill problem. Both are solvable at the prompt and context layer. Where RL becomes interesting is when the failure pattern lives deeper in the model's reasoning itself. OpenClaw-RL targets that deeper layer.

3. Only Qwen3-4B is validated so far. The current release only validates on Qwen3-4B. Broader model family support is on the roadmap but not yet released.

4. LoRA and low-precision training are not yet supported. The roadmap includes LoRA Training and low-precision training/inference, but these are not yet released. This means full-precision fine-tuning is the only current option — further increasing VRAM requirements.

5. Reward signal quality is a real risk. As noted by practitioners: a sloppy reward signal can train in a regression loop at speed. The PRM majority-voting helps, but there are no documented rollback mechanisms if training degrades the model.

6. The parent OpenClaw project is massive and complex. With nearly 500,000 lines of code, 53 config files, and 70+ dependencies, OpenClaw is the most feature-complete option. Using OpenClaw with Claude as the backing LLM runs approximately $80–120 per month in API costs for active agents. OpenClaw-RL sidesteps the API cost by running a local model, but adds GPU infrastructure cost instead.

7. Alternatives exist for personalization without GPU training:
- OpenClaw's built-in skill + memory system — stores preferences in files, no GPU needed
- IronClaw (https://github.com/nearai/ironclaw) — Rust-based OpenClaw-inspired implementation focused on privacy/security with WASM sandboxing
- NanoClaw — lightweight TypeScript alternative (~21,500 stars) for lower-resource environments
- Prompt-layer personalization — skills like "store this as a skill" in base OpenClaw handle most personalization use cases without any RL