LLEnvs

MDP-style access to evaluation environments for LLM research.

Stateless environments, multi-signal rewards, and 17+ adapters — one unified interface for benchmarking, evaluating, and training language model agents.

Get Started GitHub

---

• Stateless Environments

  ---

  step() is a pure function — state in, state out. Enables branching, checkpointing, and parallel exploration with no side effects.

  Core abstractions

• Multi-Signal Rewards

  ---

  Each step produces a SignalBundle with named, weighted signals carrying numeric rewards, textual feedback, or both.

  Evaluation guide

• 17+ Adapters

  ---

  ReasoningGym, HuggingFace, GEM, Gymnasium, WebShop, AlfWorld, Jericho, AgentGym, LMRL-Gym, Aviary, Verifiers, OpenEnv, and more.

  Browse adapters

• Tool & Function Calling

  ---

  Built-in tool environments with auto-monitoring, text-based tool parsers, and MCP support for open-source and API models.

  Tools guide

• RL Training Integration

  ---

  Drop-in scoring, dataset provision, and token masking for veRL, TRL, and OpenRLHF training loops.

  RL training guide

• Branching & Containers

  ---

  Checkpoint and branch environment states with zero-copy, process fork, or action replay. Run environments in Docker or isolated subprocesses.

  Branching guide

Design Principles

Stateless Environments

Traditional RL environments maintain internal state. llenvs uses stateless environments where step() is a pure function:

result = env.step(state, action)  # state is not modified

This enables branching, checkpointing, and parallel exploration.

Observation/Hidden Split

Each State separates what the model sees from what's needed for rewards:

State(
    observation: ObsT,   # Model sees this (prompt, messages)
    hidden: HiddenT,     # For reward computation (ground truth)
    metadata: StateMetadata
)

Multi-Signal Rewards

Transitions produce a SignalBundle with multiple named, weighted Signal instances. Each signal can carry a numeric reward, textual feedback, or both:

SignalBundle(signals=(
    Signal(name="correctness", reward_type=OUTCOME, reward=1.0, weight=1.0),
    Signal(name="format", reward_type=FORMAT, reward=1.0, weight=0.5),
    Signal(name="judge", reward_type=PROCESS, feedback="Consider edge cases..."),
))
# bundle.total = 1.0*1.0 + 1.0*0.5 = 1.5  (feedback-only signals don't contribute)
# bundle.feedback_texts() = ("Consider edge cases...",)