MolmoSpaces Adapter¶

Wraps MolmoSpaces as llenvs MDP environments. MolmoSpaces is a large-scale robot manipulation and navigation research platform built on MuJoCo. It provides 8 benchmark tasks (pick, place, navigate, open/close, etc.) with multi-view cameras, proprioceptive sensors, and continuous robot control.

The adapter provides temporal abstraction: each llenvs step() = one high-level VLM decision, executed over many physics steps by MolmoSpaces' IK solver and low-level controller.

Installation¶

pip install git+https://github.com/allenai/molmospaces.git

MolmoSpaces requires MuJoCo. See the MolmoSpaces repository for full installation instructions.

Quick Start¶

from llenvs.adapters.molmospaces import MolmoSpacesAdapter

adapter = MolmoSpacesAdapter()

env = adapter.get_environment("molmospaces:pick", max_steps=30)

state, info = env.reset(options={"task_index": 0})
print(state.observation.prompt)          # Task instruction + proprioception
print(len(state.observation.images))     # 2 (front + wrist cameras)
print([t.name for t in state.observation.available_tools])
# ['move_end_effector', 'grasp', 'release']

from llenvs.core import Action
from llenvs.core.tools import ToolCall

# Move the end-effector to the object
call = ToolCall(id="c1", name="move_end_effector", arguments={"x": 0.5, "y": 0.1, "z": 0.3})
result = env.step(state, Action(tool_calls=(call,)))

# Grasp the object
call = ToolCall(id="c2", name="grasp", arguments={})
result = env.step(result.next_state, Action(tool_calls=(call,)))

With Evaluation Runner¶

from llenvs.evaluation.runner import run_tool_evaluation

result = run_tool_evaluation(
    environment=env,
    backend=backend,
    task_indices=list(range(50)),
)
print(f"Success rate: {result.success_rate}")

Robot Control Tools¶

The VLM interacts with the robot through function-calling tools. Each tool call triggers many internal physics steps.

Tool	Parameters	Description
`move_end_effector`	x, y, z, qw?, qx?, qy?, qz?	Move end-effector via IK solver
`grasp`	(none)	Close gripper
`release`	(none)	Open gripper
`navigate_to`	x, y	Move robot base (mobile robots only)
`set_joints`	positions (array)	Direct joint control (joint controller only)

navigate_to is only available for mobile robots (e.g., RBY1). set_joints is only available when controller_type="joint".

Tool Execution¶

Each tool call is executed through MolmoSpaces' IK solver and controller:

move_end_effector → IK solver computes joint targets → controller steps physics until convergence
grasp/release → gripper actuated directly
navigate_to → base controller steps physics to target position
set_joints → controller steps physics to target joint positions

The max_physics_steps_per_action parameter (default: 200) limits the physics budget per tool call.

Observation Space¶

Each observation includes:

Multi-view camera images: RGB images from configurable cameras (default: front + wrist)
Depth images (optional): Depth maps from each camera
Proprioception (optional): End-effector position/orientation, gripper state, joint positions as text
Task instruction: Natural language description of the task

# Configure observations
env = adapter.get_environment(
    "molmospaces:pick",
    cameras=("front", "wrist", "overhead"),
    image_format="jpeg",
    image_quality=80,
    include_proprioception=True,
    include_depth=True,
)

Camera Configuration¶

Parameter	Type	Default	Description
`cameras`	`tuple[str, ...]`	`("front", "wrist")`	Camera views to render
`image_format`	`str`	`"jpeg"`	`"jpeg"` or `"png"`
`image_quality`	`int`	`80`	JPEG quality (1-100)
`include_depth`	`bool`	`False`	Include depth maps as additional images

JPEG is the default since robot camera images are natural photos where JPEG compression is efficient.

Task Presets¶

Task	Max Steps	Description
`navigate-to`	20	Navigate the robot base to a target location
`pick`	30	Pick up a specified object
`pick-and-place`	40	Pick up an object and place it at a target
`pick-and-place-next-to`	40	Pick up an object and place it next to another
`pick-and-place-color`	40	Pick up a color-specified object and place it
`open`	30	Open a container or drawer
`close`	30	Close a container or drawer
`open-door`	30	Open a door by manipulating its handle

Rewards¶

Native Reward¶

MolmoSpacesReward wraps task.get_reward(). Uses STEP type for intermediate dense rewards and OUTCOME type at terminal steps.

Success Reward (opt-in)¶

MolmoSpacesSuccessReward provides binary 0/1 from task.judge_success(). OUTCOME type, only at terminal. Available via extra_rewards:

from llenvs.adapters.molmospaces import MolmoSpacesSuccessReward

env = adapter.get_environment(
    "molmospaces:pick",
    extra_rewards=(MolmoSpacesSuccessReward(),),
)

Controller Configuration¶

Parameter	Type	Default	Description
`controller_type`	`str`	`"ik"`	`"ik"` or `"joint"`
`robot_type`	`str`	`"franka"`	`"franka"` (stationary) or `"rby1"` (mobile)
`max_physics_steps_per_action`	`int`	`200`	Physics step budget per tool call
`is_mobile`	`bool`	`False`	Auto-set from `robot_type`

# Mobile robot with joint control
env = adapter.get_environment(
    "molmospaces:navigate-to",
    robot_type="rby1",         # auto-enables is_mobile
    controller_type="joint",   # enables set_joints tool
)

Container Deployment¶

MolmoSpaces requires MuJoCo and GPU rendering. For isolated deployment:

from llenvs.container import create_container_environment, ContainerConfig

config = ContainerConfig(
    runtime="docker",
    image="molmospaces-env:latest",  # custom image with MuJoCo + GPU support
    port=8080,
)
env = create_container_environment(env_config, container_config=config)

Parameters¶