SciAgentGYM Adapter¶

Wraps SciAgentGYM as llenvs MDP environments for multi-step scientific tool-use evaluation.

Installation¶

SciAgentGYM is not on PyPI. Install from git:

pip install git+https://github.com/CMarsRover/SciAgentGYM.git

Quick Start¶

from llenvs.adapters.sciagentgym import SciAgentGymAdapter

adapter = SciAgentGymAdapter()
env = adapter.get_environment("sciagentgym", data_path="path/to/test_cases.json")

state, info = env.reset(options={"task_index": 0})
print(state.observation.prompt)
print([t.name for t in state.observation.available_tools])

from llenvs.core import Action
from llenvs.core.tools import ToolCall

# Use a scientific tool
call = ToolCall(id="c1", name="analyze_threshold", arguments={"energy": 0.001})
action = Action(tool_calls=(call,))
result = env.step(state, action)

# Submit final answer
action = Action(text="The threshold energy is \\boxed{2.6e5 GeV}")
result = env.step(result.next_state, action)
print(result.rewards.total)

Filter by Domain¶

# Only physics tasks
env = adapter.get_environment("sciagentgym:physics", data_path="path/to/data.json")

# Only chemistry tasks
env = adapter.get_environment("sciagentgym:chemistry", data_path="path/to/data.json")

With Evaluation Runner¶

from llenvs.evaluation.runner import run_tool_evaluation

result = run_tool_evaluation(
    environment=env,
    backend=backend,
    task_indices=list(range(50)),
)
print(f"Success rate: {result.success_rate}")

How It Works¶

Per-Task Tool Sets¶

Unlike fixed-tool environments, each SciAgentGYM task defines its own set of 3-16 domain-specific tools. On reset(), the adapter loads the task's tools from usage_tool_protocol (OpenAI function-calling format) and converts them to ToolDefinition objects.

Tool Execution Delegation¶

SciAgentGYM tools have internal state (history tracking, filesystem artifacts), so tool execution is delegated to MinimalSciEnv.step(). The adapter extends BaseToolEnvironment for tool infrastructure but lets SciAgentGYM handle all execution internally.

The flow:

reset() calls prepare_env_from_query() to create a MinimalSciEnv with tools for the task
Tools from usage_tool_protocol are converted to ToolDefinition via oai_tools_to_definitions()
step() converts llenvs ToolCall objects to SciAgentGYM ToolCall objects
Each tool call is executed via MinimalSciEnv.step() sequentially
Results are converted back to ToolResult objects

Termination¶

SciAgentGYM's MinimalSciEnv.step() never sets done=True. The adapter terminates on:

Text-only action: Treated as final answer submission (terminated=True)
Max steps: Truncation after max_steps tool-calling turns (default 30)

Rewards¶

Terminal steps extract \boxed{} from the model's response and compare to the gold answer:

Native scoring (when SciAgentGYM is installed): Uses calculate_answer_score() which supports numeric tolerance (5%), structural comparison, and partial scores
Fallback scoring: Uses llenvs BoxedExtractor with numeric tolerance and string comparison

Non-terminal steps return Signal(reward=None, reward_type=STEP).

Domains¶

Subject	Examples
Physics	Particle physics, optics, thermodynamics
Chemistry	Molecular analysis, reaction kinetics
Materials Science	Crystal structure, material properties
Life Science	Molecular biology, genetics
Astronomy	Stellar physics, cosmology
Statistics	Statistical analysis, hypothesis testing

Parameters¶