Rusty Valve – Chemical plant valve benchmark where ACT scores 0/36

The project, developed by Rusty Valve, includes valve stations, a common scoring program, a scripted "teacher" that generates demonstrations, and learning baselines. The public exam applies to any robot and end-effector, with the teacher serving as a reference solution for OpenArm with an S-hook tool.

Release v0 includes the runtime, standalone station models, S-hook design files, and benchmark summaries. The full datasets, ACT checkpoints, and raw exam records are provided as separate materials for comprehensive testing and development.

Public Exam Rules

The benchmark operates under three strict rules for any robot or method: the valve must be held within 5° of the target stop for at least the final second; no hard collisions with environment objects exceeding 20N are permitted; and operations must be completed within a 90-second cutoff. Valve motion must result from physical robot contact.

The exam does not prescribe specific hands, surfaces, poses, or techniques, allowing for flexible approaches to valve operation while maintaining standardized evaluation criteria.

Initial Results

Testing under Public Exam v1 shows that ACT has not yet solved Level 1. However, increased scene coverage improved performance, with runs turning the valve more than 5° rising from 2 to 8 on unseen tests, and runs that never engaged the handle falling from 4 to 0.

Getting Started

Users can begin by following installation commands to load a station, run an OpenArm + S-hook reference demonstration, and score its recording. The project provides datasets and ACT policies through complete local materials, with validation notes for clean macOS CPU checks.

This open-source benchmark represents a significant step toward standardizing robotic training for industrial environments, potentially improving safety and efficiency in chemical plant maintenance operations.