SROI V2 dataset documentation
Synchronized first-person RGB observations and 7-D end-effector actions derived from handheld SROI V2 demonstrations. This page documents the released LeRobot data, its source recordings, and the processing decisions needed to reproduce it.
01 / Releases
The LeRobot repositories are the intended inputs for model development. The MP4 and PNG trees described later are source and processing records, not alternate training releases.
1,459 episodes and 140,522 frames. The release combines a 1,125-episode base set with 177 July and 157 August recordings selected to increase coverage of occluded picking cases.
Open on Hugging Face100 episodes and 9,274 frames from the separate 14 July 2026 recording. All 100 episodes were classified ok by the documented trajectory and gripper QC rules.
02 / Feature schema
Stereo infrared images are used to estimate motion during processing. They are not included in the final observation; the released policy input is the color stream.
Observation
Intel RealSense D405 color frames encoded as AV1 video under observation.images.camera.
Action
For each frame, the target is taken from the following frame; the final frame repeats its own target. Position and rotation are derived from the transformed D405 trajectory.
ee.* labels are a learning-schema convention: the current converter writes the transformed D405 camera pose directly to these fields. The measured camera-to-gripper-tip transform is used for projection-based QC and is not composed into the released action. Likewise, robot_type: so100 is a converter compatibility label; these are handheld SROI demonstrations, not SO-100 joint logs. No joint state or observation.state field is released.| Field | Definition | Type |
|---|---|---|
observation.images.camera | D405 RGB image presented to the policy | video [480, 640, 3] |
ee.x / ee.y / ee.z | Camera-trajectory translation after conversion to X-forward, Y-left, Z-up axes | 3 × float32 |
ee.wx / ee.wy / ee.wz | Orientation of the transformed pose represented as a rotation vector | 3 × float32 |
ee.gripper_pos | AprilTag-derived gripper opening; 0 is closed and 1 is open | float32 [0, 1] |
task | Task vocabulary entry used by the release | “pick the strawberry” |
03 / Processing workflow
The diagram and records below follow the implementation in sroi_dataprocess. Every visual is derived from validation episode 050 so the intermediate products can be compared directly.
Direct acquisition
The Raspberry Pi recorder uses pyrealsense2 to capture left infrared, color, and right infrared at 640 × 480 and 30 fps. Hardware timestamps and per-stream calibration are stored with each episode. ROS is not part of the current V2 acquisition path.
record_realsense.py --encode-videoleft.mp4, color.mp4, right.mp4, timestamps.json, camera calibrationRegenerable workspace
Compressed MP4 streams are expanded into a separate PNG processing tree. Calibration, timestamps, and the generated ORB-SLAM camera configuration remain attached to the episode. The raw MP4 directory is retained as the acquisition record.
batches/decode_batch.sh / decode_videos.pyleft_*.png, color_*.png, right_*.png, times.txtStereo reconstruction
The known finger region is blacked out in temporary copies of both infrared streams. ORB-SLAM3 then reconstructs the camera trajectory from the remaining stereo features, avoiding feature tracks on the moving gripper. Original PNG frames are not altered.
apply_gripper_mask.py + orbslam_batch_local.shconfigs/gripper_mask_sroi_v2_d405.jsonCameraTrajectory.txt in KITTI 3 × 4 pose format
Pose convention
The ORB-SLAM trajectory is transformed to the project convention: X forward, Y left, and Z up. Position and orientation from this transformed D405 camera pose become the first six action values during conversion.
transform_trajectory.pyCameraTrajectoryTransformed.txtvisualization/visualize_traj_video.pyGripper state
AprilTag IDs 0 and 15 are detected in the lower region of each color image. Horizontal tag separation provides the raw opening signal, which is clipped and normalized to the [0, 1] convention used by the action vector.
gripper_estimation_april_tag.pygripper_distances.txt and normalization provenance where available0 = closed, 1 = open
Acceptance and packaging
QC classifies episodes using trajectory availability and length, gripper validity and variation, and final closure. Selected episodes are converted to AV1 RGB video, Parquet action records, episode indices, task metadata, and aggregate statistics.
visualization/qc.py + lerobot/sroi_to_lerobot.pyqc_categories=ok for the documented releasesdata/ chunk-000/file-000.parquet videos/observation.images.camera/ chunk-000/file-000.mp4 meta/ info.json stats.json tasks.parquet episodes/…
DATA_SOURCES.md instead records a range pooled over its own 100 episodes. Current pipeline documentation recommends the fixed training-reference configuration sroi_v2_d405_1000_onesb_range.json when building new comparable V2 releases. These are distinct provenance statements and should not be conflated.04 / Provenance
The final local release record documents both merge operations, input counts, codec compatibility, shared gripper range, and the exact official LeRobot merge command.
“Occlusion” in the repository name describes the targeted additions; it does not indicate that every episode in the base collection contains an occlusion.
05 / Citation
If you use the SROI datasets in your research, you are welcome to cite the following publication.
Hou, L., Lu, W., Wang, Y., Peng, C., & Fei, Z. (2025). Strawberry Robotic Operation Interface: An Open-Source Device for Collecting Dexterous Manipulation Data in Robotic Strawberry Cultivation. IFAC-PapersOnLine, 59(23), 303–308.
06 / Earlier dataset
The original SROI field collection is preserved separately on Zenodo. It uses ROS bags and OAK-D-SR or RealSense D435i sensor configurations; it is not schema-compatible with the V2 LeRobot releases above.
OAK-D-SR configuration
1280 × 720 stereo RGB at 30 fps with 400 Hz IMU data, camera calibration, and action-segmentation state.
RealSense D435i configuration
848 × 480 left/right infrared and RGB at 30 fps with accelerometer and gyroscope streams.