This page lists strawberry-picking datasets collected with three systems: SROI V1, SROI V2, and a SCARA robot. The datasets differ in collection hardware, storage format, observations, and actions. The sections after the dataset list document the SROI V2 processing pipeline.
01 / Datasets
The datasets were recorded with three different collection systems. SROI V1 is distributed as ROS bags. SROI V2 and SCARA data are distributed in LeRobot v3.0 format with different observation and action schemas.
SROI V1
Demonstrations collected with the original handheld SROI device in real agricultural settings. The archive preserves synchronized sensor streams as ROS bags rather than LeRobot episodes. See the sensor details below.
Original field demonstrations recorded with OAK-D-SR or RealSense D435i configurations. The bags contain stereo/RGB sensing, IMU streams, calibration, and collection-state records; they require the V1 ROS-bag processing path.
Open on ZenodoSROI V2
Human demonstrations recorded with the D405-equipped SROI V2 device. Each frame pairs first-person RGB with a 7-D transformed pose and gripper-opening action; no robot arm is involved during collection.
1,459 episodes and 140,522 frames. The release combines a 1,125-episode base set with 177 July and 157 August recordings selected to increase coverage of occluded picking cases.
Open on Hugging Face100 episodes and 9,274 frames from the separate 14 July 2026 recording. All 100 episodes were classified ok by the documented trajectory and gripper QC rules.
SCARA
Human-teleoperated demonstrations from a 4-DoF SCARA arm. These releases provide a 4-D joint state, a 5-D joint-and-gripper action, and multi-view RGB observations in LeRobot v3.0 format.
511 episodes and 119,830 frames collected in a controlled physical mock-up with clustered fruit and occlusions. This is the dataset associated with Learning to Pick.
Open on Hugging Face659 episodes and 127,470 frames across eight source datasets and 21 session tasks. The unified top, wrist, and wrist_down schema uses black frames where a source camera was absent. These data were not used in the paper experiments.
| Dataset family | Collection source | Release format | Data representation |
|---|---|---|---|
| SROI V1 | Original handheld SROI device in field settings | ROS bags | Synchronized camera, IMU, calibration, and collection-state streams |
| SROI V2 | D405-equipped handheld SROI V2 device in laboratory settings | LeRobot v3.0 | First-person RGB and 7-D transformed camera-pose/gripper actions |
| SCARA | Human-teleoperated 4-DoF SCARA robot | LeRobot v3.0 | Multi-view RGB, 4-D joint state, and 5-D joint/gripper actions |
02 / SROI V2 schema
Stereo infrared images are used to estimate motion during processing. They are not included in the final observation; the released policy input is the color stream.
Observation
Intel RealSense D405 color frames encoded as AV1 video under observation.images.camera.
Action
For each frame, the target is taken from the following frame; the final frame repeats its own target. Position and rotation are derived from the transformed D405 trajectory.
ee.* labels are a learning-schema convention: the current converter writes the transformed D405 camera pose directly to these fields. The measured camera-to-gripper-tip transform is used for projection-based QC and is not composed into the released action. Likewise, robot_type: so100 is a converter compatibility label; these are handheld SROI demonstrations, not SO-100 joint logs. No joint state or observation.state field is released.| Field | Definition | Type |
|---|---|---|
observation.images.camera | D405 RGB image presented to the policy | video [480, 640, 3] |
ee.x / ee.y / ee.z | Camera-trajectory translation after conversion to X-forward, Y-left, Z-up axes | 3 × float32 |
ee.wx / ee.wy / ee.wz | Orientation of the transformed pose represented as a rotation vector | 3 × float32 |
ee.gripper_pos | AprilTag-derived gripper opening; 0 is closed and 1 is open | float32 [0, 1] |
task | Task vocabulary entry used by the release | “pick the strawberry” |
03 / SROI V2 processing
The diagram and records below follow the implementation in sroi_dataprocess. Every visual is derived from validation episode 050 so the intermediate products can be compared directly.
Direct acquisition
The Raspberry Pi recorder uses pyrealsense2 to capture left infrared, color, and right infrared at 640 × 480 and 30 fps. Hardware timestamps and per-stream calibration are stored with each episode. ROS is not part of the current V2 acquisition path.
record_realsense.py --encode-videoleft.mp4, color.mp4, right.mp4, timestamps.json, camera calibrationRegenerable workspace
Compressed MP4 streams are expanded into a separate PNG processing tree. Calibration, timestamps, and the generated ORB-SLAM camera configuration remain attached to the episode. The raw MP4 directory is retained as the acquisition record.
batches/decode_batch.sh / decode_videos.pyleft_*.png, color_*.png, right_*.png, times.txtStereo reconstruction
The known finger region is blacked out in temporary copies of both infrared streams. ORB-SLAM3 then reconstructs the camera trajectory from the remaining stereo features, avoiding feature tracks on the moving gripper. Original PNG frames are not altered.
apply_gripper_mask.py + orbslam_batch_local.shconfigs/gripper_mask_sroi_v2_d405.jsonCameraTrajectory.txt in KITTI 3 × 4 pose format
Pose convention
The ORB-SLAM trajectory is transformed to the project convention: X forward, Y left, and Z up. Position and orientation from this transformed D405 camera pose become the first six action values during conversion.
transform_trajectory.pyCameraTrajectoryTransformed.txtvisualization/visualize_traj_video.pyGripper state
AprilTag IDs 0 and 15 are detected in the lower region of each color image. Horizontal tag separation provides the raw opening signal, which is clipped and normalized to the [0, 1] convention used by the action vector.
gripper_estimation_april_tag.pygripper_distances.txt and normalization provenance where available0 = closed, 1 = open
Acceptance and packaging
QC classifies episodes using trajectory availability and length, gripper validity and variation, and final closure. Selected episodes are converted to AV1 RGB video, Parquet action records, episode indices, task metadata, and aggregate statistics.
visualization/qc.py + lerobot/sroi_to_lerobot.pyqc_categories=ok for the documented releasesdata/ chunk-000/file-000.parquet videos/observation.images.camera/ chunk-000/file-000.mp4 meta/ info.json stats.json tasks.parquet episodes/…
DATA_SOURCES.md instead records a range pooled over its own 100 episodes. Current pipeline documentation recommends the fixed training-reference configuration sroi_v2_d405_1000_onesb_range.json when building new comparable V2 releases. These are distinct provenance statements and should not be conflated.04 / SROI V2 provenance
The final local release record documents both merge operations, input counts, codec compatibility, shared gripper range, and the exact official LeRobot merge command.
“Occlusion” in the repository name describes the targeted additions; it does not indicate that every episode in the base collection contains an occlusion.
05 / Publications
Publication links are listed by collection system. Dataset-specific citation guidance is available in each repository.
SROI V1 and SROI V2 context
Hou, L., Lu, W., Wang, Y., Peng, C., & Fei, Z. (2025). Strawberry Robotic Operation Interface: An Open-Source Device for Collecting Dexterous Manipulation Data in Robotic Strawberry Cultivation. IFAC-PapersOnLine, 59(23), 303–308.
Open paperSCARA clustered dataset
Fei, Z., Lu, W., Hou, L., Ye, Z., & Peng, C. (2026). Learning to Pick: A Visuomotor Policy for Clustered Strawberry Picking. IEEE Transactions on AgriFood Electronics. The SCARA extra dataset was not used in this paper’s experiments.
Open paper06 / SROI V1 details
The original SROI field collection is preserved on Zenodo. It uses ROS bags, OAK-D-SR or RealSense D435i sensor configurations, and its own processing workflow.
OAK-D-SR configuration
1280 × 720 stereo RGB at 30 fps with 400 Hz IMU data, camera calibration, and action-segmentation state.
RealSense D435i configuration
848 × 480 left/right infrared and RGB at 30 fps with accelerometer and gyroscope streams.