diff --git a/README.md b/README.md index da18514de81..3f67d0717af 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ Please make sure you have properly followed the LeRobot official installation (* [![watch the video](./src/lerobot/ufactory_usage/assets/inference-gello.png)](https://www.youtube.com/watch?v=wTiWLiHciT8) +[![watch the video](./src/lerobot/ufactory_usage/assets/inference-pika.png)](https://youtu.be/IiyvewZh5OY) ## Acknowledgement: diff --git a/src/lerobot/ufactory_usage/ReadMe.md b/src/lerobot/ufactory_usage/ReadMe.md index 98fba3599cc..b3c65cff3b4 100644 --- a/src/lerobot/ufactory_usage/ReadMe.md +++ b/src/lerobot/ufactory_usage/ReadMe.md @@ -1,6 +1,7 @@ # UFACTORY Data Collection Guide(Imitation Learning) ## ⚠️ Important Notice +Using gello to collect dataset: - Once data collection starts, **the relative position between the robot arm and the camera (D435/D435i) must not be changed**. - When making the inference, the position of the camera and the collection must be the same. Otherwise, the collected data will become invalid. - In case of change, the entire process must be re-performed: @@ -9,6 +10,10 @@ Collect Dataset → Train the model → Perform Inference and Evaluation ``` +Using pika to collect dataset: +- There are no specific requirements for the relative positions of the two pika stations and the robotic arm. However, **the pika needs to be recalibrated after the station moves**. +- The positions of the pika stations during the collection and inference processes do not have to be the same. + ## 1. Hardware Requirements ### Reference Hardware Setup @@ -20,43 +25,48 @@ Collect Dataset → Train the model → Perform Inference and Evaluation - **Camera Mount**: Provided by UFACTORY (purchase or 3D print) - Purchase: [UFACTORY Camera Stand](https://www.ufactory.cc/product-page/ufactory-xarm-camera-stand/) - 3D Model: [Realsense_Camera_Stand.STEP](https://www.ufactory.cc/wp-content/uploads/2024/05/CameraStand_1300.zip) -- **Teleoperation Device**: gello +- **Teleoperation Device**: gello, [pika](https://global.agilex.ai/products/pika) --- ## 2. Software Environment Setup -### 2.1 Install librealsense +### 2.1 Using gello + +#### Install librealsense Reference: [librealsense Linux Installation Guide](https://github.com/realsenseai/librealsense/blob/master/doc/distribution_linux.md) -#### Register Public Key +##### Register Public Key ```bash sudo mkdir -p /etc/apt/keyrings curl -sSf https://librealsense.realsenseai.com/Debian/librealsense.pgp | sudo tee /etc/apt/keyrings/librealsense.pgp > /dev/null ``` -#### Install APT HTTPS Support +##### Install APT HTTPS Support ```bash sudo apt-get install apt-transport-https ``` -#### Add the server to the list of repository +##### Add the server to the list of repository ```bash echo "deb [signed-by=/etc/apt/keyrings/librealsenseai.gpg] https://librealsense.realsenseai.com/Debian/apt-repo `lsb_release -cs` main" | \ sudo tee /etc/apt/sources.list.d/librealsense.list sudo apt-get update ``` -#### Install librealsense +##### Install librealsense ```bash sudo apt-get install librealsense2-dkms sudo apt-get install librealsense2-utils ``` +### 2.2 Using pika +Please refer to [pika tracking device calibration](https://github.com/xArm-Developer/ufactory_teleop/tree/main/pika_teleop). +First-time use of Pika Sense or when the base station position changes requires calibration. ---- -### 2.2 Create Python Virtual Environment +## 3. Lerobot Environment Setup +### 3.1 Create Python Virtual Environment #### Linux (Conda) ```bash @@ -69,9 +79,8 @@ conda activate lerobot conda install ffmpeg -c conda-forge ``` ---- -### 2.3 Install LeRobot and Dependencies +### 3.2 Install LeRobot and Dependencies #### Install LeRobot ```bash @@ -93,9 +102,9 @@ pip install pyrealsense2 # 2.56.5.9235 pip install git+https://github.com/xArm-Developer/xArm-Python-SDK.git ``` -## 3. Data Collection +## 4. Data Collection -### 3.1 Teleoperation Test +### 4.1 Teleoperation Test #### Using gello (xArm7) 1. Modify the configuration file (example: `xarm7_gello_record_config.yaml`) @@ -105,17 +114,35 @@ pip install git+https://github.com/xArm-Developer/xArm-Python-SDK.git - `start_joints`: Initial joint positions - **TeleoperatorConfig** - `start_joints`: Initial joint positions -2. Move the gello joints to match the robot's initial pose +2. Move the gello joints to match the robot's initial pose; 3. Start the script: ```bash python uf_robot_teleop_test.py --config config/xarm7_gello_record_config.yaml ``` -4. Press **Enter** to begin teleoperation -5. Press **ESC** to exit +4. Press **Enter** to begin teleoperation; +5. Press **ESC** to exit. ---- -### 3.2 Data Recording +#### Using pika (xArm7) +1. Modify the configuration file (example: `xarm7_pika_record_config.yaml`) + - **RobotConfig** + - `robot_ip`: IP address of the robot + - `robot_dof`: Degrees of freedom + - `start_joints`: Initial joint positions + - **TeleoperatorConfig** + - `start_joints`: Initial joint positions + - `gripper-port`: Pika gripper serial port + - `index_or_path`: Pika gripper camera index + - `port`: Pika sense serial port +2. Move the pika sense to the initial position; +3. Start the script: +```bash +python uf_robot_teleop_test.py --config config/xarm7_pika_record_config.yaml +``` +1. Double click pika sense to start teleoperation; +2. Press **ESC** to exit. + +### 4.2 Data Recording **Users may define their own data collection strategies. LeRobot only enforces the dataset format.** **Keyboard control** for data recording: @@ -134,24 +161,30 @@ python uf_robot_record.py --config config/xarm7_gello_record_config.yaml python uf_robot_record.py --config config/xarm7_mock_record_config.yaml ``` ---- +#### Recording with pika(teleoperation) +```bash +python uf_robot_record.py --config config/xarm7_pika_record_config.yaml +``` + +### 4.3 Resume Data Recording -### 3.3 Resume Data Recording +#### Resume gello-based recording ```bash -# Resume gello-based recording python uf_robot_record.py --config config/xarm7_gello_record_config.yaml --resume ``` - +#### Resume random target recording ```bash -# Resume random target recording python uf_robot_record.py --config config/xarm7_mock_record_config.yaml --resume ``` ---- +#### Resume pika-based recording +```bash +python uf_robot_record.py --config config/xarm7_pika_record_config.yaml --resume +``` -## 4. Training +## 5. Training -### 4.1 Initial Training(Use ACT) +### 5.1 Initial Training(Use ACT) ```bash # Note: # repo_id must match the value used during data collection @@ -168,7 +201,7 @@ python -m lerobot.scripts.lerobot_train \ --steps=800000 ``` -### 4.2 Resume Training +### 5.2 Resume Training ```bash python -m lerobot.scripts.lerobot_train \ --dataset.repo_id=ufactory/xarm7_record_datas \ @@ -184,7 +217,7 @@ python -m lerobot.scripts.lerobot_train \ --config_path=outputs/train/xarm7_record_datas/checkpoints/last/pretrained_model/train_config.json ``` -## 5. Inference & Evaluation +## 6. Inference & Evaluation ### Run with a Specified Model ```bash @@ -192,7 +225,7 @@ python uf_robot_eval.py --config config/xarm7_gello_record_config.yaml \ --policy.path=outputs/train/xarm7_record_datas/checkpoints/last/pretrained_model/ ``` -## 6. Dataset Utilities +## 7. Dataset Utilities ### Playback an Episode Example: view episode index **17** @@ -222,25 +255,24 @@ aggregate_datasets( ``` -## 7. Important Notes +## 8. Important Notes Users are expected to thoroughly study the codebase and configuration parameters. The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements. In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**. -## 8. Dataset Examples (Reference Only) +## 9. Dataset Examples (Reference Only) [Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY?usp=sharing) used during development (not reusable): - **xarm7_act_20260119**: 60 successful single-attempt grasps recorded via gello - **xarm7_act_20260127**: 80 samples (added 20 failure-and-retry cases) - **xarm7_act_mock_20260126**: 60 one-shot grasps generated programmatically +- **xarm7_pika_datas_aa_merge_0208_0209_2pi**: 150 samples grasps recorded via pika > These datasets are **for reference only**. -> They cannot be reused because the robot–camera calibration differs between users. +> They cannot be reused because the robot–camera position differs between users. -* Train using xarm7_act_20260127 for 500,000 iterations. +* Train using xarm7_act_20260127 for 500,000 training steps. [![watch the video](/src/lerobot/ufactory_usage/assets/inference-gello.png)](https://www.youtube.com/watch?v=wTiWLiHciT8) -* Train xarm7_act_mock_20260126 for 500,000 iterations. - --waiting... -* Failure Cases - --waiting... \ No newline at end of file +* Train xarm7_pika_datas_aa_merge_0208_0209_2pi for 400,000 training steps. + [![watch the video](/src/lerobot/ufactory_usage/assets/inference-pika.png)](https://youtu.be/IiyvewZh5OY) diff --git a/src/lerobot/ufactory_usage/ReadMe_cn.md b/src/lerobot/ufactory_usage/ReadMe_cn.md index e2f193a5e37..6453917ac23 100644 --- a/src/lerobot/ufactory_usage/ReadMe_cn.md +++ b/src/lerobot/ufactory_usage/ReadMe_cn.md @@ -1,7 +1,9 @@ # UFACTORY LeRobot 数据采集、训练与推理使用说明 -## ⚠️ 重要提示 +本文章仅提供数据采集方式,具体使用何种方式进行训练和推理客户可自行决定。 +## ⚠️ 重要提示 +对于gello数据采集方式: - **一旦开始数据采集,机械臂与摄像头(D435 / D435i)的相对位置必须保持不变**。 - 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化,**此前采集的数据将无效**。 - 如发生变化,必须重新执行完整流程: @@ -10,6 +12,11 @@ 重新采集数据 → 重新训练模型 → 重新推理运行 ``` +对于pika数据采集方式: +- **两个基站和机械臂相对位置没有要求,只需要保证采集时pika sense在基站范围内,但基站移动后需要重新校准**。 +- 采集和推理时基站位置可不相同。 + + ## 1. 硬件要求 ### 1.1 推荐硬件配置(示例脚本) @@ -20,40 +27,47 @@ - **摄像头支架**: UFACTORY 提供 (可购买或 3D 打印) - 购买链接: [UFACTORY 摄像头支架](https://www.ufactory.cc/product-page/ufactory-xarm-camera-stand/) - 3D 文件下载: [Realsense_Camera_Stand.STEP](https://www.ufactory.cc/wp-content/uploads/2024/05/CameraStand_1300.zip) -- **遥操设备**: gello +- **遥操设备**: gello, [pika](https://global.agilex.ai/products/pika) -## 2. 软件环境配置 +## 2. 遥操设备环境配置 -### 2.1 安装librealsense库 +### 2.1 gello采集方式 + +#### 安装librealsense库 参考:[librealsense](https://github.com/realsenseai/librealsense/blob/master/doc/distribution_linux.md) -#### 注册公钥 +##### 注册公钥 ```bash sudo mkdir -p /etc/apt/keyrings curl -sSf https://librealsense.realsenseai.com/Debian/librealsense.pgp | sudo tee /etc/apt/keyrings/librealsense.pgp > /dev/null ``` -#### Https Support +##### Https Support ```bash sudo apt-get install apt-transport-https ``` -#### 添加服务到仓库列表 +##### 添加服务到仓库列表 ```bash echo "deb [signed-by=/etc/apt/keyrings/librealsenseai.gpg] https://librealsense.realsenseai.com/Debian/apt-repo `lsb_release -cs` main" | \ sudo tee /etc/apt/sources.list.d/librealsense.list sudo apt-get update ``` -#### 安装librealsense库 +##### 安装librealsense库 ```bash sudo apt-get install librealsense2-dkms sudo apt-get install librealsense2-utils ``` -### 2.2 创建Python虚拟环境 +### 2.2 pika采集方式 +请参考[pika安装和校准说明](https://github.com/xArm-Developer/ufactory_teleop/blob/main/pika_teleop/README_ZH.md)。 +首次使用Pika Sense 或者**基站位置变动**需要校准。 +## 3. lerobot环境配置 + +### 3.1 创建Python虚拟环境 #### Linux(Using Conda) ```bash conda create -y -n lerobot python=3.10 @@ -65,7 +79,7 @@ conda activate lerobot conda install ffmpeg -c conda-forge ``` -### 2.3 安装lerobot和依赖 +### 3.2 安装lerobot和依赖 #### 安装lerobot ```bash git clone https://github.com/xArm-Developer/lerobot.git @@ -86,9 +100,9 @@ pip install pyrealsense2 # 2.56.5.9235 pip install git+https://github.com/xArm-Developer/xArm-Python-SDK.git ``` -## 3. 数据采集 +## 4. 数据采集 -### 3.1 Teleop控制测试 +### 4.1 Teleop控制测试 #### 使用gello(xArm7为例) 1. 修改配置文件这里举例使用xarm7_gello_record_config.yaml * RobotConfig: @@ -106,8 +120,28 @@ python python uf_robot_teleop_test.py --config config/xarm7_gello_record_config. 4. 按下回车开始控制 5. 按下'ESC'退出 +#### 使用pika(xArm7为例) +1. 修改配置文件这里举例使用xarm7_pika_record_config.yaml +* RobotConfig: + * robot_ip: 要控制的机械臂IP + * robot_dof: 机械臂轴数 + * start_joints: 机械臂初始位置 +* TeleoperatorConfig: + * start_joints: 机械臂初始位置 + * gripper_port:pika gripper的串口,可以不修改,会默认识别 + * index_or_path:pika gripper上的鱼眼相机index + * port:pika sense的串口,可以不修改,会默认识别 -### 3.2 采集 +2. 将pika移动到操作的初始位置 +3. 启动脚本 +```bash +python python uf_robot_teleop_test.py --config config/xarm7_pika_record_config.yaml +``` + +4. 双击pika gripper开始和关闭控制 +5. 按下'ESC'退出 + +### 4.2 采集 **用户可自定义采集方式,lerobot仅定义数据集的格式。** 以下举例使用gello采集或程序生成随机点进行采集: @@ -118,32 +152,36 @@ python python uf_robot_teleop_test.py --config config/xarm7_gello_record_config. ```bash python uf_robot_record.py --config config/xarm7_gello_record_config.yaml ``` -[![演示视频](assets/recording-with-gello.png)](https://www.bilibili.com/video/BV13jFjz2Eqq/?spm_id_from=333.1387.homepage.video_card.click) - #### 使用程序生成随机目标点后, 程序自动生成数据来采集 ```bash python uf_robot_record.py --config config/xarm7_mock_record_config.yaml ``` -[![演示视频](assets/recording-with-scripts.png)](https://www.bilibili.com/video/BV1tpf9BVEvp/?spm_id_from=333.1387.homepage.video_card.click) - +#### 使用pika采集(遥操) +```bash +python uf_robot_record.py --config config/xarm7_pika_record_config.yaml +``` -### 3.3 恢复采集 +### 4.3 恢复采集 +#### gello恢复采集 ```bash -# gello恢复采集 python uf_robot_record.py --config config/xarm7_gello_record_config.yaml --resume ``` +#### 程序生成点恢复采集 ```bash -# 程序随机生成点方式恢复采集 python uf_robot_record.py --config config/xarm7_mock_record_config.yaml --resume ``` +#### pika恢复采集 +```bash +python uf_robot_record.py --config config/xarm7_pika_record_config.yaml --resume +``` -## 4. 训练 +## 5. 训练 -### 4.1 初次训练 +### 5.1 初次训练 ```bash # 注意: repo_id就是采集时配置文件里面的repo_id @@ -159,7 +197,7 @@ python -m lerobot.scripts.lerobot_train \ --steps=800000 ``` -### 4.2 恢复训练 +### 5.2 恢复训练 ```bash python -m lerobot.scripts.lerobot_train \ --dataset.repo_id=ufactory/xarm7_record_datas \ @@ -175,7 +213,7 @@ python -m lerobot.scripts.lerobot_train \ --config_path=outputs/train/xarm7_record_datas/checkpoints/last/pretrained_model/train_config.json ``` -## 5. 推理 +## 6. 推理 ### 指定模型进行推理 ```bash @@ -184,7 +222,7 @@ python uf_robot_eval.py \ --policy.path=outputs/train/xarm7_record_datas/checkpoints/last/pretrained_model/ ``` -## 6. 数据集工具 +## 7. 数据集工具 提供一些数据集工具,方便对采集的数据集进行增删查操作。 ### 查看某个索引的episode: @@ -216,14 +254,16 @@ aggregate_datasets( ) ``` -### 7.重要提示 +### 8.重要提示 用户需要全面研究整个代码库,并了解相关的配置参数,因为代码中所写的配置并非适用于所有使用场景和设置,所以用户需要研究代码或相关理论,以获取相关知识,并自行进行修改和调整。特别是对于**扩散策略(diffusion policy)**,LeRobot 中的默认参数可能仅用于模拟,并未针对实际机器人场景进行优化。 -### 8. 数据集实例 +### 9. 数据集实例 以下为开发测试时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY?usp=sharing),**仅供参考,不可复用**,因为用户机械臂和摄像头位置和测试时不一致。 -* xarm7_act_20260119: gello手工录制一次性抓取成功的点,60个 -* xarm7_act_20260127: 在xarm7_act_20260119基础上增加了失败重抓的点20个,总共80个 -* xarm7_act_mock_20260126: 程序生成的60个一步到位的点 +* xarm7_act_20260119: gello手工录制一次性抓取成功的点(**关节运动**),60组; +* xarm7_act_20260127: 在xarm7_act_20260119基础上增加了失败重抓的点20组,共80组; +* xarm7_act_mock_20260126: 程序生成的一步到位的点,60组; +* xarm7_pika_datas_aa_merge_0208_0209_2pi:pika遥操采集的点(**笛卡尔-轴角**),150组。 + * 提示:使用笛卡尔运动时,无论使用RPY还是轴角采集时,数据Roll或Rx会存在不连续跳变问题,通过`±2π`可解决该问题,示例脚本中已处理。 训练后效果: * 使用xarm7_act_20260127进行训练,训练50w次。 @@ -232,5 +272,8 @@ aggregate_datasets( * 使用xarm7_act_mock_20260126,训练50w次。 [![演示视频](assets/inference-scripts.png)](https://www.bilibili.com/video/BV14pf9BVEox/?spm_id_from=333.1387.homepage.video_card.click) +* 使用xarm7_pika_datas_aa_merge_0208_0209_2pi,训练40w次。 +[![演示视频](assets/inference-pika.png)](https://www.bilibili.com/video/BV16ccizHE2P/?spm_id_from=333.1387.homepage.video_card.click) + * 失败案例 [![演示视频](assets/failure-cases.png)](https://www.bilibili.com/video/BV1Waf9BTEMr/?spm_id_from=333.1387.homepage.video_card.click) \ No newline at end of file diff --git a/src/lerobot/ufactory_usage/assets/inference-gello.png b/src/lerobot/ufactory_usage/assets/inference-gello.png index bb11faf7ac5..256184747b9 100644 Binary files a/src/lerobot/ufactory_usage/assets/inference-gello.png and b/src/lerobot/ufactory_usage/assets/inference-gello.png differ diff --git a/src/lerobot/ufactory_usage/assets/inference-pika.png b/src/lerobot/ufactory_usage/assets/inference-pika.png new file mode 100644 index 00000000000..83d72809d9c Binary files /dev/null and b/src/lerobot/ufactory_usage/assets/inference-pika.png differ diff --git a/src/lerobot/ufactory_usage/assets/recording-with-gello.png b/src/lerobot/ufactory_usage/assets/recording-with-gello.png deleted file mode 100644 index 1baae5d1150..00000000000 Binary files a/src/lerobot/ufactory_usage/assets/recording-with-gello.png and /dev/null differ diff --git a/src/lerobot/ufactory_usage/assets/recording-with-scripts.png b/src/lerobot/ufactory_usage/assets/recording-with-scripts.png deleted file mode 100644 index 493a10e8b28..00000000000 Binary files a/src/lerobot/ufactory_usage/assets/recording-with-scripts.png and /dev/null differ