🤖 Humanoid RL - DQN Training with ROS2 Jazzy & Gazebo

Deep Q-Network (DQN) ile humanoid robot yürüme eğitimi - ROS2 Jazzy ve Gazebo entegrasyonu.

🎯 Özellikler

• DQN Algoritması: Dueling DQN + Prioritized Experience Replay + N-step learning
• ROS2 Jazzy: Modern ROS2 node yapısı
• Gazebo Simülasyon: Gerçekçi fizik simülasyonu
• Real-time Training: Online öğrenme
• Model Persistence: Otomatik model kaydetme

📋 Gereksinimler

Sistem Gereksinimleri

• Ubuntu 22.04 veya 24.04
• ROS2 Jazzy (en son sürüm)
• Gazebo (Gazebo Classic veya Gazebo Harmonic)

Kurulum

# ROS2 Jazzy kurulumu (eğer yoksa)
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository universe
sudo apt update && sudo apt install curl -y
sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null

sudo apt update
sudo apt install ros-jazzy-desktop
sudo apt install ros-jazzy-gazebo-ros-pkgs
sudo apt install ros-jazzy-ros2-control
sudo apt install ros-jazzy-controller-manager
sudo apt install ros-jazzy-joint-state-broadcaster
sudo apt install ros-jazzy-joint-trajectory-controller
sudo apt install ros-jazzy-robot-state-publisher
sudo apt install ros-jazzy-joint-state-publisher

# Development tools
sudo apt install python3-colcon-common-extensions
sudo apt install python3-rosdep

🏗️ Build

cd ~/Desktop/humanoid_ws

# Source ROS2
source /opt/ros/jazzy/setup.bash

# Install dependencies
rosdep init  # ilk kez ise
rosdep update
rosdep install --from-paths src --ignore-src -r -y

# Build
colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release

# Source workspace
source install/setup.bash

🚀 Kullanım

Training Başlatma

# Terminal 1: ROS2 ve workspace'i source et
source /opt/ros/jazzy/setup.bash
source ~/Desktop/humanoid_ws/install/setup.bash

# Launch Gazebo + DQN training
ros2 launch humanoid_rl gazebo_train.launch.py

Training İzleme

# Terminal 2: Training istatistiklerini izle
source /opt/ros/jazzy/setup.bash
source ~/Desktop/humanoid_ws/install/setup.bash

ros2 topic echo /training_stats

Joint State İzleme

# Terminal 3: Robot joint durumlarını izle
ros2 topic echo /joint_states

📊 DQN Parametreleri

Parametreleri düzenlemek için: src/humanoid_rl/config/dqn_params.yaml

state_size: 114          # State space boyutu
action_size: 8           # Action space boyutu
update_rate_hz: 100.0    # Training update frekansı
episode_max_steps: 2000  # Maksimum episode uzunluğu
save_interval: 100       # Model kaydetme sıklığı

🧠 DQN Mimarisi

Neural Network

• Shared layers: 512 → 512 → 256
• Value stream: 256 → 128 → 1
• Advantage stream: 256 → 128 → 8

Hyperparameters

• Gamma (discount): 0.99
• Epsilon decay: 0.9995
• Learning rate: 0.0001
• Batch size: 64
• Replay buffer: 100,000
• N-step: 5
• PER alpha: 0.7

📁 Proje Yapısı

humanoid_ws/
├── src/
│   └── humanoid_rl/
│       ├── include/humanoid_rl/
│       │   ├── dqn_agent.h        # DQN core algoritması
│       │   ├── neural_net.h       # Neural network
│       │   └── stats.h            # İstatistik tracking
│       ├── src/
│       │   └── dqn_node.cpp       # ROS2 node
│       ├── urdf/
│       │   └── humanoid.urdf      # Robot modeli
│       ├── launch/
│       │   └── gazebo_train.launch.py
│       ├── config/
│       │   └── dqn_params.yaml
│       ├── package.xml
│       └── CMakeLists.txt
└── README.md

🎮 ROS2 Topics

Published

• /joint_trajectory_controller/joint_trajectory - Robot action commands
• /training_stats - Training istatistikleri (episode, reward, epsilon)

Subscribed

• /joint_states - Robot joint durumları (positions, velocities)

🔧 Troubleshooting

Gazebo açılmıyor

# Gazebo ortam değişkenlerini kontrol et
echo $GAZEBO_MODEL_PATH
export GAZEBO_MODEL_PATH=/usr/share/gazebo-11/models:$GAZEBO_MODEL_PATH

DQN node başlamıyor

# Build hatalarını kontrol et
colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release --event-handlers console_direct+

Model kaydedilmiyor

# Yazma izinlerini kontrol et
chmod +w ~/Desktop/humanoid_ws

📈 Training Tips

1. İlk 100 episode: Robot rastgele hareket eder (exploration)
2. Episode 100-500: Temel dengeler öğrenilir
3. Episode 500+: Yürüme patternleri gelişir
4. Model kaydetme: Her 100 episode'da otomatik kaydedilir

🚀 Gelecek Geliştirmeler

• DDPG/TD3 continuous control
• Gazebo reset service entegrasyonu
• Multi-robot parallel training
• Curriculum learning
• RViz visualization
• TensorBoard logging

📝 Lisans

MIT License - Eğitim amaçlıdır.

---

Developed with ROS2 Jazzy + Gazebo + C++ DQN