Edge Accelerated Robotic Arm for Adaptive Pick and Place

🏆 3rd Place, Creative Design Group - 2025 Andes RISC-V Awards 🏆

Overview

This project implements a vision-based robotic manipulation system using the Franka Emika Panda arm. It integrates deep learning (YOLO), 3D point cloud processing, and inverse kinematics (IK) to autonomously detect, localize, and manipulate objects (e.g., trash cans, soup cans) in a workspace.

Key Features:

• Object detection (YOLOv5/YOLOv8)
• 3D point cloud acquisition and transformation
• Camera-to-robot coordinate transformation and calibration
• Inverse kinematics (Pinocchio, IKPy)
• ROS-based control and data acquisition
• Demonstration learning and trajectory replay
• Modular scripts for each pipeline stage

---

Project Structure

• scripts/: Main Python scripts for detection, point cloud, IK, robot control, and demonstration learning
• assets/: Pretrained models, URDF files, calibration data
• tmp/: Runtime temporary data (joints, point clouds, detection results, etc.)
• ikpy/: IKPy inverse kinematics library (submodule/fork)
• vamp_mvt/: Motion validation and planning utilities (C++/Python)
• soup/: Example data for soup can tasks

---

Pipeline Overview

1. Initialization

• Move robot to home position (go_home.py)

2. Object Detection & Localization

• Capture RGB/depth images and camera intrinsics (get_ImageWithInfo.py)
• Run YOLO detection (run_yolo.py, trashcan_inference.sh)
• Compute 3D coordinates of detected object center (get_coordi.py)

3. Environment Sensing

• Acquire and transform environment point cloud (get_env_pc.py)

4. Inverse Kinematics

• Solve IK for target pose using Pinocchio (ik_pin.py) or IKPy (ikpy_transform.py)

5. Motion Execution

• Move robot using computed joint angles, control gripper (grip.py, drop.py)

6. Demonstration Learning

• Record and replay trajectories (Learning_from_demonstration.py, move_path.py)

---

Quick Start

Prerequisites

• Hardware: Franka Emika Panda arm, RealSense or compatible RGB-D camera
• Software:
  • Ubuntu 20.04+
  • ROS Noetic
  • Python 3.8+
  • Conda (environment management)
  • Main Python packages: see requirements.txt or environment.yaml
  • Pinocchio, IKPy
  • CUDA GPU (recommended for YOLO)

Installation

1. Clone the repository:

git clone --recursive https://github.com/yourusername/nody_robot.git
cd nody_robot

2. Set up Python environments:

• For YOLO/trashcan tasks: conda env create -f environment.yaml -n trash
• For VAMP/validation: conda env create -f vamp_mvt/environment.yaml -n vamp

3. Install ROS dependencies:

rosdep install --from-paths scripts/ --ignore-src -r -y

4. (Optional) Build C++ modules in vamp_mvt/ if needed

Usage

• Full pipeline (trash can demo):

  ./run.sh
  

  This process includes:

  • Move robot to home
  • Object detection and localization
  • Environment point cloud acquisition
  • IK computation and path visualization
  • Manual mode switch before execution

• Individual module testing: Run any script in scripts/ directly

---

Key Scripts

• go_home.py — Move robot to home position
• get_ImageWithInfo.py — Capture images and camera info
• run_yolo.py — YOLO object detection
• get_coordi.py — Compute 3D coordinates from detection
• get_env_pc.py — Environment point cloud acquisition and transformation
• ik_pin.py / ikpy_transform.py — Inverse kinematics solvers
• grip.py, drop.py — Gripper control
• Learning_from_demonstration.py — Demonstration learning
• move_path.py — Trajectory replay

---

Data & Models

• assets/:
  • trash_can.pt, yolosoup.pt: YOLO models
  • panda.urdf, panda.URDF: Robot description
  • Calibration files: cam2gripper.npy, origin_pc.npy, etc.
• tmp/:
  • Runtime data (joints, point clouds, detection results, etc.)

---

Notes & Assumptions

• Camera-to-robot calibration must be completed in advance; calibration files should be placed in assets/ or tmp/
• The architecture is modular; detection models, IK solvers, or hardware can be swapped with minor changes
• For detailed workflow and theory, refer to the handwritten report.pdf

---

Acknowledgements & Credits

This project stands on the shoulders of several incredible open-source projects. We sincerely thank the original authors:

• VAMP (Kavraki Lab): The core collision checking engine (including SIMD and AoS-to-SoA optimizations) is powered by VAMP. We integrated and modified it to run efficiently with our ROS pipeline on the Tinker V board. (See vamp_mvt/ directory)
• IKPy: Used as a git submodule for solving complex Inverse Kinematics.
• Pinocchio: For rigid body dynamics and fast kinematics computations.
• Ultralytics YOLO: For foundational object detection models.