6DoF Perception Pipeline with Edge MLOps

A real-time 6DoF object pose estimation system designed for industrial pick-and-place applications, tailored for edge devices like the NVIDIA Jetson AGX Orin. This project features a fully automated Continuous Machine Learning (MLOps) pipeline integrated with GitHub Actions and GitHub Container Registry (GHCR).

🏗️ Architecture & ROS 2 Nodes

The original monolithic proof-of-concept has been refactored into a highly modular ROS 2 (Humble) C++ workspace consisting of the following custom packages:

1. realsense_node (Video Emulator):
   • Modified from original specs: Instead of relying on physical Intel RealSense D435 hardware, this node uses OpenCV to read a pre-recorded video file from disk (/data/video.mp4).
   • It publishes frames to /camera/color/image_raw and synthesizes aligned depth matrices and camera intrinsics to keep the downstream pipeline functional without hardware.
2. yolo_detector:
   • A C++ node leveraging the ONNX Runtime API. It ingests the RGB frames, runs inference using our trained yolo11s-seg model, applies Non-Maximum Suppression (NMS), and publishes 2D bounding boxes and class data.
3. depth_projection:
   • Synchronizes 2D detections with the depth feed to perform pinhole back-projection. It lifts the 2D data into colored 3D point clouds (pcl::PointXYZL) corresponding to detected objects.
4. pose_estimator:
   • Uses the Point Cloud Library (PCL) to perform Iterative Closest Point (ICP) alignment. It matches the generated 3D clusters against known .ply meshes (from the TUD-L dataset) to compute the final 6DoF pose (translation + quaternion rotation).
5. pose_visualizer:
   • A debugging node that publishes coordinate axes, bounding boxes, and text markers to RViz 2.

🧠 MLOps Automation & Data Loop

The system features a closed-loop "Continuous Learning" architecture powered by GitHub Actions.

confidence_monitor (Python)

A background ROS 2 Python node continuously monitors the output of the YOLO detector. If it detects a "hard frame" (an image where YOLO's confidence score drops below a configured threshold, indicating poor lighting or occlusion), it buffers the RGB frame, depth data, and metadata. Periodically, this node automatically commits and pushes these hard frames to the data branch of the repository.

CI/CD Pipelines

Our workflows have been consolidated and optimized for automation:

• ci.yml: Runs on the main branch to perform C++ linting (ament_clang_tidy) and executes Python unit tests (unittest) for the evaluation scripts.
• mlops.yml: Triggers automatically whenever the confidence_monitor pushes to the data branch. This is an unbroken chain with three jobs:
  1. Train: Dispatches to a self-hosted GPU runner, downloads the current ONNX model, fine-tunes it on the newly ingested data using PyTorch, and exports a new ONNX model.
  2. Evaluate (Validation Gate): Runs the newly trained model against the BOP19 test set. If the new Mean Average Precision (mAP) is not significantly better than the old model, the pipeline aborts.
  3. Deploy: If evaluation passes, the pipeline automatically bumps the semantic version, creates a GitHub Release with the .onnx artifact, builds the Dockerfile.jetson Docker image, and pushes it to GitHub Container Registry (GHCR).

🚀 Quick Start (Deployment)

Because the pipeline is hosted on GHCR as a private package, you must authenticate your Jetson device before pulling.

1. Authenticate Docker

Generate a Personal Access Token (PAT) with read:packages scope on GitHub.

echo "YOUR_GITHUB_PAT" | docker login ghcr.io -u <YOUR_GITHUB_USERNAME> --password-stdin

2. Run the Edge Pipeline

# Pull the latest approved image from the automated MLOps pipeline
docker pull ghcr.io/perception-mlops/perception-pipeline:latest

# Launch the ROS 2 network, Video publisher, YOLO inference, and ICP nodes
docker compose up

🛠️ Local Development Build

If you wish to build the C++ workspace manually:

cd perception_ws
source /opt/ros/humble/setup.bash
colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release
source install/setup.bash
ros2 launch perception_ws perception.launch.py