Module 3 Assessment: Isaac AI Perception and Navigation Pipeline

Overview

This project assesses your ability to leverage NVIDIA Isaac Sim and Isaac ROS for building AI-powered perception and navigation systems. You will create a photorealistic simulation environment, implement perception pipelines, and demonstrate sim-to-real principles.

Duration: Weeks 8-10 Weight: 25% of final grade Submission: End of Week 10

Learning Objectives

By completing this project, you will demonstrate your ability to:

1. Set up and configure photorealistic environments in Isaac Sim
2. Implement hardware-accelerated perception using Isaac ROS
3. Integrate AI models for object detection and segmentation
4. Implement navigation using Nav2 with Isaac ROS
5. Demonstrate sim-to-real transfer concepts
6. Optimize performance for real-time robotics applications

Project Requirements

1. Isaac Sim Environment Setup (15 points)

Requirement: Create a photorealistic Isaac Sim scene

Scene Components:

• Use Isaac Sim's warehouse or office environment (or create custom)
• Add 10+ objects from USD asset library
• Include proper lighting (HDRI skybox or area lights)
• Configure realistic materials (PBR textures)
• Add humanoid or mobile robot (Carter, Nova, or custom)
• Create at least 3 distinct goal locations for navigation

Deliverable: USD scene file or Python script to generate scene

2. Perception Pipeline Implementation (30 points)

Implement a complete AI perception pipeline using Isaac ROS:

2A. Object Detection (15 points)

Using: Isaac ROS DNN Inference + YOLO or Detectron2

Implement real-time object detection:

• Detect at least 5 object classes (person, chair, table, box, etc.)
• Process RGB camera feed at ≥15 FPS
• Publish detections as vision_msgs/Detection2DArray
• Visualize bounding boxes in Isaac Sim or RViz2

Expected Performance:

• Inference latency: <50ms per frame
• Detection accuracy: >80% for trained classes

2B. Semantic Segmentation (15 points)

Using: Isaac ROS UNET or SegFormer

Implement pixel-wise scene understanding:

• Segment scene into semantic categories (floor, walls, obstacles, etc.)
• Output segmentation masks at 5-10 FPS
• Overlay segmentation on RGB feed
• Publish segmentation results as sensor_msgs/Image

Expected Output:

• Color-coded segmentation overlay
• Per-class pixel counts and percentages

3. Navigation System (25 points)

Implement autonomous navigation using Isaac ROS Nav2:

3A. Visual SLAM (10 points)

Using: Isaac ROS Visual SLAM

Configure and run VSLAM:

• Process stereo camera or RGB-D input
• Generate odometry estimates
• Build occupancy map of environment
• Publish TF transforms (map → odom → base_link)

Validation:

• Compare VSLAM odometry to ground truth from simulation
• Demonstrate loop closure detection

3B. Path Planning and Navigation (15 points)

Using: Nav2 with Isaac ROS integration

Implement goal-based navigation:

• Accept navigation goals via RViz2 or action interface
• Plan collision-free paths using costmaps
• Execute paths with dynamic replanning
• Handle moving obstacles gracefully

Test Scenarios:

1. Navigate from spawn point to 3 distinct goals
2. Navigate while avoiding dynamic obstacles
3. Recover from blocked paths (alternate route planning)

4. Manipulation Task (Bonus: +15 points)

Using: Isaac ROS Manipulation or MoveIt 2

Implement pick-and-place:

• Detect target object using perception pipeline
• Generate grasp pose using depth data
• Plan arm trajectory to grasp position
• Execute grasp and place at target location

Requirements:

• Successful grasp in at least 3/5 attempts
• Collision-free motion planning
• Use of Isaac Sim's physics engine for realistic grasping

5. Performance Optimization (10 points)

Demonstrate performance tuning:

Metrics to Report:

• Perception pipeline FPS
• Navigation planning time
• Memory usage (GPU and CPU)
• Inference latency breakdown

Optimizations to Implement:

• TensorRT optimization for AI models
• Appropriate ROS 2 QoS settings
• Efficient message serialization
• GPU memory management

6. Sim-to-Real Analysis (10 points)

File: sim_to_real_analysis.pdf

Provide a 2-3 page analysis covering:

Domain Randomization:

• How would you apply domain randomization to this scenario?
• What parameters would you vary (lighting, textures, object poses)?

Reality Gap:

• Identify potential sim-to-real gaps in your implementation
• Discuss sensor noise, timing, and physics differences

Transfer Strategy:

• Propose a deployment strategy for real hardware (Jetson Orin Nano)
• Estimate performance degradation (FPS, latency)
• Suggest calibration and fine-tuning procedures

7. Documentation and Presentation (10 points)

README.md:

• Setup instructions for Isaac Sim and Isaac ROS
• System architecture diagram
• How to run each component
• Troubleshooting common issues

Demo Video: 4-6 minutes showing:

• Isaac Sim scene walkthrough
• Perception pipeline in action (object detection + segmentation)
• VSLAM building a map
• Full navigation from start to goal
• Manipulation task (if attempted)

Technical Report: 4-5 pages covering:

• System design and architecture
• Perception model selection and rationale
• Navigation algorithm configuration
• Performance benchmarks and analysis
• Sim-to-real considerations
• Lessons learned and future improvements

Deliverables

Submit via the course portal:

1. Source Code: ROS 2 workspace with Isaac ROS packages (zip or Git repo)
2. Isaac Sim Scene: USD files or scene generation scripts
3. AI Models: TensorRT optimized models or ONNX files
4. Configuration Files: Nav2 params, VSLAM config, launch files
5. Demo Video: Full system demonstration
6. Technical Report: PDF document
7. Sim-to-Real Analysis: PDF document
8. Performance Log: CSV or JSON with benchmark results

Grading Rubric

Component	Points	Criteria
Isaac Sim Scene	15	Photorealistic environment, proper lighting, robot integration
Object Detection	15	Real-time detection, accurate bounding boxes, proper ROS integration
Semantic Segmentation	15	Pixel-wise classification, overlay visualization, acceptable FPS
Visual SLAM	10	Accurate odometry, map building, loop closure
Navigation	15	Goal-based navigation, obstacle avoidance, replanning
Performance Optimization	10	Benchmarks provided, optimization techniques applied
Sim-to-Real Analysis	10	Thoughtful analysis, realistic transfer strategy
Documentation	10	Clear instructions, video, technical report
Bonus: Manipulation	+15	Functional pick-and-place with high success rate
Total	100 (+15)

Grade Breakdown

• 90-100: Exceptional - All requirements exceeded, excellent performance, thorough analysis, manipulation bonus
• 80-89: Proficient - All core components functional, good performance, solid documentation
• 70-79: Developing - Most components working, acceptable performance, adequate documentation
• 60-69: Beginning - Basic perception and navigation functional, significant issues
• Below 60: Incomplete - Major components not working or missing

Testing Checklist

Before submission, verify:

• Isaac Sim scene loads without errors
• Robot spawns correctly in scene
• RGB camera publishes to ROS 2: ros2 topic echo /camera/rgb/image_raw
• Object detection node runs: ros2 launch isaac_ros_yolo yolo.launch.py
• Detections published: ros2 topic echo /detections
• VSLAM odometry publishing: ros2 topic echo /visual_slam/tracking/odometry
• Nav2 launches successfully: ros2 launch nav2_bringup navigation_launch.py
• Goals can be sent via RViz2
• Robot navigates to goals without collisions
• All benchmarks collected and documented
• Demo video covers all required components

Tips for Success

1. Start with Isaac Sim Tutorials: Complete NVIDIA's official tutorials before starting
2. Use Pre-trained Models: Leverage Isaac ROS's pre-trained models initially
3. Incremental Integration: Test perception → SLAM → navigation separately before combining
4. Monitor GPU Usage: Use nvidia-smi to track VRAM and utilization
5. Optimize Early: Apply TensorRT optimization from the start, not at the end
6. Version Compatibility: Ensure Isaac Sim, Isaac ROS, and ROS 2 versions match
7. Documentation: Take notes and screenshots throughout development

Common Pitfalls to Avoid

• VRAM Overflow: Isaac Sim + multiple AI models can exceed GPU memory
  • Solution: Reduce camera resolution, batch size, or model complexity
• TF Frame Errors: Incorrect transforms between camera, base_link, and map
  • Solution: Carefully configure URDF and Isaac Sim sensor frames
• Slow Perception: AI inference bottlenecks the pipeline
  • Solution: Use TensorRT optimization, reduce input resolution
• Navigation Failures: Costmap configuration issues
  • Solution: Tune inflation radius, obstacle layer parameters
• Sim Crash: Physics instabilities or memory leaks
  • Solution: Restart Isaac Sim regularly, simplify scene complexity

Example Commands

# Launch Isaac Sim (from Omniverse)
# Open scene: <yourname>_isaac_scene.usd

# Terminal 1: Launch Isaac ROS perception
ros2 launch <yourname>_isaac_perception perception.launch.py

# Terminal 2: Launch VSLAM
ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam.launch.py

# Terminal 3: Launch Nav2
ros2 launch nav2_bringup navigation_launch.py

# Terminal 4: Launch RViz2
rviz2 -d config/isaac_nav_config.rviz

# Terminal 5: Send navigation goal
ros2 action send_goal /navigate_to_pose nav2_msgs/action/NavigateToPose \
  "{pose: {header: {frame_id: 'map'}, pose: {position: {x: 5.0, y: 3.0, z: 0.0}}}}"

Performance Benchmarking

Use the provided benchmark script:

ros2 run <yourname>_isaac_perception benchmark_pipeline.py --duration 60

Expected output:

=== Perception Pipeline Benchmark ===
Object Detection FPS: 18.3
Segmentation FPS: 8.7
VSLAM Update Rate: 30.2 Hz
GPU Memory: 4.2 / 8.0 GB
Average Detection Latency: 42ms
P95 Detection Latency: 67ms

Resources

• Isaac Sim Documentation
• Isaac ROS Documentation
• Nav2 Documentation
• TensorRT Optimization Guide
• Isaac Sim Python API

Submission

Due Date: End of Week 10 (see course schedule) Submit to: Course portal Late Policy: 10% deduction per day, maximum 3 days late File Size Limit: 500 MB (use Git LFS for large models if needed)

Academic Integrity

This is an individual project. You may use NVIDIA's example code and pre-trained models, but system integration and analysis must be your own work. Cite all external resources used.

---

Questions? Contact the instructor or TA during office hours or via the course forum.

Hardware Note: This project requires an NVIDIA RTX GPU. If you don't have access, cloud instances (AWS g5.2xlarge) or lab workstations are available.