Skip to main content

The AI-Robot Brain: Isaac ROS for VSLAM & Navigation

Once we have a simulated robot that can perceive its world, the next step is to give it autonomy. How does a robot know where it is? And how does it get from point A to point B? This chapter dives into the NVIDIA Isaac ROS packages, a collection of hardware-accelerated ROS 2 packages specifically designed for AI-based robotics.

We will focus on two critical tasks:

  1. VSLAM (Visual Simultaneous Localization and Mapping): Using a camera and an IMU, the robot will build a map of its environment and track its own position within that map in real-time.
  2. Navigation: Using the map generated by VSLAM, the robot will plan and execute a path to a goal destination, avoiding obstacles along the way.

Isaac ROS packages are optimized to run on NVIDIA's Jetson platform for edge AI and can be used on any machine with a powerful NVIDIA GPU, taking full advantage of hardware acceleration for massive performance gains over CPU-based alternatives.

The Isaac ROS Ecosystem​

The power of Isaac ROS is that it provides a suite of pre-built, high-performance "GEMs" (GPU-accelerated ROS packages) that can be chained together to create powerful perception pipelines. For VSLAM, we use Isaac ROS vslam.

How Isaac ROS VSLAM Works: The vslam node is a sophisticated pipeline that subscribes to camera images and IMU data. It performs the following steps under the hood:

  1. Feature Detection: It identifies key interest points (features) in the camera images.
  2. Feature Tracking: It tracks how these features move from one frame to the next.
  3. IMU Pre-integration: It integrates the IMU's accelerometer and gyroscope data to predict the robot's motion.
  4. Pose Graph Optimization: It combines the visual information (feature movement) and the inertial information (IMU prediction) into a mathematical structure called a pose graph. An optimizer then solves this graph to find the most likely trajectory of the robot and the 3D positions of the features, simultaneously creating the map and localizing the robot within it.

This process is computationally intensive, but by leveraging the GPU, Isaac ROS can perform VSLAM in real-time on high-resolution sensor data.

graph TD
    subgraph Input Sensor Data
        A[Camera Images (/camera/image_raw)]
        B[IMU Data (/camera/imu)]
    end
    subgraph Isaac ROS VSLAM Node
        C{vslam_node}
        D[GPU: Feature Tracking]
        E[GPU: Pose Graph Optimization]
        C --> D; C --> E;
    end
    subgraph Output
        F[Map Data (/map)]
        G[Robot Pose (/tf)]
    end
    A --> C; B --> C;
    C -- Publishes --> F;
    C -- Publishes --> G;

Lab 1: Running VSLAM in Isaac Sim​

In this lab, we will use Isaac Sim to provide the sensor data and run the Isaac ROS vslam node in a Docker container to build a map of a simulated environment.

Prerequisites:

  • NVIDIA Isaac Sim installed.
  • NVIDIA Docker (nvidia-container-toolkit) installed.
  • The Isaac ROS Docker image pulled: docker pull nvcr.io/isaac/isaac-ros-vslam:2.0.0

Step 1: Launch the Isaac Sim Scene with a Robot​

First, we need a robot in a scene that can be moved around. Isaac Sim comes with pre-built examples that are perfect for this.

  1. Launch Isaac Sim.
  2. Go to Window > Extensions and enable omni.isaac.ros_bridge.
  3. Go to Isaac Examples > ROS > Navigation. Click Load.
  4. This will load a scene with a "Carter" robot, which has a differential drive, a camera, and an IMU.
  5. Press Play in Isaac Sim. The simulation will start, and Isaac Sim will begin publishing sensor data to ROS 2 topics.

Image: Carter robot in the Isaac Sim warehouse environment. A screenshot of the Isaac Sim viewport showing the Carter robot in a warehouse environment.

Step 2: Launch the Isaac ROS VSLAM Docker Container​

Now, we'll run the VSLAM node inside the official Docker container. This ensures all dependencies are correctly installed.

  1. Create a workspace on your host machine: mkdir -p ~/isaac_ros_ws/src
  2. Launch the Docker container, mounting your workspace: 
      docker run --rm -it --net=host --gpus all \
        -v ~/isaac_ros_ws:/workspaces/isaac_ros_ws \
        nvcr.io/isaac/isaac-ros-vslam:2.0.0
  3. Inside the container, build the workspace and launch the VSLAM node: 
      # Inside the container
      cd /workspaces/isaac_ros_ws
      source /opt/ros/humble/setup.bash
      colcon build
      source install/setup.bash
      ros2 launch isaac_ros_vslam isaac_ros_vslam_nav2.launch.py

This launch file starts the vslam node and also configures it to publish data in a format that Nav2 (the ROS 2 navigation stack) can understand.

Step 3: Drive the Robot and Build the Map​

  1. Open RViz2 on your host machine (or inside the container if you have graphics forwarding set up). 
      # On your host machine
      rviz2
  2. Change the Fixed Frame to odom.
  3. Add a Map display and set the topic to /map.
  4. Add a Path display to visualize the robot's trajectory.
  5. In Isaac Sim, find the Viewport window. Right-click inside it and select Create > Isaac > ROS > Teleop. A small window with keyboard driving controls will appear.
  6. Click inside the Viewport to give it focus, and use the W, A, D keys to drive the robot around the environment.

As you drive the robot, you will see the map being built in real-time in RViz2! The VSLAM node is consuming the images and IMU data from Isaac Sim, building a map, and publishing it.

Image: VSLAM map creation in RViz2. A screenshot of RViz2 showing a partially built map of the warehouse environment. The robot's current position is visible, along with the path it has taken.

Lab 2: Path Planning with Nav2​

Now that we have a map, we can use the ROS 2 Navigation Stack (Nav2) to autonomously navigate the robot. The isaac_ros_vslam_nav2.launch.py file we used earlier already started the necessary Nav2 nodes.

Step 1: Set a Goal in RViz2​

  1. Ensure the VSLAM lab is still running (Isaac Sim is playing, the VSLAM docker container is running, and RViz2 is open).
  2. In RViz2, find the 2D Pose Estimate button in the top toolbar. Click it.
  3. Click and drag on the map near the robot's current position to give Nav2 an initial idea of where the robot is. This initializes the localization system.
  4. Now, click the Nav2 Goal button in the top toolbar.
  5. Click and drag on the map to set a destination for the robot. The arrow indicates the desired final orientation.

Step 2: Watch the Robot Navigate​

Once you set the goal, you will see several things happen in RViz2:

  • A green line will appear. This is the global path planned by Nav2.
  • A blue line will appear. This is the local path that the robot is currently trying to follow.
  • The robot model in Isaac Sim will start to move!

The Nav2 stack is calculating velocity commands based on the planned path and publishing them to the /cmd_vel topic. A ROS bridge script in the Isaac Sim scene subscribes to this topic and actuates the robot's joints, making it move. The robot will drive along the path, avoiding any obstacles it detects, until it reaches the goal.

Image: Nav2 path planning in RViz2. A screenshot of RViz2 showing the full map, the robot, a green global path snaking through the environment, and the blue local path immediately in front of the robot. The goal position is marked with an arrow.

Common Pitfalls​

  1. Error: VSLAM fails to initialize or loses tracking easily.

    • Cause: Poor lighting, lack of visual features in the environment, or moving too fast. VSLAM needs distinct, static points to track. A large, blank white wall is a nightmare for VSLAM.
    • Fix: Ensure your simulated environment has plenty of textures and distinct objects. Drive the robot slowly and avoid sudden, rapid rotations.

  2. Error: Nav2 fails to find a path or the robot gets stuck.

    • Cause: The map might be poor quality (with false obstacles), or the robot's physical footprint (configuration) in Nav2 is incorrect, causing it to think it can't fit through gaps.
    • Fix: Spend more time driving the robot around to build a more complete and accurate map before starting navigation. Check the Nav2 configuration files (nav2_params.yaml) to ensure the footprint parameter accurately matches your robot's shape.

  3. Error: Docker container cannot connect to ROS 2 running in Isaac Sim.

    • Cause: This is almost always a network issue. Docker containers have their own network unless configured otherwise.
    • Fix: Use the --net=host flag when running docker run. This tells the container to share the host machine's network stack, making it easy for ROS 2 nodes inside and outside the container to discover each other.

Student Exercises​

Exercise 1: Navigate a Custom Environment

Task: Instead of the default warehouse, create your own environment in Isaac Sim. Place some obstacles from the asset library. Run the VSLAM and Nav2 labs in your custom world.

Solution Steps:

  1. Create a new, empty scene in Isaac Sim.
  2. Add a ground plane.
  3. From the Isaac Sim Content browser, drag and drop various props (like cubes, cylinders, tables) to create a simple maze.
  4. Add the Carter robot to the scene (Create > Isaac > Robots > Carter).
  5. Add the ROS Teleop keyboard controller.
  6. Press Play and run through the VSLAM and Nav2 labs as before.
Exercise 2: Python Scripting for Navigation Goals

Task: Instead of setting the goal in RViz2, write a simple Python ROS 2 node that sends a navigation goal programmatically.

Solution Steps:

  1. Create a new Python ROS 2 package.
  2. The node needs to use an Action Client. Nav2's navigation goal is an action, not a simple service.
  3. Import nav2_msgs.action.NavigateToPose.
  4. Create an action client for /navigate_to_pose.
  5. Create a NavigateToPose.Goal message, fill in the pose.pose.position (x, y, z) and pose.pose.orientation (a quaternion).
  6. Send the goal using action_client.send_goal_async(). You can then wait for the result to see if the navigation was successful.
  7. Run this node after the Nav2 system is up and running.

Further Reading​

← Previous: Isaac Sim for Synthetic Data | Next: Nav2 for Bipedal Path Planning →