Robotis OP2 Persona-Aware Controller

A ROS 2 package that controls a simulated Robotis OP2 humanoid robot in Webots. The robot uses LLM-powered conversation with persona-aware behavior, performing context-appropriate gestures and animations based on semantic intent matching.

System Requirements
Directory Structure
Architecture Overview
File Descriptions
Configuration Reference
Installation
Building the Project
Usage
Troubleshooting

System Requirements

Component	Version
OS	Ubuntu 22.04 LTS (Jammy Jellyfish)
ROS 2	Humble Hawksbill
Simulator	Webots R2023b or newer
Python	3.10+

Directory Structure

~/ros2_ws/src/op2_controller/
├── package.xml                 # ROS 2 package manifest
├── setup.py                    # Python package setup & entry points
├── setup.cfg                   # Package metadata
├── run.py                      # Helper script for setup and execution
│
├── config/
│   └── config.py               # Central configuration (API keys, paths, parameters)
│
├── docs/
│   └── README.md
├── launch/
│   └── robot_launch.py         # ROS 2 launch file for Webots + driver
│
├── op2_controller/             # Main Python package
│   ├── __init__.py
│   ├── op2_driver.py           # Animation engine (Webots controller)
│   ├── op2_brain.py            # Main AI node (STT → LLM → TTS → Action)
│   ├── op2_action_vocab.py     # Action vocabulary definitions
│   ├── personas.py             # Persona definitions (personality profiles)
│   ├── generate_action_embeddings.py  # Pre-compute action embeddings
│   ├── test_op2_actions.py     # Manual action tester
│   └── action_embeddings.pkl   # Pre-computed embeddings (generated)
│
├── resource/
│   ├── op2_controller          # Ament resource marker
│   └── op2.urdf                # Robot description linking to Python driver
└── worlds/
    └── op2_world.wbt           # Webots simulation world file

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                              OP2 BRAIN (op2_brain.py)                       │
│                                                                             │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌──────────────┐             │
│  │   VAD   │───▶│   STT   │───▶│   LLM   │───▶│ ActionSelect │             │
│  │ (Silero)│    │ (Riva)  │    │ (Llama) │    │ (Embeddings) │             │
│  └─────────┘    └─────────┘    └─────────┘    └──────┬───────┘             │
│       ▲                              │               │                      │
│       │                              ▼               ▼                      │
│   [Microphone]                 ┌─────────┐    ┌─────────────┐              │
│                                │   TTS   │    │  Publish to │              │
│                                │ (Riva)  │    │/perform_action│             │
│                                └────┬────┘    └──────┬──────┘              │
│                                     │                │                      │
│                                     ▼                │                      │
│                               [Speaker]              │                      │
└──────────────────────────────────────────────────────┼──────────────────────┘
                                                       │
                           ROS 2 Topic: /perform_action│
                                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           OP2 DRIVER (op2_driver.py)                        │
│                                                                             │
│  ┌────────────────┐    ┌─────────────────┐    ┌─────────────────┐          │
│  │ Action Callback│───▶│ Animation Engine│───▶│  Webots Motors  │          │
│  │ (ROS Subscriber)│   │ (Sine Wave Math)│    │ (Joint Control) │          │
│  └────────────────┘    └─────────────────┘    └─────────────────┘          │
│                                                                             │
│  Action Types:                                                              │
│  • Static: Instant pose (set position once)                                 │
│  • Complex: Time-based animation (sine wave interpolation)                  │
└─────────────────────────────────────────────────────────────────────────────┘
                                       │
                                       ▼
                              ┌─────────────────┐
                              │     WEBOTS      │
                              │  OP2 Simulator  │
                              └─────────────────┘

Data Flow

Voice Input: User speaks → VAD detects speech → Audio recorded
Speech-to-Text: Nvidia Riva transcribes audio to text
LLM Processing: Persona-aware prompt + conversation history → generates intent | speech
Action Selection: Semantic search matches intent to closest action using embeddings
Text-to-Speech: Nvidia Riva synthesizes speech audio
Parallel Execution: Action published to robot + audio played (concurrent)
Animation: Driver receives action → executes sine-wave animation on joints

File Descriptions

Core Files

File	Purpose
`op2_brain.py`	Main AI node. Runs the conversation loop: listens for speech (VAD), transcribes (STT), generates persona-aware responses (LLM), maps intent to action (semantic search), synthesizes speech (TTS), and publishes actions. Supports conversation memory.
`op2_driver.py`	Animation engine. A Webots controller that subscribes to `/perform_action` and animates the robot. Uses sine-wave interpolation for smooth complex motions.
`op2_action_vocab.py`	Action vocabulary. Defines `OP2_ACTIONS` (static poses) and `OP2_COMPLEX_ACTIONS` (timed animations with joint curves and descriptions for embedding).
`personas.py`	Persona definitions. Contains personality profiles (e.g., `polite_teacher`, `angry_cab_driver`) with speaking style, example phrases, and TTS voice settings.
`generate_action_embeddings.py`	Embedding generator. Pre-computes sentence embeddings for action descriptions using `all-MiniLM-L6-v2`. Outputs `action_embeddings.pkl`.
`test_op2_actions.py`	Manual tester. Publishes action commands sequentially to test robot movements without the full AI pipeline.

Configuration

File	Purpose
`config/config.py`	Central configuration. All tunable parameters: API keys, model settings, paths, audio config, timeouts, thresholds, ROS settings, and logging options.

Launch & Resources

File	Purpose
`launch/robot_launch.py`	ROS 2 launch file. Starts Webots with the world file and attaches the `Op2Driver` controller.
`resource/op2.urdf`	Robot description. Links the Webots robot to the Python driver class via `<plugin type="...">`.
`worlds/op2_world.wbt`	Webots world. Contains the Robotis OP2 robot with `controller "<extern>"` to use external ROS control.

Configuration Reference

All configuration is centralized in config/config.py:

API Credentials

NVIDIA_API_KEY = "nvapi-..."       # Nvidia NIM API key
STT_FUNCTION_ID = "..."            # Speech-to-Text function ID
TTS_FUNCTION_ID = "..."            # Text-to-Speech function ID

Persona Selection

CURRENT_PERSONA_ID = "angry_cab_driver"  # Options: polite_teacher, polite_receptionist, angry_cab_driver

Path Configuration

CLIENTS_REPO_PATH = "./python-clients"                              # Nvidia Riva client scripts
EMBEDDING_FILE = "./src/op2_controller/.../action_embeddings.pkl"
AUDIO_INPUT = "assets/user_input.wav"
AUDIO_OUTPUT = "assets/robot_response.wav"

Audio Settings

SAMPLE_RATE = 16000          # Hz
SILENCE_THRESHOLD = 2.0      # Seconds of silence before stopping
VAD_CONFIDENCE_THRESHOLD = 0.5

LLM Settings

LLM_MODEL = "meta/llama3-8b-instruct"
LLM_TEMPERATURE = 0.7
LLM_MAX_TOKENS = 128

Memory & Actions

ENABLE_CONVERSATION_MEMORY = True
MAX_CONVERSATION_HISTORY = 20        # Exchanges to remember
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
ACTION_CONFIDENCE_THRESHOLD = 0.1    # Below this → fallback to stand_neutral
DEFAULT_ACTION = "stand_neutral"

Performance

ENABLE_PARALLEL_EXECUTION = True     # TTS + action search in parallel
STT_TIMEOUT = 30
TTS_TIMEOUT = 30

Installation

Step 1: Install ROS 2 Humble

sudo apt update && sudo apt install locales
sudo locale-gen en_US.UTF-8
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
export LANG=en_US.UTF-8

sudo apt install software-properties-common
sudo add-apt-repository universe

sudo apt update && sudo apt install curl -y
sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key \
    -o /usr/share/keyrings/ros-archive-keyring.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] \
    http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" | \
    sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null

sudo apt update
sudo apt install ros-humble-desktop ros-dev-tools

Step 2: Install Webots

⚠️ Do NOT use apt install webots — it may install an older version.

cd ~/Downloads
wget https://github.com/cyberbotics/webots/releases/download/R2023b/webots_2023b_amd64.deb
sudo apt install ./webots_2023b_amd64.deb -y

Step 3: Install ROS-Webots Bridge

sudo apt update
sudo apt install ros-humble-webots-ros2-driver

Step 4: Install Python Dependencies

cd ~/ros2_ws
sudo apt install libportaudio2
pip install "numpy<2.0" torch torchaudio scikit-learn sentence-transformers scipy sounddevice requests openai python-dotenv
pip install -r requirements.txt
pip install -U nvidia-riva-client

Step 5: Clone Nvidia Riva Clients (for STT/TTS)

cd ~/ros2_ws
git clone https://github.com/nvidia-riva/python-clients.git
cd python-clients && mkdir COLCON_IGNORE  # Prevent colcon from building it

Step 6: Generate Action Embeddings

cd ~/ros2_ws/src/op2_controller/op2_controller
python3 generate_action_embeddings.py

Building the Project

cd ~/ros2_ws

# Clean previous builds (recommended on fresh setup)
rm -rf build/ install/ log/

# Build
colcon build --symlink-install --packages-select op2_controller

# Source the environment
source install/setup.bash

Environment Variables

echo 'export WEBOTS_HOME=/usr/local/webots' >> ~/.bashrc
source ~/.bashrc

Usage

Terminal 1: Launch the Robot

Starts Webots, loads the OP2 robot, and attaches the animation driver.

cd ~/ros2_ws
source install/setup.bash
ros2 launch op2_controller robot_launch.py

Wait until you see: "OP2 Driver Ready (Action Vocabulary with Descriptions)"

Terminal 2: Run the AI Brain

Full conversation loop with voice input/output and gestures.

cd ~/ros2_ws
source install/setup.bash
ros2 run op2_controller op2_brain

Terminal 2 (Alternative): Test Actions Manually

Sends predefined action commands to verify robot movement.

cd ~/ros2_ws
source install/setup.bash
ros2 run op2_controller test_actions

Available Actions

Defined in op2_action_vocab.py:

Action	Description
`stand_neutral`	Reset to standard standing posture
`shake_head`	Head rotates left/right (disagreement)
`nod_head`	Head moves up/down (agreement)
`wave_right_hand`	Friendly wave greeting

Action Definition Format

"action_name": {
    "description": "Text for semantic embedding search",
    "duration": 4.0,       # Total animation time (seconds)
    "repetitions": 1,      # Number of sine wave cycles
    "curves": [
        {
            "joint": "ShoulderL",
            "min": 0.0,
            "max": 1.5,
            "start_from_max": True  # Optional: start at peak
        }
    ]
}

Personas

Defined in personas.py. Switch via CURRENT_PERSONA_ID in config.py:

ID	Name	Personality
`polite_teacher`	Professor OP2	Patient, encouraging, educational
`polite_receptionist`	Receptionist OP2	Professional, helpful, courteous
`angry_cab_driver`	Cabbie OP2	Irritable, impatient, blunt

Each persona includes:

personality: Character traits
speaking_style: How it talks
example_phrases: Sample expressions
tts_settings: Voice configuration (language, voice name)

Troubleshooting

Error	Cause	Fix
`Class Op2Driver cannot be found`	Typo in class name or wrong file location	Ensure class is `class Op2Driver(WebotsController):` in `op2_controller/op2_driver.py`
`No executable found` (ros2 run)	Entry points not registered or not rebuilt	Update `setup.py`, then `rm -rf build/ install/` and `colcon build --symlink-install`
Webots crash / version mismatch	Incompatible Webots and ROS driver versions	Install Webots R2023b+ via `.deb` and `ros-humble-webots-ros2-driver`
`NotInitializedException`	Creating ROS node before `rclpy.init()`	Add `if not rclpy.ok(): rclpy.init(args=None)` at start of `init()`
`action_embeddings.pkl not found`	Embeddings not generated	Run `python3 generate_action_embeddings.py` in `op2_controller/`
STT/TTS timeout	API issues or network	Check `NVIDIA_API_KEY` and function IDs in `config.py`

Quick Checklist

Ubuntu 22.04 + ROS 2 Humble installed
Webots R2023b+ installed (not from apt)
ros-humble-webots-ros2-driver installed
python-clients cloned with COLCON_IGNORE marker
action_embeddings.pkl generated
config.py has valid Nvidia API credentials
Workspace built with colcon build --symlink-install
WEBOTS_HOME=/usr/local/webots exported

License

MIT License

Maintainer

Vashu Chauhan — vashu22606@iiitd.ac.in

FilesExpand file tree

README.md

Latest commit

History