qrafty-ai/teleop_xr: Transforms your VR/AR headset into a powerful, precise robot controller

Transform your VR/AR headset into a powerful, precise robot controller. TeleopXR provides a lightweight, installation-free teleoperation interface with low-latency video streaming and full WebXR state tracking. Check full documentation here.

• 🕶️ VR/Passthrough: Seamlessly switch between fully immersive VR and high-fidelity AR Passthrough modes, allowing you to choose between total focus and situational awareness.
• 📡 WebRTC Video Streaming: Get ultra-low latency, real-time video feedback directly in the headset, providing a near-instantaneous visual link to your robot’s perspective.
• 🤖 Robot Visualization: Benefit from real-time 3D visualization of the robot model, ensuring your digital twin is always perfectly synchronized with the physical robot’s state.
• 🕹️ Realtime Teleoperation based on Whole-Body IK: Achieve precise and intuitive control through advanced Whole-Body Inverse Kinematics, enabling complex coordination with minimal effort.

Use the built-in demo to verify connectivity and visualize the XR state data in real-time.

Basic installation (teleop mode only):

With IK support:

The IK solver requires additional dependencies. Install with:

pip install teleop-xr
# Install IK dependencies from PyPI
pip install spatialmath-python>=1.1.15 gitpython>=3.1.46 xacro>=2.1.1 filelock>=3.20.3 viser>=1.0.21
# Install pyroki and ballpark from GitHub (not available on PyPI)
pip install git+https://github.com/chungmin99/pyroki.git
pip install git+https://github.com/chungmin99/ballpark.git

Note: pyroki and ballpark are not available on PyPI, so pip install teleop-xr[ik] will not work. You must install them manually from GitHub as shown above.

Alternatively, if you have npm installed, you can also install the whole package from source:

pip install "teleop-xr[ik]@git+https://github.com/qrafty-ai/teleop_xr"

The demo supports two operation modes:

• Teleop Mode (Default): Visualizes raw XR state data and button events.

  python -m teleop_xr.demo --mode teleop

• IK Mode: Enables the high-performance IK solver (configured for Unitree H1 by default). Requires IK dependencies installed.

  python -m teleop_xr.demo --mode ik

1. Open the displayed URL (https://:4443) in your headset.
2. Enter VR mode.
3. Observe the live state data and event logs in your terminal.

For detailed guides on integrating TeleopXR into your own projects, including the Generic Python API and ROS2 Interface, please visit our official documentation website:

👉 https://qrafty-ai.github.io/teleop_xr/generic/

For developers contributing to TeleopXR or customizing the frontend:

• Python 3.10+ with pip
• uv (recommended for development)
• Node.js & npm (for WebXR frontend)

1. Clone the repository:

   git clone https://github.com/qrafty-ai/teleop_xr.git
   cd teleop_xr

2. Install Python dependencies:

   Option A: Using uv (recommended)

   Option B: Using pip

   pip install -e .
   
   # For IK support, install additional dependencies:
   pip install spatialmath-python>=1.1.15 gitpython>=3.1.46 xacro>=2.1.1 filelock>=3.20.3 viser>=1.0.21
   pip install git+https://github.com/chungmin99/pyroki.git
   pip install git+https://github.com/chungmin99/ballpark.git

3. Build the WebXR frontend:

   cd webxr
   npm install
   npm run build

   (The build output will be used by the Python server)

4. Run from source:

   # From the root directory
   # With uv:
   uv run python -m teleop_xr.demo
   
   # Or with pip:
   python -m teleop_xr.demo

The IK solver requires pyroki and ballpark, which are not on PyPI. During development with uv, these are automatically installed from git. For pip-based installations, install them manually from GitHub as shown above.

This project is forked from SpesRobotics/teleop. We are grateful for their foundational work in creating a WebXR-based teleoperation solution.

We also leverage powerful libraries for robotics:

• Pyroki: For high-performance, differentiable Inverse Kinematics and collision checking.
• Ballpark: For robust collision geometry generation and sphere decomposition.

This project is licensed under the Apache License 2.0. See the LICENSE file for details.