autonomy_benchmarks

‍This repository will be part of the Autonomy.Hub Ecosystem

As part of the Autonomy.Hub Ecosystem, Autonomy.Benchmarks enables the Automated Driving community to easily benchmark their automated driving building blocks across different tasks and datasets:

• 🔄 Unified ROS 2 Interface: Work with multiple datasets using the benefits of the ROS 2 ecosystem
• 📊 Comprehensive Benchmarks: Use the provided benchmarks with Autonomy.Datasets to benchmark building blocks across different automated driving tasks
• ⚡ Efficient Data Pipeline: Works seamlessly with preprocessed Rosbag files from Autonomy.Datasets for fast execution during development
• 🐳 Dockerized Environment: Reproducible setup with all dependencies included
• 🔌 Modular Architecture: Easy integration with other ROS 2 packages

Supported Benchmarks

This repository supports various automated driving evaluation benchmarks.

Detailed metric definitions and computation notes are documented in docs/IMPLEMENTATION.md.

‍Contributions adding more benchmarks are welcome

Benchmark	Challenge	Dataset	Task
nuScenes 3D Lidar Object Detection		nuScenes	3D bounding box detection from lidar

🚀 Quick Start • 💻 Development • 📝 Documentation

🚀 Quick Start

Clone autonomy_datasets and follow its setup instructions to prepare your dataset.

Use the provided docker-compose.yml to start the full pipeline — dataset publisher, system-under-test, and benchmark node:

# enable GUI output from Docker container
xhost +local:
 
# pull and start Docker containers
export COMPOSE_PROFILES="focalformer3d"  # or 'centerpoint'
docker compose pull
docker compose up -d
# stop containers once finished
docker compose down

Configure the benchmark task and dataset via ROS launch arguments in docker-compose.yml:

command: ros2 launch autonomy_benchmarks autonomy_benchmarks.launch.py benchmark:=nuscenes_lidar_object_detection prediction:=$your_prediction_topic label:=$your_label_topic

💻 Development

Set up Development Environment

1. Clone the repository.

   git clone https://github.com/thinking-cars/autonomy_benchmarks.git

1. Initialize the .openads-dev-environment submodule containing development environment configuration.

   cd autonomy_benchmarks
   git submodule update --init --recursive

1. Open the repository in Visual Studio Code.

   code .

1. Install the recommended VS Code extensions.

   ‍Ctrl+Shift+P / Extensions: Show Recommended Extensions / Install Workspace Recommended Extensions (Cloud Download Icon)

1. Reopen the repository in a Dev Container. Ctrl+Shift+P / Dev Containers: Rebuild and Reopen in Container

Build

‍Ctrl+Shift+B

colcon build

Run Tests

‍Ctrl+Shift+P / Tasks: Run Test Task

colcon build --cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=1
colcon test
colcon test-result --verbose

📝 Documentation

Package and node interfaces are documented in the respective package READMEs listed below. Implementation details are found in the Source Code Documentation.

Package	Description
autonomy_benchmarks	Benchmarking suite for automated driving tasks

⚖️ Licensing

The source code in this repository is licensed under Apache-2.0, see [LICENSE](LICENSE). Container images provided by this repository may contain third-party software shipped with their own license terms.

🙏 Acknowledgements

This project is maintained by Thinking Cars. We appreciate contributions and are happy to discuss potential collaborations.