🚗 HazardNet

Intelligent V2X Hazard Detection & Voice-Controlled Driving Assistant

Real-time traffic simulation with AI-powered voice assistance and 3D visualization

Features • Quick Start • Architecture • Documentation

---

🌟 Features

🎤 Voice Assistant

• Real-time Speech Recognition with AssemblyAI WebSocket API
• AI-Powered Responses using Google Gemini 2.0
• Text-to-Speech with ElevenLabs
• Conversation Memory - Contextual 10-message history
• Anti-Spam Protection - Prevents request flooding

🚦 Traffic Simulation

• SUMO Integration - Realistic traffic simulation
• Multi-Vehicle Support - Handle 100+ vehicles
• Reinforcement Learning - PPO-trained autonomous agents
• V2X Communication - Vehicle-to-Everything hazard detection
• Dynamic Hazard Broadcasting - Real-time hazard sharing

🗺️ 3D Visualization

• Real-Time Rendering with React Three Fiber
• Smooth Camera Tracking - Cinematic vehicle following
• Route Visualization - Dynamic path display
• Interactive Map - SUMO network in 3D
• Hazard Markers - Visual hazard indicators

🔌 Real-Time Communication

• WebSocket Server - Low-latency vehicle data streaming
• Live Updates - 20Hz simulation data broadcast
• Automatic Reconnection - Robust connection handling
• JSON Protocol - Structured vehicle & hazard data

---

🚀 Quick Start

Prerequisites

• Python 3.10+
• Node.js 18+
• SUMO 1.24.0 (Download)
• Git

1️⃣ Clone the Repository

git clone https://github.com/yousefalwahami/hazardnet.git
cd hazardnet

2️⃣ Backend Setup

cd backend

# Create virtual environment
python -m venv venv

# Activate virtual environment
# Windows:
venv\Scripts\activate
# Mac/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Create .env file
copy .env.example .env  # Windows
# cp .env.example .env  # Mac/Linux

# Add your API keys to .env:
# GEMINI_API_KEY=your_gemini_key_here
# ELEVENLABS_API_KEY=your_elevenlabs_key_here

Get API Keys:

• 🧠 Gemini: https://aistudio.google.com/app/apikey
• 🔊 ElevenLabs: https://elevenlabs.io/ → Profile → API Keys
• 🎤 AssemblyAI: https://www.assemblyai.com/ → Dashboard

# Run the backend server
python app.py

Server runs on http://localhost:5000

3️⃣ Frontend Setup

cd frontend

# Install dependencies
npm install

# Create .env file (optional for AssemblyAI)
echo VITE_ASSEMBLYAI_API_KEY=your_assemblyai_key_here > .env

# Run development server
npm run dev

Frontend runs on http://localhost:5173

4️⃣ Simulation Setup

cd simulation

# Install dependencies (if using separate venv)
pip install -r requirements.txt

# Ensure SUMO_HOME is set
# Windows: set SUMO_HOME=C:\Program Files (x86)\Eclipse\Sumo
# Mac/Linux: export SUMO_HOME=/usr/share/sumo

# Run simulation with WebSocket server
python eval_demo_simple.py

WebSocket server runs on ws://localhost:8765

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         HazardNet System                         │
└─────────────────────────────────────────────────────────────────┘

┌──────────────────┐         ┌──────────────────┐         ┌──────────────────┐
│  SUMO Simulation │         │   WebSocket      │         │   React Frontend │
│                  │ ──────> │   Port 8765      │ ──────> │                  │
│  eval_demo_      │ Real-   │                  │ Live    │  MapViewer 3D    │
│  simple.py       │ time    │  Vehicle Data    │ Updates │  + Voice UI      │
│                  │ Data    │  Hazard Events   │         │                  │
└──────────────────┘         └──────────────────┘         └──────────────────┘
        ↓                                                           ↓
┌──────────────────┐                                     ┌──────────────────┐
│  V2X Hazard      │                                     │  Voice Assistant │
│  Detection       │                                     │                  │
│  - V2E Events    │                                     │  AssemblyAI STT  │
│  - V2V Comms     │                                     │  Gemini AI       │
│  - Broadcasting  │                                     │  ElevenLabs TTS  │
└──────────────────┘                                     └──────────────────┘
                                                                  ↓
                                                         ┌──────────────────┐
                                                         │  Flask Backend   │
                                                         │  Port 5000       │
                                                         │  AI Processing   │
                                                         └──────────────────┘

Data Flow

┌─────────────────────────────────────────────────────────────────┐
│  User speaks → AssemblyAI → Gemini AI → ElevenLabs → Audio     │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  SUMO → TraCI → Python → WebSocket → React → Three.js → GPU    │
└─────────────────────────────────────────────────────────────────┘

---

📂 Project Structure

hazardnet/
├── backend/                    # Python Flask API server
│   ├── app.py                 # Main Flask application
│   ├── services/
│   │   ├── gemini_service.py  # Gemini AI integration
│   │   └── elevenlabs_service.py  # Text-to-speech
│   └── requirements.txt
│
├── frontend/                   # React + TypeScript frontend
│   ├── src/
│   │   ├── components/
│   │   │   ├── MapViewer.tsx  # 3D SUMO visualization
│   │   │   ├── pages/
│   │   │   │   ├── driver.tsx # Driver interface
│   │   │   │   └── system.tsx # System view
│   │   │   └── voice/         # Voice UI components
│   │   └── hooks/
│   │       └── useVoiceAssistant.ts  # Voice logic
│   └── package.json
│
├── simulation/                 # SUMO traffic simulation
│   ├── eval_demo_simple.py    # Main simulation with WebSocket
│   ├── train_ppo_multi.py     # RL training script
│   ├── envs/
│   │   └── city_grid_multi_env.py  # Custom SUMO environment
│   └── sim/
│       ├── city.net.xml       # SUMO network
│       └── city.sumocfg       # SUMO configuration
│
└── docs/                       # Documentation
    ├── VOICE_ASSISTANT_SETUP.md
    ├── INTEGRATION_GUIDE.md
    └── ASSEMBLYAI_SETUP.md

---

🎯 Usage

Driver Interface

1. Navigate to http://localhost:5173
2. Toggle to "Driver" view (top-right)
3. Click the microphone button
4. Speak your command:
   • "Where is the nearest gas station?"
   • "What's the traffic like ahead?"
   • "Navigate to the hospital"
   • "Show me hazards on my route"

Voice Console Debugging

Open browser DevTools (F12) to see real-time logs:

✅ AssemblyAI WebSocket connected
🎧 Session started
🎤 Partial: Hello how
✅ Final: Hello how are you
📝 Transcript received: Hello how are you
🤖 Response received: I'm doing well, thanks for asking!

Simulation Controls

In SUMO GUI:

• Space - Play/Pause
• D - Delay (slower simulation)
• A - Accelerate
• S - Single step

Python console shows:

🚗 Vehicle count: 45
📡 Broadcasting to 1 clients
⚠️  Hazard detected at (123.45, 678.90)

---

📚 Documentation

Complete Guides

• 📖 Voice Assistant Setup - Complete voice system guide
• 🎤 AssemblyAI Setup - Speech recognition configuration
• 🔗 Integration Guide - Connect SUMO to frontend
• 💬 Voice Console - Debugging voice features

API Endpoints

Backend (Flask)

Method	Endpoint	Description
GET	/health	Health check
POST	/api/voice/process	Process voice command
POST	/api/voice/text-to-speech	Generate speech audio

WebSocket (Simulation)

Event	Description	Data Format
connect	Client connected	-
message	Vehicle update	{ vehicles: [...], hazards: [...], communications: [...] }
disconnect	Client disconnected	-

Vehicle Data Format

{
  "vehicles": [
    {
      "id": "vehicle_0",
      "x": 123.45,
      "y": 678.90,
      "speed": 10.5,
      "angle": 90.0,
      "type": "passenger",
      "edge": "edge_123",
      "route": [{"x": 130, "y": 680}, ...],
      "isEgo": false
    }
  ],
  "hazards": [
    {
      "type": "obstacle",
      "position": {"x": 100, "y": 200},
      "severity": "high",
      "timestamp": 1234567890
    }
  ],
  "communications": [
    {
      "sender": "vehicle_0",
      "receiver": "vehicle_1",
      "type": "V2V",
      "message": "hazard_ahead"
    }
  ]
}

---

🛠️ Advanced Configuration

Simulation Parameters

Edit simulation/eval_demo_simple.py:

# Simulation speed
SIMULATION_STEP_LENGTH = 0.05  # 50ms per step

# Hazard detection range
HAZARD_DETECTION_RADIUS = 50.0  # meters

# V2X communication range
V2X_RANGE = 100.0  # meters

Camera Settings

Edit frontend/src/components/MapViewer.tsx:

// Camera follow distance
const cameraDistance = 6; // units behind vehicle

// Camera height
const cameraHeight = 4; // units above ground

// Smoothing factor (0-1)
const lerpFactor = 0.05; // lower = smoother

Voice Assistant Tuning

Edit frontend/src/hooks/useVoiceAssistant.ts:

// Conversation history length
const MAX_HISTORY = 10; // messages

// Audio sample rate
const SAMPLE_RATE = 16000; // Hz

---

🐛 Troubleshooting

WebSocket not connecting

• Ensure eval_demo_simple.py is running
• Check port 8765 is not blocked by firewall
• Verify frontend WebSocket URL: ws://localhost:8765

No vehicles showing in 3D view

• Check browser console for errors
• Verify SUMO simulation has spawned vehicles
• Ensure WebSocket is sending data (check Python console)

Voice assistant not working

• Grant microphone permissions in browser
• Check API keys in .env files
• Verify backend is running on port 5000
• Open DevTools console to see error messages

SUMO won't start

• Set SUMO_HOME environment variable:
  • Windows: C:\Program Files (x86)\Eclipse\Sumo
  • Mac: /usr/local/opt/sumo/share/sumo
  • Linux: /usr/share/sumo
• Add %SUMO_HOME%\bin to PATH
• Verify installation: sumo --version

Camera jittering in 3D view

• Already fixed! Uses ref-based Vector3 lerping
• Adjust lerpFactor in MapViewer.tsx if needed
• Ensure stable 60 FPS (check GPU usage)

---

🚀 Performance Optimization

For Large-Scale Simulations (200+ vehicles)

1. Reduce WebSocket broadcast rate:

# In eval_demo_simple.py
time.sleep(0.05)  # 20 Hz instead of 60 Hz

2. Enable frustum culling:

// In MapViewer.tsx
<mesh frustumCulled={true}>

3. Use Level of Detail (LOD):

<Lod distances={[0, 20, 50]}>
  <DetailedCar />
  <SimpleCar />
  <DotCar />
</Lod>

---

🧪 Testing

Run Backend Tests

cd backend
pytest tests/

Run Frontend Tests

cd frontend
npm test

Run Simulation Tests

cd simulation
python -m pytest tests/

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

---

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• SUMO - Traffic simulation platform
• React Three Fiber - 3D rendering in React
• AssemblyAI - Real-time speech recognition
• Google Gemini - AI conversation engine
• ElevenLabs - Natural text-to-speech

---

⭐ Star this repo if you find it helpful!

Made with ❤️ and lots of ☕