Assetto Corsa Autonomous Driving Agent

An end-to-end autonomous driving agent for Assetto Corsa (AC) built using PyTorch. The agent employs a hybrid neural architecture: a 3-frame stacked ResNet-18 CNN for spatial visual processing, fused with a Multi-Layer Perceptron (MLP) processing real-time telemetry inputs (Speed, RPM, Gear, G-forces).

The pipeline supports two major training phases:

1. Behavioral Cloning (BC): Learning directly from human driving demonstrations.
2. Reinforcement Learning (RL): Fine-tuning the policy in-game using Proximal Policy Optimization (PPO) combined with a KL-divergence penalty toward the BC prior to prevent catastrophic forgetting.

---

Project Architecture

graph TD
    A[Assetto Corsa Game] -->|Shared Memory| B(AssettoMemory.py)
    A -->|DXGI Capture| C(screen_capture.py)
    B -->|Telemetry Vector| D[TelemetryNormalizer]
    C -->|RGB Stack| E[FrameStack]
    D -->|6x1 Tensor| F[DrivingModel / ActorCritic]
    E -->|9x180x320 Tensor| F
    F -->|Raw Means| G[Control Signals]
    G -->|vJoy API| H(controller.py)
    H -->|Virtual Inputs| A

Key Modules

• model.py: Defines DrivingModel (ResNet-18 backbone + Telemetry MLP fusion network).
• inference.py: Contains input pipeline components (TelemetryNormalizer and FrameStack).
• controller.py: Interactivity layer mapping outputs to standard vJoy steering, throttle, and brake axes, with an automatic rule-based GearShifter.
• screen_capture.py: High-performance DXGI-based screen grabber utilizing bettercam running in a dedicated capture thread.
• AssettoMemory.py: Connects to the Assetto Corsa Shared Memory (Local\acpmf_physics, Local\acpmf_graphics, Local\acpmf_static) via ctypes.
• record.py: Data recording tool to log human demonstrations (images + telemetry parquet tables).
• train.py: Phase A training script utilizing behavioral cloning (supervised MSE/L1 loss).
• train_rl.py: Phase B training script utilizing PPO and automatic restarts for reinforcement learning fine-tuning.
• main.py: Real-time evaluation loop and state machine driving agent.

---

Setup & Installation

Important

This project requires Windows 8+ and a running installation of Assetto Corsa. It relies on Windows-specific libraries (pyvjoy, Windows Named Shared Memory).

1. Prerequisites

• Assetto Corsa with the ACPMF (Assetto Corsa Shared Memory) Plugin enabled.
• vJoy Device: Install vJoy and configure Device 1 with at least 3 axes (X, Y, Z for Steering, Throttle, Brake) and gear shift buttons.
• Python: Python 3.10 to 3.14.

2. Install Dependencies

Activate your virtual environment and install the required Python packages:

.venv\Scripts\activate
pip install -r requirements.txt

Note: For PyTorch CUDA support on modern GPUs (e.g., RTX 4080), install the matching CUDA-enabled wheel:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

3. Bind vJoy to Assetto Corsa

Run the binding helper script to register your virtual controller inputs in Assetto Corsa's settings menu:

python vjoy_bind_to_ac.py

To verify vJoy axes output correctness independently, run:

python vjoy_test.py

---

Workflow Pipeline

Phase 1: Data Collection (Human Driving)

Launch Assetto Corsa, enter a track session, and run record.py to save images and telemetry rows:

python record.py

• Press R to toggle recording on/off in-game.
• Press Q or ESC to save the recorded lap and exit.
• Data is saved under data/lap_YYYYMMDD_HHMMSS/.

Phase 2: Behavioral Cloning Training

Train the fusion model on your recorded driving data:

python train.py --data-dir data --checkpoint-dir checkpoints --epochs 100 --device cuda

This saves the best-performing weights as checkpoints/bc_best.pt.

Phase 3: Reinforcement Learning Fine-Tuning

Fine-tune the behavioral cloning weights in-game using PPO:

python train_rl.py --bc-checkpoint checkpoints/bc_best.pt --total-timesteps 1000000 --device cuda

• The script uses simulated key sequences via keyboard/pydirectinput to automatically restart the session on crashes or lap completions.

Phase 4: Autonomous Driving Agent

Evaluate or run the trained agent in real-time:

python main.py --model-path checkpoints/bc_best.pt --device cuda

• Press F10 to toggle agent control (enable/disable driving).
• Press F11 to stop the agent, reset steering/throttle inputs, and exit.

---

Safety and Crash Recovery

The driving loop (main.py) integrates a state machine with the following states:

1. WAITING_FOR_SESSION: Waits for Assetto Corsa status to be LIVE and the vehicle to exit the pits.
2. DRIVING: Runs inference and sets controls. Detects crashes based on:
   • Stuck: Speed < 5 km/h while off-track boundary for > 3 seconds.
   • Damage: Sudden damage spikes.
   • Reversing: Moving backwards on the track.
3. CRASHED: Instantly resets control inputs.
4. RESETTING: Executes key emulation (ESC -> down -> down -> ENTER) to restart the session, returning to WAITING_FOR_SESSION.