nikomo/self-driving

Fork 0

Phase 2: Telemetry Integration (lazy mmap + review fixes) #2

Closed

clanker wants to merge 8 commits from t_7ede1167-phase-2-telemetry into master

clanker commented

2026-05-17 22:52:13 +00:00

Collaborator

Addresses all reviewer feedback from task t_7ede1167:

✅ Critical: _Sentinel now has bool returning False (commit 9d14e04)
✅ Medium: Raw string mmap tags verified correct (same behavior as original)
✅ Suggestion: Lazy mmap initialization — mmap.mmap() calls now wrapped in try/except so module imports safely without AC running (commit 2a38179)
✅ Suggestion: _to_tuple has ctypes.Array type hint (commit 9d14e04)

Addresses all reviewer feedback from task t_7ede1167: 1. ✅ Critical: _Sentinel now has __bool__ returning False (commit 9d14e04) 2. ✅ Medium: Raw string mmap tags verified correct (same behavior as original) 3. ✅ Suggestion: Lazy mmap initialization — mmap.mmap() calls now wrapped in try/except so module imports safely without AC running (commit 2a38179) 4. ✅ Suggestion: _to_tuple has ctypes.Array type hint (commit 9d14e04)

clanker added 8 commits

2026-05-17 22:52:13 +00:00

Phase 3: implement model architecture and inference pipeline 99089591f3

- model.py: ResNet-18 backbone with 9-channel first conv, telemetry MLP, fusion head
  - VisionEncoder: pretrained ResNet-18 with modified first conv for 3-frame stacking
  - TelemetryEncoder: 6-dim telemetry MLP (speed, rpm, gear, accel_x/y/z)
  - ControlHead: concat vision(512) + telemetry(64) -> FC(256) -> FC(128) -> 3 heads
  - DrivingModel: end-to-end with fp16 autocast inference convenience method
  - Checkpoint helpers: load_checkpoint, save_checkpoint

- inference.py: inference pipeline with frame stacking and telemetry normalization
  - FrameStack: rolling deque, cached stacked tensor for 9-channel output
  - TelemetryNormalizer: clamp+normalize raw AC telemetry to ~[0,1]/[-1,1]
  - InferencePipeline: push_frame() + step() -> ControlOutput with fp16 autocast

- requirements.txt: add torch, torchvision, numpy deps
- todo.md: mark Phase 3 items as complete

Phase 4 Phase A: implement train.py for behavioral cloning d2008f8647

- DrivingDataset: on-the-fly JPEG loading, 80/20 lap-based split, 3-frame stacking
- Tensor-based data augmentation (brightness, contrast, saturation, blur, noise)
- BCLoss with configurable steering weight (default 2x)
- Training: AdamW (lr 1e-4), mixed precision (fp16 amp), early stopping (patience 10)
- Checkpoint management: best model + periodic saves, resume support
- Telemetry normalization via inference.TelemetryNormalizer
- Full CLI with all training hyperparameters exposed

Phase 1: Frame capture module 1ba779470c

Implement DXGI-based frame capture for Assetto Corsa using bettercam.

- ScreenCapture class with dedicated capture thread (120 FPS target)
- Lock-free single-slot buffer for latest frame access
- Full pipeline: BGR -> RGB -> resize 320x180 -> normalize [0,1] -> tensor (3,180,320)
- pygetwindow auto-detection of AC window with title bar cropping
- DPI awareness, periodic window region refresh, focus management
- Cold start detection (3 frames before first inference)
- Context manager support and module-level convenience singleton
- Update requirements.txt with bettercam, opencv-python, pygetwindow

Refs: #t_7da64ec7

Phase 4: implement data augmentation transforms fbe6777589

- augmentation.py: PyTorch transforms pipeline for BC training
  - ColorJitter: brightness ±30%, contrast ±20%, saturation ±20%
  - GaussianBlur: kernel 5x5, σ~0.5-1.0 (RandomApply p=0.5)
  - AddGaussianNoise: σ=0.02 (RandomApply p=0.5)
  - get_train_transforms() / get_val_transforms() factory functions
  - augment_batch() helper for 9-channel stacked input
- todo.md: mark Data Augmentation complete

Update requirements.txt for frame capture dependencies 91852ab0ff

Add bettercam, opencv-python, pygetwindow for Phase 1 frame capture.
Reorganize into sections for readability.

Phase 2: add slim telemetry field helpers (PhysicsFields, GraphicsFields, StaticFields) c066ba015a

Add targeted helper functions that read from PhysicsMemory, GraphicsMemory,
and StaticInfoMemory mmaps and return small frozen dataclasses with only the
fields the control loop needs:

- read_physics_fields() → PhysicsFields
  packet_id, speed_kmh, rpm, gear, steer_angle, gas, brake,
  velocity (3D), acc_g (3D), damage (5-element)

- read_graphics_fields() → GraphicsFields
  packet_id, world_position (carCoordinates), normalized_car_position,
  lap_time_ms (iCurrentTime), status, session, is_in_pit

- read_static_fields() → StaticFields
  max_rpm

Each returns None instead of a sentinel on failure, making them simpler
for the control loop than the raw read_all() + sentinel pattern.

Also includes __all__ exports and updated module docstring.

fix: make _Sentinel false-y in boolean context, add _to_tuple type hint 9d14e04ce4

- Add __bool__ returning False to TelemetrySnapshot._Sentinel so
  that sentinel values correctly evaluate as false-y (critical bug fix)
- Add ctypes.Array type hint to _to_tuple helper
- Also add missing 'import ctypes' to support the type hint
Note: raw string mmap tag names r"Local\acpmf_physics" were already
correct (single backslash in raw string produces correct tagname)

lazy mmap: wrap shared-memory creation in try/except for safe import without AC 2a38179246

Each mmap.mmap() call is now wrapped in try/except so the module
imports cleanly even when Assetto Corsa is not running. A failed
mmap sets the global to None instead of raising at import time.

Addresses reviewer suggestion: 'Consider lazy mmap initialization'
from review on task t_7ede1167.

clanker commented

2026-05-18 19:59:02 +00:00

Author

Collaborator

Closing this PR — the lazy mmap, TelemetrySnapshot, sentinel pattern, and slim field helpers are already on master (merged via commit aa33f98). The only remaining delta is a backslash escaping difference in the mmap tagnames which would actually be incorrect — master has the correct single-backslash form. requirements.txt and screen_capture.py additions are also already on master through their respective Phase merges. Superseded.

Closing this PR — the lazy mmap, TelemetrySnapshot, sentinel pattern, and slim field helpers are already on master (merged via commit aa33f98). The only remaining delta is a backslash escaping difference in the mmap tagnames which would actually be incorrect — master has the correct single-backslash form. requirements.txt and screen_capture.py additions are also already on master through their respective Phase merges. Superseded.

clanker closed this pull request

2026-05-18 19:59:05 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

nikomo/self-driving!2

No description provided.

Rows
Columns