Problem Definition
Inspection-grade synchronization of multi-modal corridor data at extreme scale
Visual task to be solved
Produce seamless 360° video aligned with ESRI while enabling synchronized LiDAR inspection across very large datasets.
Key technical constraints
- Millions of frames and heavy overlaps
- Different sampling rates across data streams
- Multi-day single-thread runtimes
- Browser memory limits for point clouds
- Secure, reliable upload and recovery
Technical success criteria
- Minutes-level processing
- Reliable handling of hundreds-GB uploads
- Improved duplicate detection via QA
- High first-pass success with retry
- Limited cost growth despite speed gains
Data Characteristics
High-volume, multi-sensor corridor capture with real-world edge cases
Data source and scale
Road-survey imagery from 10k frames to multi-million projects, plus LiDAR, telemetry, GIS basemaps, and CSV assets.
Labeling method
Algorithmic duplicate detection refined through visual QA review and operator feedback.
Primary data challenges
- Overlap duplicates and repeated coordinates
- Huge panoramas needing tiling
- Near-identical corridor frames
- Unstable field-network uploads
- Browser limits for dense point clouds
Technical Approach
Chunk-first, event-driven architecture for synchronized large-scale processing
High-level pipeline design
Chunked S3 uploads → metadata and tiling → FFmpeg stitching → redundancy pruning → LiDAR decompression, denoising, tiling → master timeline sync → streamed browser visualization.
Key design decisions or trade-offs
- Short bursts on large compute vs long small-instance runs
- Fixed-size chunks for linear scaling
- Ordinal alignment where GPS repeats
- Prefetching instead of recompute
- Object storage as source of truth
Non-obvious constraints handled
- Fly-over/under-pass duplication
- Memory-safe LiDAR tile streaming
- Hash-chain frame integrity
- Expiring role-scoped storage
- Rapid regional fail-over
Models and Algorithms
Classical video processing with targeted detection and spatial methods
Model architectures or families
Similarity correlation, hashing, YOLOv8 symbol detection, Three.js rendering, tiled point-cloud visualization.
Core algorithms or methods
- Histogram overlap pruning
- Content-aware frame hashing
- 4k tile segmentation
- Voxel-grid LiDAR denoising
- Quaternion-based synchronization
Training or optimization strategy
- Multithreaded GPU-friendly batching
- Priority-based auto-scaling
- Cached FFmpeg builds
- Human-in-loop QA feedback
- Client-side WASM preview processing
Evaluation and Results
Minutes-level rendering with strong operational impact
Metrics used
Processing time, cost per batch, success rate, duplicate accuracy, viewer latency, decision time.
Quantified performance
~24h to ~5 min
10k images processing time
72h to ~8 min
12k images processing time
~100x throughput
for ~2x spend
97% success
on 500 GB uploads
~60% reduction
in storage cost
~85% faster
quote-to-decision cycle
Baseline comparison
Replaced costly proprietary tools lacking LiDAR and scale.
Removed manual duplicate correction and fragmented GIS workflows.
Delivered single-link browser inspection with synchronized imagery, LiDAR, and ESRI data.