Senior Perception Engineer, Obstacle Foundation Models - Autonomous Vehicles

Overview

• Intelligent machines powered by artificial intelligence—computers that can learn, reason, and interact with people—are transforming every industry.
• GPU-accelerated deep learning provides the foundation for machines to perceive, reason, and solve complex problems.
• NVIDIA GPUs run deep learning algorithms that simulate aspects of human intelligence.
• They act as the brain of computers, robots, and self-driving cars.
• These machines can perceive and interpret their surroundings.
• We are seeking an exceptional Senior Perception Engineer to help design and productize NVIDIA’s next-generation autonomous driving perception stack.
• You will work on the core 3D obstacle perception pipeline, contribute to architecture and algorithm design, and remain deeply hands-on with implementation, including modern transformer-based, multi-modal, and vision-language techniques where they add real value.
• What you'll be doing: Develop and improve the technical build, architecture, and roadmap for 3D obstacle perception to support end-to-end autonomous driving.
• Use innovative CNN and transformer-based architectures when appropriate.
• Design and implement advanced 3D perception models using multi-camera inputs and/or multi-sensor fusion (camera, radar, lidar) for obstacle detection and tracking, including opportunities to explore BEV and transformer-based 3D perception.
• Build efficient, production-grade deep learning models by defining objectives with the team.
• Select and prototype architectures, run experiments, and follow training and evaluation guidelines.
• Use techniques like large-scale pretraining, distillation, and parameter-efficient fine-tuning (e.g., LoRA).
• Help define and maintain KPI frameworks to quantify perception performance; analyze large-scale real and synthetic datasets to identify failure modes and systematically improve accuracy, robustness, and efficiency, incorporating approaches like self-supervised and representation learning when beneficial.
• Contribute to the data strategy for perception by specifying data and labeling requirements.
• Help prioritize data collection and annotation.
• Collaborate with data and ground-truth teams, including model-assisted workflows such as active learning, auto-labeling, and multimodal AI systems combining vision and language.
• Also work with model-in-the-loop tooling.
• Collaborate with safety, systems, and software teams to ensure perception solutions meet product requirements for safety, latency, resource usage, and software robustness, and are ready for deployment at scale.
• What we need to see: PhD with 4+ years, MS with 6+ years, or BS (or equivalent experience) with 8+ years of relevant experience in Computer Science, Computer Engineering, or a related technical field.
• Hands-on experience developing deep learning–based perception or closely related systems for complex real-world problems, with strong proficiency in frameworks such as PyTorch and a track record of taking models from prototype to production.
• Proven experience in data-driven development, including close collaboration with data, labeling, and validation teams on data strategy, labeling quality, and iterative model improvement.
• Strong programming skills in Python and/or C++, with experience building reliable, high-performance, production-quality software.
• Excellent communication and collaboration skills, with the ability to work effectively across multidisciplinary teams.
• Ways to stand out from the crowd: Experience designing and deploying perception solutions for autonomous driving or robotics using camera-based deep learning at scale.
• Hands-on experience architecting and deploying DNN-based perception pipelines on embedded or real-time platforms.
• This includes optimizing for latency, memory, and compute constraints.
• Experience with modern architectures such as CNNs and transformers is required.
• Familiarity with methods such as extensive pretraining, efficient tuning of parameters (e.g., LoRA), or vision-language models (VLMs) is also needed.
• Strong publication record or recognized contributions in deep learning, computer vision, or autonomous systems at leading conferences/journals (e.g., CVPR, ICCV, NeurIPS, IROS).
• Deep understanding of 3D computer vision fundamentals, including camera modeling and calibration (intrinsic and extrinsic), multi-view geometry, and 3D representations, ideally with experience applying these concepts in transformer-based 3D or BEV perception pipelines.
• Experience with CUDA development and optimizing training or inference pipelines through custom CUDA kernels or other GPU-accelerated components.