Senior Solutions Architect, AI Factory Deployment - NVIS

We are in search of a curious and motivated Senior Solutions Architect to join our NVIDIA Infrastructure Specialists team. In this capacity, you'll support the creation, implementation, and verification of AI factories, focusing on running and debugging AI/LLM workloads and benchmarks on Linux-based GPU clusters. You'll engage with NCCL and collectives like AllReduce and AllToAll to boost performance and scalability, receiving mentorship from senior architects on the team. You will apply observability and automation to improve our benchmarking and validation efforts. You will be a key contact for troubleshooting workloads and benchmarks that fail, hang, or perform poorly. Additionally, you will collaborate with various NVIDIA teams to prepare AI factories for customers, validating both hardware and software for current AI applications. What You Will Be Doing: Set up, adjust, and verify AI factory environments across multi-GPU and multi-node Linux clusters. Validate configurations against guidelines for NCCL, collectives, and distributed training frameworks. Run key AI/LLM benchmarks — setup, orchestration, result collection, and analysis. Investigate and address problems when training jobs or benchmarks fail, hang, or perform below expectations. Build and improve observability for AI factories (metrics, logs, traces, dashboards) to understand workload behavior and system health. Build automation using Python and Shell for conducting benchmarks, retrieving results, and completing regression checks. Analyze communication patterns and NCCL usage for AI/LLM workloads, concentrating on collectives such as AllReduce and AllToAll. Help identify and recommend improvements to job configuration, parallelism strategies, and cluster settings to improve throughput, latency, and scaling efficiency. Work closely with hardware, software, networking, and product teams to prepare AI factories for customer use. Contribute to documentation and readiness materials for internal and custo