NCP-AIN NVIDIA-Certified Professional AI Networking Questions and Answers

For optimal performance in multi-node, multi-GPU AI workloads:

NVLink provides high-speed, low-latency communication between GPUs within the same node.

InfiniBand offers efficient, scalable communication between nodes in a data center. Combining these technologies ensures both intra-node and inter-node communication needs are effectively met.​

The mlxlink tool is used to check and debug link status and issues related to them. The tool can be used on different links and cables (passive, active, transceiver, and backplane). It is intended for advanced users with appropriate technical background.​

In InfiniBand networks, Partitioning using Partition Keys (PKeys) is the standard method for implementing multi-tenancy and traffic isolation. PKeys allow administrators to define logical partitions within the fabric, ensuring that traffic is confined to designated groups of nodes. This mechanism is essential for creating secure and isolated environments in multi-tenant architectures.​

The Unified Fabric Manager (UFM) leverages PKeys to manage these partitions effectively, enabling administrators to assign and control access rights across different tenants. This approach ensures that each tenant's traffic remains isolated, maintaining both security and performance integrity within the shared fabric.​

Direct Memory Access (DMA) in InfiniBand networks allows data to be transferred directly between the memory of two devices without involving the CPU. This capability significantly reduces CPU overhead, lowers latency, and increases throughput, making it ideal for AI workloads that demand efficient data transfers.

NVIDIA NetQ is a network operations tool that provides real-time visibility and automated validation of the network fabric. It helps in identifying misconfigurations, monitoring network health, and ensuring that the fabric meets the required specifications for AI workloads.​

MLNX-OS, the operating system for NVIDIA's networking devices, defines two primary user account types: admin and monitor . The admin account has full administrative privileges, allowing for complete configuration and management of the system. The monitor account, on the other hand, is designed for users who need to view system configurations and statuses without making any changes. This separation ensures a clear distinction between users who manage the system and those who monitor its operations.​

Reference Extracts from NVIDIA Documentation:

"There are two user roles or account types: admin and monitor. As 'admin', the user is privileged to run all the available commands. As 'monitor', the user can run commands that show system configuration and status, or set terminal settings."​

MLNX-OS is the network operating system used on NVIDIA’s Mellanox Ethernet switches, including the Spectrum family (e.g., Spectrum-4 switches in the Spectrum-X platform), designed for high-performance Ethernet networking in AI and HPC data centers. MLNX-OS provides a command-line interface (CLI) for configuring and managing switch operations, with user accounts controlling access to various commands and functions. The question asks for the two general user account types in MLNX-OS, which define the primary privilege levels for user access.

According to NVIDIA’s official MLNX-OS documentation, the two general user account types in MLNX-OS are:

monitor : This account type has read-only access, allowing users to view configurations, status, and logs but not modify settings. It is used for monitoring and troubleshooting without risking unintended changes.

admin : This account type has full read-write access, enabling users to view and modify all configurations, execute commands, and manage the switch’s operations. It is intended for administrators with complete control over the system.

These two account types represent the primary privilege levels in MLNX-OS, providing a clear distinction between read-only monitoring and full administrative access.

Exact Extract from NVIDIA Documentation:

“MLNX-OS supports two primary user account types for managing switch operations:

monitor : Users with monitor privileges have read-only access to the system. They can view configuration details, system status, and logs but cannot make changes to the configuration.

admin : Users with admin privileges have full read-write access, allowing them to configure, manage, and troubleshoot all aspects of the switch, including executing privileged commands. These account types ensure secure and controlled access to the switch’s management functions.” — NVIDIA MLNX-OS User Manual

This extract confirms that options B (monitor) and C (admin) are the correct answers. These account types are the standard privilege levels in MLNX-OS, used to manage access for monitoring and administrative tasks on Spectrum switches, including those in Spectrum-X deployments.

To enforce strict multi-tenancy , where:

Tenant A’s GPU cannot talk to Tenant B’s GPU

But both can access shared storage

The correct solution is:

Storage system → Full PKey membership

Each tenant’s GPU → Limited PKey membership

From the NVIDIA InfiniBand P_Key Partitioning Guide :

"A port with limited membership can only communicate with full members of the same PKey. It cannot communicate with other limited members, even within the same partition."

This isolates tenants from each other , while allowing shared access to storage .

Incorrect Options:

A permits tenant-to-tenant communication.

B isolates everything, including access to storage.

C prevents GPU access to storage.

NVIDIA BlueField Data Processing Units (DPUs) are integral to securing AI infrastructures, especially in environments requiring stringent data privacy and security measures. BlueField DPUs offload and accelerate critical infrastructure tasks such as encryption, firewall enforcement, and intrusion detection, thereby isolating sensitive data paths from potential threats.​

In the context of AI workloads, BlueField DPUs enable secure and efficient data movement between GPUs and storage systems, ensuring that sensitive information, like financial data, is protected during both training and inference processes. Their integration into NVIDIA's reference architectures provides a hardware root of trust, essential for maintaining data integrity and compliance with security standards.​

The CloudAI Benchmark is designed to evaluate and establish performance baselines in AI-optimized networking environments like NVIDIA Spectrum-X. It assesses various performance metrics, including throughput and latency, ensuring that the network meets the demands of AI workloads. This benchmarking is essential for validating the benefits of Spectrum-X and for ongoing performance monitoring.​

Accessing the BlueField DPU Operating System (OS) is possible through rshim , either over PCIe or USB, and via SSH through the OOB interface when in DPU mode.

From the NVIDIA BlueField Software Documentation :

"You can access the BlueField OS through the rshim interface. The rshim module enables host-to-DPU communication either via PCIe (default) or USB."

B. rshim over PCIe : Default when BlueField is installed in a host.

D. rshim over USB : Useful for provisioning or systems without PCIe drivers.

Incorrect Options:

A (NIC mode) : BlueField acts as a transparent NIC; OS access is not available to the host.

C (Redfish) : Redfish is for out-of-band management, not direct OS-level access.

The SN5600 smart-leaf/spine/super-spine switch offers 64 ports of 800GbE in a dense 2U form factor. The SN5600 offers diverse connectivity in combinations of 1 to 800GbE and boasts an industry-leading total throughput of 51.2Tb/s.​

According to the NVIDIA Spectrum-X Switch Operating System (SX_OS) Troubleshooting Guide , the System Status LED behavior is a critical indicator of the switch’s internal operational state.

From the document:

“The System Status LED will blink green during system initialization. If the LED continues blinking for more than 5 minutes, it indicates that the Onyx OS has failed to load properly. The system may be stuck in the boot process, or the file system may be corrupted.”

This blinking LED beyond normal initialization time indicates that the system has either encountered a failure during software boot or is unable to transition from bootloader to the OS runtime environment (i.e., Onyx).

Key causes include:

Corrupted or missing system files.

Failed firmware or OS upgrade attempts.

Boot device (e.g., eMMC or SSD) issues or corrupted partitions.

Technically, during power-on:

The switch performs POST (Power-On Self Test).

Then the Onyx OS attempts to load from the boot partition.

If the Onyx OS kernel or root filesystem is invalid, the system halts boot, and the LED remains in a blinking state , as no successful OS load confirmation is triggered.

Remediation Steps (as per NVIDIA guide):

Access the switch through console and monitor boot logs .

Use ONIE recovery or re-flash a stable Onyx OS version.

Check system storage integrity using built-in diagnostics.

Exact Extract Reference:

Source: NVIDIA SX_OS 3.9.3000 Documentation

Topic: Troubleshooting System Status LED

Extract: "If the LED blinks for more than 5 minutes and the switch is not accessible via CLI, the Onyx software failed to load properly and recovery procedures must be initiated."

=============

The RDMA Shared Device Plugin is a critical component in the NicClusterPolicy custom resource for enabling Remote Direct Memory Access (RDMA) capabilities in Kubernetes clusters. RDMA allows for high-throughput, low-latency networking, which is essential for efficient GPU-to-GPU communication across nodes in AI workloads. By deploying the RDMA Shared Device Plugin, the cluster can leverage RDMA-enabled network interfaces, facilitating direct memory access between GPUs without involving the CPU, thus optimizing performance.​

Reference Extracts from NVIDIA Documentation:

"RDMA Shared Device Plugin: Deploy RDMA Shared device plugin. This plugin enables RDMA capabilities in the Kubernetes cluster, allowing high-speed GPU-to-GPU communication across nodes."​

"The RDMA Shared Device Plugin is responsible for advertising RDMA-capable network interfaces to Kubernetes, enabling pods to utilize RDMA for high-performance networking."​

To enable SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) aggregation using the NCCL-SHARP plugin , the following two environment variables are required:

NCCL_COLLNET_ENABLE=1

Enables NCCL’s support for CollNet (Collective Network) operations, including SHARP.

NCCL_SHARP_AUTOINIT=1

Automatically initializes the SHARP plugin when available, activating SHARP-based collectives.

From the NVIDIA NCCL User Guide – SHARP Plugin Section :

"NCCL_COLLNET_ENABLE must be set to enable collective network acceleration features."

"NCCL_SHARP_AUTOINIT enables automatic SHARP plugin integration at NCCL runtime."

Incorrect Options:

B. NCCL_ALGO=CollNet – This variable controls the algorithm used for collectives but does not enable SHARP.

C. NCCLSPECTRUM_ENABLE – This is not a documented NCCL variable.

The correct answer is PXN (Peer eXchange Network) .

From the NVIDIA NCCL Documentation :

"PXN enables communication between GPUs connected via NVLink and NICs by treating the GPUs as a distributed switch. This architecture improves bandwidth utilization by enabling any GPU to communicate with the NIC via the shortest path available, even if it's not directly connected to the NIC."

This enhances GPU-to-NIC and NIC-to-GPU transfers, leveraging the NVLink topology. It significantly boosts performance in multi-GPU setups where not every GPU is directly connected to the NIC.

Other options:

Adaptive Routing is a fabric-level feature for dynamic path rerouting.

Graph Search Optimization is used internally for topology modeling in NCCL.

SHARP v2 is a switch-based collective acceleration method, unrelated to PXN.

To check the status and link layer of InfiniBand interfaces, the ibstat command is used. For example:​

ibstat -d mlx5_0

This command provides detailed information about the InfiniBand device, including its state (e.g., Active), physical state (e.g., LinkUp), and link layer (e.g., InfiniBand).​

EVPN Multi-homing enables active-active redundancy without inter-switch links by using overlay routing over VXLAN and distributed control plane using BGP EVPN.

From the official NVIDIA Cumulus Linux EVPN Multihoming Documentation :

"EVPN multihoming allows multiple Top-of-Rack (ToR) switches to connect to the same server while maintaining full layer-2 redundancy without the need for inter-switch links or traditional MLAG configuration."

Key benefits:

Simplified topology (no ISL/peer-link needed)

BGP-based control plane

Fast convergence

Active-active links per host NIC

Incorrect Options:

MLAG requires ISL between switches and peer-link configuration.

VSS (Virtual Switching System) is a Cisco term, not supported in NVIDIA networking.

The Local Route Header (LRH) in InfiniBand is termed "local" because it is used exclusively for routing packets within a single subnet. The LRH contains the destination and source Local Identifiers (LIDs), which are unique within a subnet, facilitating efficient routing without the need for global addressing. This design optimizes performance and simplifies routing within localized network segments.​

InfiniBand is a high-performance, low-latency interconnect technology widely used in AI and HPC data centers, supported by NVIDIA’s Quantum InfiniBand switches and adapters. The Local Routing Header (LRH) is a critical component of the InfiniBand packet structure, used to facilitate routing within an InfiniBand fabric. The question asks why the LRH is called a “local header,” which relates to its role in the InfiniBand network architecture.

According to NVIDIA’s official InfiniBand documentation, the LRH is termed “‘local’ because it contains the addressing information necessary for routing packets between nodes within the same InfiniBand subnet.” The LRH includes fields such as the Source Local Identifier (SLID) and Destination Local Identifier (DLID), which are assigned by the subnet manager to identify the source and destination endpoints within the local subnet. These identifiers enable switches to forward packets efficiently within the subnet without requiring global routing information, distinguishing the LRH from the Global Routing Header (GRH), which is used for inter-subnet routing.

Exact Extract from NVIDIA Documentation:

“The Local Routing Header (LRH) is used for routing InfiniBand packets within a single subnet. It contains the Source LID (SLID) and Destination LID (DLID), which are assigned by the subnet manager to identify the source and destination nodes in the local subnet. The LRH is called a ‘local header’ because it facilitates intra-subnet routing, enabling switches to forward packets based on LID-based forwarding tables.”

— NVIDIA InfiniBand Architecture Guide

This extract confirms that option A is the correct answer, as the LRH’s primary function is to route traffic between nodes within the local subnet, leveraging LID-based addressing. The term “local” reflects its scope, which is limited to a single InfiniBand subnet managed by a subnet manager.