NCP-AIN NVIDIA-Certified Professional AI Networking Questions and Answers

EVPN Multi-homingenablesactive-active redundancy without inter-switch linksby usingoverlay routingover VXLAN and distributed control plane using BGP EVPN.

From the officialNVIDIA Cumulus Linux EVPN Multihoming Documentation:

"EVPN multihoming allows multiple Top-of-Rack (ToR) switches to connect to the same server while maintaining full layer-2 redundancy without the need for inter-switch links or traditional MLAG configuration."

Key benefits:

Simplified topology (no ISL/peer-link needed)

BGP-based control plane

Fast convergence

Active-active links per host NIC

Incorrect Options:

MLAGrequires ISL between switches and peer-link configuration.

VSS(Virtual Switching System) is a Cisco term, not supported in NVIDIA networking.

TheNVIDIA DGX SuperPODis a turnkey AI supercomputing infrastructure designed for large-scale deep learning and high-performance computing workloads. It integrates multiple DGX systems with high-speed networking and storage solutions, providing a scalable and efficient platform for AI research institutions. The architecture supports rapid deployment and is optimized for training complex models, making it the ideal choice for environments demanding top-tier AI performance.​

The "What Just Happened" (WJH) service on Cumulus switches provides real-time visibility into network problems by monitoring various layers and components, including layer 1, layer 2, layer 3, tunnel, buffer, and Access Control List (ACL) related issues. WJH streams detailed and contextual telemetry data, enabling administrators to diagnose and troubleshoot network problems effectively.​

Reference Extracts from NVIDIA Documentation:

"WJH can monitor layer 1, layer 2, layer 3, tunnel, buffer and ACL related issues."​

"The WJH service enables you to diagnose network problems by looking at dropped packets."​

The ibdiagnet utility is a fundamental tool for InfiniBand fabric discovery, error detection, and diagnostics. It provides comprehensive reports on the fabric's health, including error reporting, switch and Host Channel Adapter (HCA) configuration dumps, various counters reported by the switches and HCAs, and parameters of devices such as switch fans, power supply units, cables, and PCI lanes. Additionally, ibdiagnet performs validation for Unicast Routing, Adaptive Routing, and Multicast Routing to ensure correctness and a credit-loop-free routing environment.​

Reference Extracts from NVIDIA Documentation:

"The ibdiagnet utility is one of the basic tools for InfiniBand fabric discovery, error detection and diagnostic. The output files of the ibdiagnet include error reporting, switch and HCA configuration dumps, various counters reported by the switches and the HCAs."​

"ibdiagnet also performs Unicast Routing, Adaptive Routing and Multicast Routing validation for correctness and credit-loop free routing."​

Virtual Routing and Forwarding (VRF)is the most effective method to achievenetwork segmentation and isolationin a multi-tenant environment.

From theNVIDIA Cumulus Linux Documentation – VRF Section:

"VRF allows multiple instances of routing tables to coexist within the same switch, effectively isolating traffic between tenants or departments."

Each department can:

Operate in its own VRF domain

Have independent routing tables

Maintain strict separation of Layer 3 paths

Incorrect Options:

A (Port Mirroring)– Used for traffic monitoring, not isolation.

C (ACLs)– Useful for fine-grained filtering, but not scalable tenant isolation.

D (LLDP)– Used for neighbor discovery, not security or isolation.

MLNX-OS, the operating system for NVIDIA's networking devices, defines two primary user account types:adminandmonitor. Theadminaccount has full administrative privileges, allowing for complete configuration and management of the system. Themonitoraccount, on the other hand, is designed for users who need to view system configurations and statuses without making any changes. This separation ensures a clear distinction between users who manage the system and those who monitor its operations.​

Reference Extracts from NVIDIA Documentation:

"There are two user roles or account types: admin and monitor. As 'admin', the user is privileged to run all the available commands. As 'monitor', the user can run commands that show system configuration and status, or set terminal settings."​

MLNX-OS is the network operating system used on NVIDIA’s Mellanox Ethernet switches, including the Spectrum family (e.g., Spectrum-4 switches in the Spectrum-X platform), designed for high-performance Ethernet networking in AI and HPC data centers. MLNX-OS provides a command-line interface (CLI) for configuring and managing switch operations, with user accounts controlling access to various commands and functions. The question asks for the two general user account types in MLNX-OS, which define the primary privilege levels for user access.

According to NVIDIA’s official MLNX-OS documentation, the two general user account types in MLNX-OS are:

monitor: This account type has read-only access, allowing users to view configurations, status, and logs but not modify settings. It is used for monitoring and troubleshooting without risking unintended changes.

admin: This account type has full read-write access, enabling users to view and modify all configurations, execute commands, and manage the switch’s operations. It is intended for administrators with complete control over the system.

These two account types represent the primary privilege levels in MLNX-OS, providing a clear distinction between read-only monitoring and full administrative access.

Exact Extract from NVIDIA Documentation:

“MLNX-OS supports two primary user account types for managing switch operations:

monitor: Users with monitor privileges have read-only access to the system. They can view configuration details, system status, and logs but cannot make changes to the configuration.

admin: Users with admin privileges have full read-write access, allowing them to configure, manage, and troubleshoot all aspects of the switch, including executing privileged commands.These account types ensure secure and controlled access to the switch’s management functions.”—NVIDIA MLNX-OS User Manual

This extract confirms that options B (monitor) and C (admin) are the correct answers. These account types are the standard privilege levels in MLNX-OS, used to manage access for monitoring and administrative tasks on Spectrum switches, including those in Spectrum-X deployments.

In the Spectrum-X architecture,BlueField-3 SuperNICsare responsible for executing the congestion control algorithm. They handle millions of congestion control events per second with microsecond reaction latency, applying fine-grained rate decisions to manage data flow effectively. This ensures optimal network performance by preventing congestion and packet loss.​

AI workloads, particularly in large-scale training scenarios, are characterized by a small number of high-bandwidth, long-lived flows known as "elephant flows." These flows can dominate network traffic and are prone to causing congestion if not managed effectively.​

Traditional flow-based load balancing mechanisms, such as Equal-Cost Multipath (ECMP), distribute traffic based on flow hashes. However, in AI workloads with lowentropy (i.e., limited variability in flow characteristics), ECMP can lead to uneven traffic distribution and congestion on certain paths.​

Adaptive routing techniques, which dynamically adjust paths based on real-time network conditions, are more effective in managing AI traffic patterns and mitigating congestion risks.​

To check the status and link layer of InfiniBand interfaces, the ibstat command is used. For example:​

ibstat -d mlx5_0

This command provides detailed information about the InfiniBand device, including its state (e.g., Active), physical state (e.g., LinkUp), and link layer (e.g., InfiniBand).​

Configuring NVUE to ignore certain underlying Linux files allows administrators to manage specific settings manually or through automation tools like Ansible withoutNVUE overwriting these configurations. This approach enables the persistent manipulation of settings using both NVUE and flat-file methods, providing flexibility in network management.

NVIDIA AIR (AI-Ready Infrastructure) is a cloud-based simulation platform designed to model and validate data center network deployments, including Spectrum-X Ethernet networks, using realistic topologies and configurations. When creating a custom topology in NVIDIA AIR, users can define network links between devices (e.g., spine and leaf switches) using a DOT file format, which is based on the Graphviz graph visualization software. The question asks for the correct syntax to define a link between port 1 on a spine switch (spine-01) and port 41 on a leaf switch (gpu-leaf-01) in a NVIDIA AIR simulation.

According to NVIDIA’s official NVIDIA AIR documentation, the DOT file format is used to specify network topologies, including nodes (devices) and links (connections between ports). The syntax for defining a link in a DOT file uses a double dash (--) to indicate a connection between two ports, with each port specified in the format "":"". For Spectrum-X networks, which typically use Cumulus Linux or SONiC on NVIDIA Spectrum switches, ports are commonly labeled as swpX (switch port X) rather than ethX (Ethernet interface), especially for switch-to-switch connections in a leaf-spine topology. The correct syntax for the link between port 1 on spine-01 and port 41 on gpu-leaf-01 is:

"spine-01":"swp01" -- "gpu-leaf-01":"swp41"

This syntax uses swp01 and swp41 to denote switch ports, consistent with Cumulus Linux conventions, and the double dash (--) to indicate the link, as required by the DOT file format.

Exact Extract from NVIDIA Documentation:

“You can create custom topologies in Air using a DOT file, which is the file type used with the open-source graph visualization software, Graphviz. DOT files define nodes, attributes, and connections for generating a topology for a network. The following is an example of a link definition in a DOT file:

"leaf01":"swp31" -- "spine01":"swp1"

This specifies a connection between port swp31 on leaf01 and port swp1 on spine01. Port names typically follow the switch port naming convention (e.g., swpX) for Cumulus Linux-based switches.”

—NVIDIA Air Custom Topology Guide

This extract confirms that option A is the correct answer, as it uses the proper DOT file syntax with swp01 and swp41 for port names and the double dash (--) for the link, aligning with NVIDIA AIR’s topology definition process for Spectrum-X simulations.

Analysis of Other Options:

B. "spine-01":"swp1" to "gpu-leaf-01":"swp41": This option uses the correct port naming convention (swp1 and swp41) but incorrectly uses the word to as the connector instead of the double dash (--). The DOT file format requires -- to define links, making this syntax invalid for NVIDIA AIR.

C. "spine-01":"eth1" to "gpu-leaf-01":"eth41": This option uses ethX port names, which are typically used for host interfaces (e.g., servers) rather than switch ports in Cumulus Linux or SONiC environments. Switch ports in Spectrum-X topologies are labeled swpX. Additionally, the use of to instead of -- is incorrect for DOT file syntax, making this option invalid.

D. "spine-01":"eth1" - "gpu-leaf-01":"eth41": This option uses a single dash (-) instead of the required double dash (--) and incorrectly uses ethX port names instead of swpX. The ethX naming is not standard for switch ports in Spectrum-X, and the single dash is not valid DOT file syntax, making this option incorrect.

Why "spine-01":"swp01" -- "gpu-leaf-01":"swp41" is the Correct Answer:

Option A correctly adheres to the DOT file syntax used in NVIDIA AIR for defining network links:

Node and Port Naming: The nodes spine-01 and gpu-leaf-01 are specified with their respective ports swp01 and swp41, following the swpX convention for switch ports in Cumulus Linux-based Spectrum-X switches.

Link Syntax: The double dash (--) is the standard connector in DOT files to indicate a link between two ports, as required by Graphviz and NVIDIA AIR.

Spectrum-X Context: In a Spectrum-X leaf-spine topology, connections between spine and leaf switches (e.g., Spectrum-4 switches) use switch ports labeled swpX, making swp01 and swp41 appropriate for this simulation.

This syntax ensures that the NVIDIA AIR simulation accurately models the physical connection between spine-01 port 1 and gpu-leaf-01 port 41, enabling validation of the Spectrum-X network topology. The DOT file can be uploaded to NVIDIA AIR to generate the topology, as described in the documentation.

In Cumulus Linux, microbursts are short-lived, high-volume traffic bursts that often go undetected by coarse-grained monitoring like SNMP.

The two tools specifically used for this purpose are:

What Just Happened (WJH)

"WJH provides real-time packet drop visibility and classifies drops by reason (e.g., congestion, ACLs, etc.), enabling microburst detection."

ASIC monitoring at millisecond granularity

"Deep telemetry is enabled via the switch ASIC, which provides sub-second counters that capture microburst patterns otherwise missed by SNMP."

Incorrect Options:

AandCprovide low-frequency sampling, insufficient for microbursts which last milliseconds.

The correct answer isPXN (Peer eXchange Network).

From theNVIDIA NCCL Documentation:

"PXN enables communication between GPUs connected via NVLink and NICs by treating the GPUs as a distributed switch. This architecture improves bandwidth utilization by enabling any GPU to communicate with the NIC via the shortest path available, even if it's not directly connected to the NIC."

This enhances GPU-to-NIC and NIC-to-GPU transfers, leveraging the NVLink topology. It significantly boosts performance in multi-GPU setups where not every GPU is directly connected to the NIC.

Other options:

Adaptive Routingis a fabric-level feature for dynamic path rerouting.

Graph Search Optimizationis used internally for topology modeling in NCCL.

SHARP v2is a switch-based collective acceleration method, unrelated to PXN.

TheMINHOProuting protocol, while efficient in finding minimal paths, does not inherently prevent credit loops. This can lead to deadlocks in the network. In contrast, routing protocols like UPDOWN and FAT TREE are designed to avoid such loops, ensuring more reliable network operation.​

InDPU Mode, theARM cores on BlueFieldown theNIC data path, while still allowing thehost system to access the DPU OS (via OOB or virtio).

From NVIDIA BlueField Documentation:

"In DPU Mode, the data path is offloaded to the BlueField Arm cores, enabling advanced security and networking functions, while still allowing host access to the BlueField OS."

This is different from:

NIC Mode: Data path controlled by host, ARM cores inactive.

Separated Host Mode: Complete isolation; host cannot access DPU OS.

Restricted Mode: Limited host access to DPU OS, but without full offload capabilities.

UFM Telemetry: Provides real-time monitoring and analysis of network performance, collecting data such as port counters and cable information to assess the health and efficiency of the network.

UFM Enterprise: Focuses on comprehensive network management and optimization, enabling administrators to monitor, operate, and optimize InfiniBand scale-out computing environments effectively.

UFM Cyber AI: Detects and mitigates network security threats by analyzing telemetry data to identify anomalies and potential security issues within the network infrastructure.​

Reference Extracts from NVIDIA Documentation:

"UFM Telemetry provides real-time monitoring and analysis of network performance."​

"UFM Enterprise is a powerful platform for managing InfiniBand scale-out computing environments."​

"UFM Cyber-AI enhances the benefits of UFM Telemetry and UFM Enterprise services by detecting and mitigating network security threats."​

Before upgrading Cumulus Linux, it's essential to back up configuration files to a different server. The /etc directory is the primary location for all configuration data in Cumulus Linux. Specifically, the following files and directories should be backed up:​

/etc/frr/ - Routing application (responsible for BGP and OSPF)

/etc/hostname - Configuration file for the hostname of the switch

/etc/network/ - Network configuration files, most notably /etc/network/interfaces and /etc/network/interfaces.d/

/etc/cumulus/acl - Access control list configurations​

Cumulus Linux is a network operating system used on NVIDIA Spectrum switches, including those in the Spectrum-X platform, to provide a Linux-based environment forEthernet networking in AI and HPC data centers. When upgrading Cumulus Linux to a new version, it’s critical to migrate specific configuration files to preserve network settings and ensure continuity. The question asks for the two configuration file locations that should be migrated from the old installation during an upgrade.

According to NVIDIA’s official Cumulus Linux documentation, the key directories containing configuration files that should be migrated during an upgrade are /etc/cumulus/acl (for access control list configurations) and /etc/network (for network interface configurations). These directories store critical network settings that define the switch’s behavior, such as ACL rules and interface settings, which must be preserved to maintain network functionality after the upgrade.

Exact Extract from NVIDIA Documentation:

“When upgrading Cumulus Linux, you must back up and migrate specific configuration files to ensure continuity of network settings. The following directories should be included in the backup:

/etc/cumulus/acl: Contains access control list (ACL) configuration files that define packet filtering and security policies.

/etc/network: Contains network interface configuration files, such as interfaces and ifupdown2 settings, which define the network interfaces and their properties.Back up these directories before upgrading and restore them after the new version is installed to maintain consistent network behavior.”—NVIDIA Cumulus Linux Upgrade Guide

This extract confirms that options A and B are the correct answers, as /etc/cumulus/acl and /etc/network contain essential configuration files that must be migrated during a Cumulus Linux upgrade. These files ensure that ACL policies and network interface settings are preserved, which are critical for Spectrum-X configurations in AI networking environments.

FromNVIDIA Performance Tuning Guide (ib_write_bw Tool Usage):

"-S <SL>: Specifies the Service Level (SL) to use for the InfiniBand traffic. SL is used for setting priority and mapping to virtual lanes (VLs) on the IB fabric."

This flag is useful when testing QoS-aware setups or validating SL/VL mappings.

Incorrect Options:

A– No such flag for burst size.

B– -q defines number of QPs.

C– --rate or -R is used for rate-limiting.