Eric Sloof - NTPRO.NL

This article describes what the consequences are when configuring an NSX-T Tier 1 Gateway with an Edge Cluster. In my lab environment, I have attached virtual machine sa-web-02 to the Web-Segment, this segment is attached to Gateway-A. The sa-app-01 VM is attached to the App-Segment and this segment is attached to Gateway-B. Both gateways are attached to the BGP-T0-GW-01, a tier 0 gateway.

In the Network Topology overview, there’s also a service active on both gateways. This is the gateway firewall, but since no edges are configured on the Tier 1 gateways. The firewall isn’t active. There is no edge available to host the SRs of these gateway firewalls.

Let’s take a look at the Traceflow troubleshooting tool, available in the Manager user interface.

The packets flow east west through the DR of Gateway-A, the DR of the BGP-T0-GW-01 tier 0 gateway and the DR of Gateway-B. A DR is a distributed router hosted in the kernel of an ESXi host. This all happens on sa-esxi-05 because routing is performed on the host where the packet originates. The source and destination virtual machines are hosted on different transport nodes, that’s why the packets travel through the overlay network between TEPs 172.20.11.152 and 172.20.11.151 after all the routing has been completed. 

An edge cluster is needed for hosting service routers for the Tier 1 gateways. Services like NAT, gateway firewall, and DHCP are not distributed. When you want to offer these services to a Tier 1 gateway, you need to configure an edge cluster. Let’s configure the gateways with an edge cluster and see what happens. 

After performing a Re-Trace with the Traceflow utility, we can clearly see that the logical path is the same, but the physical path now contains two edges. This is because both gateways are automatically configured with a north-south firewall. The SR component of these firewalls is hosted on the edges, so the traffic must travel through these edges for the egress and ingress gateway firewall.

Both tier 1 gateways are assigned to a different edge instance in the cluster, so traffic jumps from sa-nsxedge-02 to sa-nsxedge-01. Every time traffic jumps from a transport node to another transport node, the traffic needs to be encapsulated and decapsulated to enter and leave the overlay network. The default firewall rule on a gateway is any-any-allow.

If you want to avoid hair pinning, don’t configure an edge cluster on a Tier 1 gateway. The gateway firewall is always enabled. When you want to offer NAT to segment attached to the Tier 1 gateway, and you must configure an edge cluster, the firewall can be bypassed. 

The sequential steps in Traceflow show traffic entering sa-nsx-edge-01 for NAT only, the firewall is not in the data path.

In this technical white paper, VMware walks through the steps to enable RoCE SR-IOV on a dual-port Mellanox ConnectX-5 VPI adapter card in VMware vSphere 7.x.

They cover the steps from the BIOS, ESXi, and the vSphere Client to the functionality test on the VM guest operating system. They also introduce how to use the vHPC toolkit, an open-source tool developed by VMware, to speed up the deployment of an HPC cluster in vSphere.

Some of the steps are referenced from VMware documentation on how to configure a VM to use SR-IOV devices and NVIDIA documentation on how to set up and configure the firmware and driver of Mellanox ConnectX adapter cards in a vSphere environment.

Finally, they present a performance study that uses five HPC applications across multiple vertical domains. They conclude that a virtual HPC cluster can perform nearly as well as a bare metal HPC cluster while offering all the advantages of virtualization with vSphere like increased IT agility, flexibility, scalability, and significant cost savings of hardware.

View over 100 unique, on-demand technical sessions and session recordings from VMware Explore.

In this document, VMware walks through the steps to enable InfiniBand SR-IOV on a dual-port Mellanox ConnectX-5 VPI adapter card in vSphere 7.x.

They build on the most current VMware and Mellanox documentation and cover the steps from BIOS, ESXi, and vCenter to the functionality test on the VM guest operating system using ibverbs, Intel cluster checker, and the Ohio State University (OSU) microbenchmark suite.

They also introduce how to use the vHPC toolkit, an open-source tool developed by VMware, to speed up the deployment of an HPC cluster in vSphere.

Finally, they present a performance study of five HPC applications across multiple vertical domains, all concerned with dynamic systems (including manufacturing, weather forecasting, and the life sciences).

They conclude that virtual HPC clusters with VMware vSphere perform nearly as well as bare-metal HPC clusters while offering all the advantages of virtualization with vSphere like increased IT agility, flexibility, scalability, and significant cost savings of hardware.