Eric Sloof

This article describes what the consequences are when configuring an NSX-T Tier 1 Gateway with an Edge Cluster. In my lab environment, I have attached virtual machine sa-web-02 to the Web-Segment, this segment is attached to Gateway-A. The sa-app-01 VM is attached to the App-Segment and this segment is attached to Gateway-B. Both gateways are attached to the BGP-T0-GW-01, a tier 0 gateway.

In the Network Topology overview, there’s also a service active on both gateways. This is the gateway firewall, but since no edges are configured on the Tier 1 gateways. The firewall isn’t active. There is no edge available to host the SRs of these gateway firewalls.

Let’s take a look at the Traceflow troubleshooting tool, available in the Manager user interface.

The packets flow east west through the DR of Gateway-A, the DR of the BGP-T0-GW-01 tier 0 gateway and the DR of Gateway-B. A DR is a distributed router hosted in the kernel of an ESXi host. This all happens on sa-esxi-05 because routing is performed on the host where the packet originates. The source and destination virtual machines are hosted on different transport nodes, that’s why the packets travel through the overlay network between TEPs 172.20.11.152 and 172.20.11.151 after all the routing has been completed. 

An edge cluster is needed for hosting service routers for the Tier 1 gateways. Services like NAT, gateway firewall, and DHCP are not distributed. When you want to offer these services to a Tier 1 gateway, you need to configure an edge cluster. Let’s configure the gateways with an edge cluster and see what happens. 

After performing a Re-Trace with the Traceflow utility, we can clearly see that the logical path is the same, but the physical path now contains two edges. This is because both gateways are automatically configured with a north-south firewall. The SR component of these firewalls is hosted on the edges, so the traffic must travel through these edges for the egress and ingress gateway firewall.

Both tier 1 gateways are assigned to a different edge instance in the cluster, so traffic jumps from sa-nsxedge-02 to sa-nsxedge-01. Every time traffic jumps from a transport node to another transport node, the traffic needs to be encapsulated and decapsulated to enter and leave the overlay network. The default firewall rule on a gateway is any-any-allow.

If you want to avoid hair pinning, don’t configure an edge cluster on a Tier 1 gateway. The gateway firewall is always enabled. When you want to offer NAT to segment attached to the Tier 1 gateway, and you must configure an edge cluster, the firewall can be bypassed. 

The sequential steps in Traceflow show traffic entering sa-nsx-edge-01 for NAT only, the firewall is not in the data path.

In this technical white paper, VMware walks through the steps to enable RoCE SR-IOV on a dual-port Mellanox ConnectX-5 VPI adapter card in VMware vSphere 7.x.

They cover the steps from the BIOS, ESXi, and the vSphere Client to the functionality test on the VM guest operating system. They also introduce how to use the vHPC toolkit, an open-source tool developed by VMware, to speed up the deployment of an HPC cluster in vSphere.

Some of the steps are referenced from VMware documentation on how to configure a VM to use SR-IOV devices and NVIDIA documentation on how to set up and configure the firmware and driver of Mellanox ConnectX adapter cards in a vSphere environment.

Finally, they present a performance study that uses five HPC applications across multiple vertical domains. They conclude that a virtual HPC cluster can perform nearly as well as a bare metal HPC cluster while offering all the advantages of virtualization with vSphere like increased IT agility, flexibility, scalability, and significant cost savings of hardware.

View over 100 unique, on-demand technical sessions and session recordings from VMware Explore.

In this document, VMware walks through the steps to enable InfiniBand SR-IOV on a dual-port Mellanox ConnectX-5 VPI adapter card in vSphere 7.x.

They build on the most current VMware and Mellanox documentation and cover the steps from BIOS, ESXi, and vCenter to the functionality test on the VM guest operating system using ibverbs, Intel cluster checker, and the Ohio State University (OSU) microbenchmark suite.

They also introduce how to use the vHPC toolkit, an open-source tool developed by VMware, to speed up the deployment of an HPC cluster in vSphere.

Finally, they present a performance study of five HPC applications across multiple vertical domains, all concerned with dynamic systems (including manufacturing, weather forecasting, and the life sciences).

They conclude that virtual HPC clusters with VMware vSphere perform nearly as well as bare-metal HPC clusters while offering all the advantages of virtualization with vSphere like increased IT agility, flexibility, scalability, and significant cost savings of hardware.

Whether you are a creator embarking on a new path of discovery, or a decision-maker aiming for the outer reaches, your journey starts here. VMware CEO Raghu Raghuram kicks off VMware Explore’s general session with a behind-the-scenes look at how multi-cloud is enabling and enhancing our always-on, digital world. He’ll shine a spotlight on the ecosystem’s role driving transformation from servers to storage to cloud to apps. Everybody’s talking about the increasing value of multi-cloud, but what separates the leaders from the laggards?

Get inspired to conquer cloud complexity and transcend chaos by moving from a cloud-first to a cloud-smart approach. Next, VMware President Sumit Dhawan joins CVS Health CIO Roshan Navagamuwa onstage to discuss the U.S. healthcare leader’s customer-centric approach as it puts multi-cloud into action. You’ll hear fresh insights from S&P Global on why multi-cloud matters more than ever for businesses in a dynamic, unpredictable world.

Then, VMware CTO Kit Colbert and CTO Amanda Blevins will help you align with the right set of technology priorities to gain a competitive advantage. We’ll double-click on segments covering cloud-native-platform, cloud and edge transformation, and the hybrid workforce with VMware experts DaShaun Carter, Dave Morera and Teresa Chen. What does it take to power our digital lives safely and seamlessly?

McLaren Racing’s Head of Commercial Technology Edward Green shares how racing teams in the pit and hospitality suite guests are gaining the same secure, frictionless access to apps. Widen your perspective on Kubernetes with best-selling author and DevOps guru Kelsey Hightower, and the future of work with renowned expert Jacob Morgan.