Eric Sloof

With the growing demand for scalable and high-performance messaging solutions, VMware Tanzu RabbitMQ 1.3 and Confluent Kafka have emerged as key players in event streaming and message queue technologies. A recent performance study on VMware Tanzu Kubernetes Grid (TKG) highlights key improvements in Tanzu RabbitMQ and provides insights into Apache Kafka’s deployment on Kubernetes. Below, we summarize the key takeaways.

Key Technologies Evaluated

The study focused on the following technologies running on VMware Tanzu Kubernetes Grid:

• Tanzu RabbitMQ 1.3 – An enterprise-grade message broker with improvements in Quorum Queues and streaming capabilities.

• Confluent Kafka – A widely used distributed event streaming platform with support for large-scale real-time data processing.

• Strimzi – A Kubernetes operator for deploying Kafka clusters with ease.

Performance Highlights

Tanzu RabbitMQ 1.3: Significant Throughput Gains

• 30% higher throughput compared to previous versions, especially in Quorum Queue scenarios.

• Increased efficiency in handling messages of different sizes:

• Small messages (~55,634 messages/sec)

• Large messages (~15,334 messages/sec)

• RabbitMQ Streams reached up to 2.47 million messages/sec in a multi-stream setup.

Apache Kafka: High Throughput with Tuning

• Kafka was tested using 3 brokers, 12 partitions, and different replication settings.

• Achieved peak throughput of 149MB/sec with single-producer configurations.

• Replication overhead was noticeable, but asynchronous replication provided a balance between performance and durability.

• End-to-end message latency was 3ms for 1KB messages.

Architecture & Deployment Insights

• Both solutions were deployed on a 4-node VMware Tanzu Kubernetes Grid cluster.

• Hardware Setup:

• 24 vCPUs per node

• 72GB RAM per node

• 2TB storage per node

• Kafka’s Strimzi Operator was leveraged for automation and scaling.

• Tanzu RabbitMQ’s warm standby replication ensured high availability and disaster recovery.

Conclusion: Which One to Choose?

• Tanzu RabbitMQ 1.3 is ideal for low-latency message delivery, flexible routing, and high availability.

• Apache Kafka is better suited for event-driven architectures and large-scale real-time data streaming.

For businesses leveraging VMware Tanzu Kubernetes Grid, both technologies offer powerful messaging and event streaming capabilities, each with distinct advantages depending on the workload.

In today’s fast-paced digital world, low-latency computing is critical for applications in financial services, media streaming, and real-time automation. VMware vSphere 8 introduces a suite of performance enhancements designed to optimize virtualized environments for latency-sensitive workloads. This guide summarizes key takeaways from Broadcom’s latest whitepaper on performance tuning for vSphere 8.

Understanding Latency Optimization in vSphere 8

VMware vSphere 8 provides a powerful platform for virtualized workloads, balancing high performance with efficient resource utilization. However, for latency-sensitive applications, additional tuning is required to minimize response times. The key tradeoff here is increased CPU utilization in exchange for reduced latency.

Baseline Recommendations

To maximize performance for low-latency applications, it’s essential to ensure your environment is running the latest hardware and software:

• Processor generations: Use the latest supported CPUs.
• BIOS and microcode: Ensure firmware is up to date.
• vSphere version: At least vSphere 8.0 U3 or 7.0 U3.
• Virtual Hardware Version: Use Virtual Hardware 21 or newer.
• VM Tools: Version 12.4.5 or newer.

Additionally, minimize the use of vSphere overlays like NSX and vSAN, as these services consume CPU cycles and may increase latency.

Host-Level Optimizations

Optimizing ESXi host configurations is crucial for reducing latency:

• BIOS Settings: Set power management to "High Performance," disable C-states and P-states, and enable Turbo Boost.
• Disable EVC: Enhanced vMotion Compatibility (EVC) can mask CPU instructions and increase latency.
• vMotion and DRS Scheduling: Avoid live migrations during peak performance periods, as they can momentarily disrupt workloads.
• Advanced Settings:
  • Disable action affinity
  • Enable SplitRX and SplitTX for improved network processing
  • Disable queue pairing to prevent network bottlenecks

Virtual Machine-Level Optimizations

Configuring VMs correctly can significantly impact latency:

• Rightsizing: Ensure VMs are not oversized beyond NUMA node limits.
• Disable Hot-Add: This prevents NUMA topology from being exposed, reducing performance.
• Set High Latency Sensitivity:
  • Reserve CPU and memory exclusively for latency-sensitive VMs.
  • Configure VMXNET3 network adapters for optimal performance.
• NUMA Awareness: Assign VMs to NUMA nodes manually to align with the underlying hardware.
• Use SR-IOV or DirectPath I/O: These features allow direct hardware access for ultra-low latency networking.

Networking Enhancements

For network-intensive workloads, consider:

• VMXNET3: This paravirtualized NIC offers optimized performance.
• Enhanced Datapath Mode: For NFV workloads requiring high throughput.
• SmartNICs and DPUs: Offload network processing to dedicated hardware.
• Queue Balancing: Adjust NIC ring buffer sizes and enable multiple network queues.

Guest OS and Application Tuning

Within the guest operating system, use:

• Photon OS Real-Time Kernel: Optimized for low-latency applications.
• NUMA-aware applications: Configure workloads to take advantage of NUMA topology.
• Thread Affinity: Assign VM workloads to specific CPU cores to prevent contention.

Operational Best Practices

• Use esxtop and net-stats to monitor performance.
• Adjust ring buffers and queue sizes using esxcli.
• Test all changes in a non-production environment before deployment.

Conclusion

By following these best practices, VMware vSphere 8 can be fine-tuned to deliver low-latency performance for critical workloads. Whether optimizing host settings, configuring VMs, or leveraging SmartNICs, these strategies help ensure a responsive and efficient virtualized infrastructure.

For a more in-depth guide, refer to Broadcom’s whitepaper on performance tuning in vSphere 8.

In today’s ever-changing business landscape, disasters such as natural calamities and cyberattacks can disrupt operations without warning. VMware Cloud Disaster Recovery (DR) offers a powerful solution for ensuring business continuity by seamlessly recovering workloads in a cloud-based environment. This field guide provides insights into best practices for configuring and managing networking for disaster recovery (DR) using VMware Cloud on AWS.

Introduction to VMware Cloud DR

The VMware Cloud Disaster Recovery solution is designed to protect critical business applications from outages. This involves replicating workloads from a Protected Site to a Recovery Site hosted in VMware Cloud on AWS. The solution addresses various disaster scenarios, including complete site failures and targeted ransomware attacks. Effective network configuration is crucial to ensure that DR operations run smoothly, both during routine protection and emergency recovery.

Key Networking Considerations

1. Basic Networking Topology

The Protected Site (e.g., on-premises or another cloud) connects to the Recovery Site through VMware’s DRaaS Connector. This connection facilitates data replication to the Scale-out Cloud File System (SCFS), where recovery snapshots are stored. Components such as vCenter, vSphere hosts, and virtual machines rely on a robust network configuration to ensure connectivity.

2. Networking Setup for VMware Cloud on AWS

VMware Cloud DR leverages Software-Defined Data Centers (SDDCs) in AWS to manage cloud-based resources. Organizations can establish secure connectivity using one of the following methods:

• Public Internet Access with SDDC firewall rules (simpler but less secure)
• VPN Connection for encrypted data transfer
• AWS Direct Connect for a dedicated, high-performance connection

Review these options with your network team to determine the best fit for your environment.

3. Segmenting Networks for Disaster Recovery

Each network segment at the Protected Site requires a corresponding segment at the Recovery Site. This allows for seamless failover of virtual machines without disrupting production workloads. Additional segments can be created specifically for testing DR plans without impacting operational services.

Managing Connectivity and Security

Outbound and Inbound Access

To enable secure communication, network administrators configure gateway firewalls using NSX capabilities. Outbound access allows VMs at the Recovery Site to connect to the internet, while inbound access is managed through published VPNs or public IPs for critical workloads.

Inter-Site Connectivity

During partial failovers, applications may require communication between Protected and Recovery sites. VMware recommends using layer 2 VPNs or Direct Connect to facilitate secure inter-site application connectivity.

Ransomware Recovery Isolation

VMware Cloud DR includes features to isolate compromised networks in a Ransomware Recovery Isolated Recovery Environment (IRE). Using NSX Distributed Firewall rules, organizations can ensure that affected VMs remain isolated during recovery operations, preventing further spread of malware.

Testing and IP Remapping

Non-Disruptive Testing

VMware Cloud DR allows for comprehensive testing of recovery plans by using isolated network segments. This ensures that network configurations and failover procedures can be validated without affecting live applications.

IP Address Mapping

For scenarios where static IPs need to change during failover, VMware Cloud DR provides IP remapping capabilities. Administrators can define IP address rules in the recovery plan to ensure consistency across sites.

Conclusion

VMware Cloud Disaster Recovery is a robust solution for safeguarding business operations in the face of unexpected disruptions. Properly configuring and testing the networking components—ranging from connectivity and firewall rules to network segmentation and isolation—is critical to the success of any DR strategy. VMware offers extensive resources to guide organizations through setup and optimization, ensuring maximum uptime and data protection.

NFS is a reliable and viable storage option for virtualization. VMware supports both NFS version 3 and NFS version 4.1, providing features similar to those available with block-based storage like SAN. With proper configuration, vSphere on NFS delivers strong performance, stability, and a rich feature set.

Key Considerations for Deployment

   1. Networking Configuration

Efficient networking is essential for NFS performance. VMware recommends isolating storage traffic using dedicated switches or VLANs. The minimum NIC speed should be 1GbE, with 10GbE preferred for better throughput. Avoid network congestion by ensuring the LAN connection to the storage array is not over-subscribed.

   2. Throughput and Latency Optimization

NFS throughput can be enhanced using several methods:

• Jumbo Frames: Increases frame payload size for higher efficiency but requires consistent support across all devices.
• Load Sharing: Configure multiple datastores using separate IP connections to distribute network traffic.
• Link Aggregation: Multiple physical interfaces can be combined to provide redundancy and improved performance.

To reduce latency, minimize the number of network hops between ESXi hosts and storage arrays.

Security Best Practices

Security is crucial when deploying NFS:

• NFS v3 relies on root access and lacks encryption, making it essential to isolate NFS traffic on trusted networks.
• NFS v4.1 introduces support for Kerberos authentication and encryption, providing improved security with non-root access and data integrity checks.

High Availability

To ensure high availability:

• Use NIC teaming and redundant physical switches to eliminate single points of failure.
• Implement Link Aggregation Control Protocol (LACP) for improved failover and path redundancy.

Advanced Features

VMware integrates NFS with key vSphere features such as:

• Storage I/O Control (SIOC): Prevents resource contention by dynamically adjusting I/O priorities based on latency.
• Network I/O Control (NIOC): Prioritizes critical network traffic to prevent bottlenecks on shared NICs.
• Storage DRS: Balances workloads across datastores, optimizing both capacity and I/O performance.
• VAAI-NAS: Offloads storage operations to the array, improving efficiency for tasks like cloning and provisioning.

Additional Attributes of NFS Storage

NFS provides several storage advantages:

• Thin Provisioning: Efficiently allocates storage by using only the space that VMs actually consume.
• Deduplication: Some NAS arrays support deduplication, which can significantly reduce storage requirements.
• Backup and Restore: NFS offers granular backup and restore options, including the ability to restore individual files or entire datastores from snapshots.

Sizing and Configuration Tips

• There is no strict performance limitation on NFS volume size, but most datastores are typically in the tens of terabytes range.
• The recommended number of VMs per NFS datastore depends on workload intensity and backup/recovery SLAs.
• Advanced parameters such as TCP heap size, heartbeat intervals, and locking mechanisms can be tuned based on vendor recommendations.

Conclusion

NFS has evolved to become a robust storage solution for VMware vSphere environments. By following these best practices, organizations can maximize the performance, security, and availability of their virtual infrastructure. Collaboration with storage vendors ensures optimal configurations tailored to each deployment.

This guide offers a practical overview of how to leverage NFS storage effectively with VMware vSphere, ensuring reliable and high-performing virtualization.

VMware continuously enhances its vSphere platform to address scalability and reliability challenges in modern IT infrastructures. One key innovation is the vSphere Clustering Service (vCLS), introduced in vSphere 7 Update 1. Let’s explore how this service improves cluster operations and availability.

What is vCLS?

vCLS provides a distributed and decoupled control plane for clustering services, improving the reliability of features like the Distributed Resource Scheduler (DRS). Traditionally, clustering services depended on the vCenter Server’s availability. This dependency posed limitations on scalability and resilience. With vCLS, these challenges are mitigated, as it allows core clustering services to function even if the vCenter Server experiences downtime.

Architecture and Agent VMs

The vCLS architecture relies on lightweight agent VMs that are deployed automatically within each vSphere cluster. Key points include:

• Up to 3 agent VMs per cluster are created, forming a quorum to maintain service availability.
• In smaller clusters with fewer than 3 hosts, the number of agent VMs matches the number of ESXi hosts.
• These agent VMs are managed by vSphere and require no manual intervention from administrators.

Each vCLS agent VM runs a minimal configuration with a Photon OS, utilizing minimal resources:

• Memory: 128 MB (100 MB reserved)
• vCPU: 1 (100 MHz reserved)
• Disk: 2 GB (thin provisioned)
• No network adapter

Cluster Service Health

vCLS monitors and self-manages the availability of its agent VMs, with three health states:

1. Healthy: At least one agent VM is active and functional.
2. Degraded: One or more agent VMs are temporarily unavailable, though DRS operations continue.
3. Unhealthy: DRS operations are interrupted due to a lack of available agent VMs.

If any agent VM becomes unavailable, vCLS automatically re-deploys or powers it on to maintain service integrity.

Operational Guidelines

Administrators do not need to maintain or interact with the agent VMs directly. The following practices help ensure smooth operations:

• Avoid deleting, renaming, or powering off agent VMs.
• When placing a host in maintenance mode, the agent VMs are automatically migrated to another host.
• Agent VMs are visible in the vSphere Client under the vCLS folder, but not in the Hosts and Clusters view.

Automation scripts should be configured to ignore vCLS agent VMs to prevent accidental disruptions. These VMs can be identified using specific properties such as ManagedByInfo.

Why vCLS Matters

vCLS significantly enhances the resilience and scalability of VMware clusters by enabling essential services to function independently of vCenter Server. This innovation reduces service downtime risks and optimizes operations in both on-premises and cloud environments.

By leveraging vCLS, organizations can achieve a higher level of automation, stability, and performance in their virtual infrastructure. VMware’s forward-thinking approach ensures that clusters remain robust and efficient, even under challenging conditions.

In today’s fast-paced IT environments, ensuring the availability of data and applications is critical. VMware’s vSAN technology offers an innovative approach to distributed storage, delivering resilience and scalability. Let’s explore how vSAN Availability Technologies in version 8 U3 support both enterprise needs and disaster recovery.

What is vSAN?

VMware vSAN is a software-defined storage solution that aggregates storage resources from hosts in a cluster to create a single datastore. By embedding storage within the hypervisor, vSAN eliminates the need for traditional storage arrays. This architecture enhances scalability and resilience while simplifying storage management.

There are two main vSAN architectures:

• Original Storage Architecture (OSA): Uses a tiered system with cache and capacity devices.

• Express Storage Architecture (ESA): Introduced in 2022, ESA improves performance, resilience, and efficiency by eliminating the need for separate cache devices.

Resilience Through Storage Policies

vSAN’s storage policies define resilience settings through parameters like “Failures to Tolerate” (FTT). For instance, an FTT=2 policy with RAID-6 can survive two host failures without data loss. The system automatically adjusts data placement to maintain compliance with the policy, ensuring uninterrupted access even during failures.

Handling Failures and Maintenance

vSAN categorizes components of data objects into various states (e.g., active, degraded, or absent) to determine data availability. The system uses a quorum-based approach to maintain integrity, requiring more than 50% of components to be accessible for data to remain available.

During planned maintenance, vSAN offers options such as “Full data migration” or “Ensure accessibility,” optimizing data movement and availability depending on the scenario.

Advanced Features in ESA

The Express Storage Architecture introduces several enhancements:

• Durability Components: Temporary data structures created during maintenance to ensure data consistency.

• Low-level Metadata Resilience: Improved handling of Unrecoverable Read Errors (UREs) to protect critical metadata.

• Network Redundancy and Adaptive Traffic Shaping: ESA optimizes network performance with features like RDMA support for high-speed environments.

Integration with Disaster Recovery

VMware’s disaster recovery strategies, such as asynchronous replication and integration with tools like VMware Live Site Recovery, complement vSAN’s high availability features. These solutions ensure business continuity even in the face of catastrophic events.

Maximizing Uptime

By utilizing vSAN’s built-in mechanisms for proactive hardware management, degraded device handling, and network partition detection, organizations can achieve exceptional uptime and resilience. Skyline Health tools provide real-time monitoring and diagnostics to quickly address issues.

Final Thoughts

VMware vSAN Availability Technologies offer a robust foundation for modern data centers. With adaptive storage policies, enhanced resilience, and integrated disaster recovery capabilities, vSAN supports scalable and high-performing virtualized environments.

Efficient resource management in large-scale VMware environments is critical, and tagging plays a key role in organizing virtual machines (VMs), hosts, and datastores. The latest VMware vCenter 8.0 U3 release introduces performance improvements and best practices for managing tags effectively.

1. Optimized API Calls:

   • The attach() call for associating tags to inventory items is 40% faster than the previous release.
   • attachTagToMultipleObjects() delivers a 200% speed improvement when associating a single tag to multiple VMs.
   • attachMultipleTagsToObject() provides a 31%–36% performance boost when applying multiple tags to a single VM.

2. API Choices for Tag Operations:

   • Attaching tags: For better efficiency, use attachMultipleTagsToObject() for single objects or attachTagToMultipleObjects() for bulk operations.
   • Querying tags: The listAttachedTagsOnObjects() and the pagination-based list() API are recommended to handle large-scale queries.

• Parallelism: Spread tags across multiple categories to enhance performance, especially when using multiple API clients simultaneously.
• Linked Mode Performance: Enhanced Linked Mode (ELM) allows replication of tag definitions across vCenters. In a ring topology, tag propagation latency is minimized to under 120-180 seconds in large environments with up to 15 linked vCenters.
• Scaling Limits: For optimal performance, vCenter supports up to 6,000 categories, 8,000 tags, and 150,000 tag associations per node.

While tags work well for categorical organization, scenarios requiring unique data per object (e.g., asset IDs or timestamps) are better suited to custom attributes. Balancing tag distribution and avoiding excessive associations can maintain performance stability in large deployments.

VMware vCenter 8.0 U3 tagging enhancements make it easier for organizations to manage vast virtual infrastructure with improved efficiency. By following best practices and leveraging optimized APIs, enterprises can ensure scalability and performance in dynamic environments.

The demand for AI is expanding beyond traditional GPU-based data centers to diverse environments, including edge and hybrid clouds. VMware and Intel have joined forces to bring AI capabilities to CPU-driven infrastructure, demonstrating how AI workloads can thrive without GPUs by leveraging Intel's 4th Gen Xeon Scalable Processors and VMware's Private AI infrastructure.

VMware Private AI integrates AI infrastructure with privacy, compliance, and security, built on VMware Cloud Foundation (VCF). Paired with Intel’s Advanced Matrix Extensions (AMX), it enables scalable and efficient AI operations without requiring specialized hardware like GPUs.

• Intel 4th Gen Xeon Scalable Processors with AMX:
  AMX accelerates both AI training and inference directly within the CPU, optimizing performance for workloads like large language models (LLMs).

• VMware Cloud Foundation:
  This software-defined platform virtualizes compute, storage, and networking, providing a unified management interface for containerized AI workloads using VMware Tanzu Kubernetes Grid (TKG).

In a real-world application, VMware and Intel tested the Llama 2-7B model on Intel's hardware, showing how AMX-enabled processors efficiently handle inference workloads. Key results include:

• Inference latency under 50ms for small batches with INT8 precision.
• Scalability for multiple instances per socket with sub-100ms latency, even at high token counts.

• Intel's AMX technology speeds up inference by up to 1.8x compared to BF16 models.
• VMware's integration with Kubernetes makes deploying AI models on existing infrastructure fast and seamless.

This solution is ideal for businesses looking to expand AI capabilities without investing heavily in GPU infrastructure. It offers:

• Cost Efficiency: Leverage existing CPU resources for AI tasks.
• Flexibility: Run AI workloads across cloud, edge, and on-premise environments.
• Scalability: Easily manage resource-intensive tasks like LLM inference with VMware's orchestration tools.

By combining VMware and Intel technologies, enterprises can unlock the full potential of AI with optimized infrastructure, reducing costs and simplifying deployment.

In today’s security-conscious world, data encryption is no longer optional—it's a critical requirement. VMware's vSAN Encryption Services provide robust security for data both at rest and in transit, ensuring compliance with organizational and regulatory standards. Here's a closer look at how VMware's latest offerings can protect your infrastructure.

• Data-at-Rest Encryption:
  Protects all data stored on vSAN clusters, encrypting it at the final stage of I/O processing.

• Data-in-Transit Encryption:
  Safeguards data transmitted across vSAN hosts without requiring a Key Management Server (KMS).

These services are enabled on a per-cluster basis, offering flexibility depending on your needs.

• Original Storage Architecture (OSA):
  Supports both encryption and space-saving features like deduplication and compression.

• Express Storage Architecture (ESA):
  Encrypts data efficiently at the upper layers of the storage stack, minimizing CPU and network overhead.

Organizations can manage encryption keys using either an External Key Management Server (KMS) or the vSphere Native Key Provider (NKP). For enhanced security, VMware recommends using Trusted Platform Modules (TPM) to store keys locally on hosts.

• Performance Impact:
  Encryption is optimized to minimize performance overhead by utilizing advanced AES-NI CPU instructions.

• Rekeying and Secure Wiping:
  Both shallow and deep rekey operations are supported, allowing administrators to rotate encryption keys without disrupting operations. Additionally, VMware offers secure device wiping options compliant with NIST standards.

• Enable both data-at-rest and data-in-transit encryption for comprehensive protection.
• Ensure proper DNS configurations for KMS to avoid connectivity issues.
• Implement TPM on all vSAN hosts to improve key persistence and recovery processes.

VMware's vSAN Encryption Services provide scalable and flexible security solutions, helping organizations meet their data protection goals with minimal complexity. Secure your infrastructure today with VMware vSAN!

As data volumes grow, efficient storage management becomes critical to controlling costs. VMware vSAN offers both opportunistic and deterministic space-saving technologies to help organizations optimize storage while maintaining performance.

• Opportunistic Techniques:
  These depend on data characteristics and conditions. Examples include:

  • Deduplication and Compression (DD&C) in the Original Storage Architecture (OSA).
  • Compression-only options for performance-sensitive workloads.
  • Thin provisioning, which allocates space as needed, and TRIM/UNMAP for reclaiming unused capacity.

• Deterministic Techniques:
  These ensure consistent savings through methods like erasure coding, which protects data using less capacity than traditional mirroring.

• Express Storage Architecture (ESA):
  ESA integrates compression at the top of the storage stack, reducing CPU and network usage while enhancing performance. Unlike OSA, ESA does not currently support deduplication but offers highly efficient erasure coding.

• Advanced RAID Options:
  ESA enables RAID-5 and RAID-6 erasure coding without performance penalties, making it suitable for both small and large clusters.

• For Performance-Driven Workloads:
  The compression-only feature in OSA is ideal. It minimizes latency and reduces the risk of storage device failure affecting the entire cluster.

• For Maximum Capacity Savings:
  Deduplication and compression, combined with RAID-5/6 erasure coding, offer substantial storage reduction in environments that can support the additional processing requirements.

• Align storage policies with workload needs.
• Use modern high-performance storage and networking hardware.
• Enable TRIM/UNMAP for automated space reclamation, particularly for thin-provisioned environments.

VMware vSAN continues to evolve, providing flexible, scalable, and efficient storage solutions tailored to meet both performance and capacity demands. By leveraging these features, organizations can better manage their storage infrastructure while reducing costs.