Memory compute resources are required for all VMs to be able to execute the application workload. Memory management with VMware Cloud on an AWS SDDC is not very different from on-premises, with just a couple of best practices to follow:
- Do not overcommit host memory for your production application workload.
While it’s possible to assign more memory to VMs than a physical host has, it might seriously affect the performance of your application. Most modern applications are designed to use memory as a disk/application cache even if the real application payload is not consuming all the assigned memory. If swapping or ballooning occurs on the physical host, an application might experience up to a tenfold increase in latency compared to a non-overcommitted environment. This is not different on VMware Cloud on AWS, especially with vSAN and NSX consuming additional memory on the host. It’s recommended to leave at least 10% of the physical memory available for additional SDDC services.
- Consider sizing memory to fit into a single NUMA node.
Single NUMA VMs tend to perform better and utilize fewer resources than wide NUMA VMs. As the configuration of your host on-premises might be different, make sure to recheck the memory allocation after migration. You can use the following table for references:
*If your application workload requires significantly more memory, make sure to create a symmetrical CPU/memory configuration and expose vNUMA correctly.
| Instance Type | NUMA Configuration | Recommended Max Memory Setting per VM in GB* |
| i3.metal | 2 Nodes, 18 cores, 256 GB RAM | 256 |
| i3en.metal | 2 Nodes, 48 cores, 384 GB RAM | 384 |
| I4i.metal | 2 Nodes, 64 cores, 512 GB RAM | 512 |
Table 11.3 – VMware Cloud on AWS host physical NUMA memory configuration
The correct configuration of vNUMA settings for your workload may improve the performance of your applications and can help you to use the compute infrastructure more efficiently.
Storage
As discussed in Chapter 5, vSAN is the default storage type for your SDDC. vSAN storage provides a lot of benefits but also has a number of trade-offs:
- Dependency on network
- Host compute resource consumption
- Different queue limitations and maximum thresholds
These challenges and overall vSAN architecture define some of the best practices that might be different from the existing storage configuration in your on-premises estate. You should treat migration to vSAN as comparable to a migration to a new storage array (for example, a move from NetApp NFS storage to Pure storage with FC connectivity). Each storage vendor, vSAN included, has its own set of recommendations and best practices for your workload. Do not expect the same approach that worked well for an NFS datastore to work the same on vSAN.
We will review the most important configuration settings:
- I/O queues
When migrating to vSAN, your primary goal is to spread the I/O flow in as many streams as possible. To achieve this, we recommend implementing the following configuration changes:
- Split large (>1 TB), high-I/O VMDKs into smaller disks and use as many VMDKs as possible.
- Distribute VMDKs between the maximum number of virtual SCSI controllers (up to four per VM).
- Choose the best-performing storage policy aligning with SLAs. For workloads with a heavy write pattern (SQL Server transaction log, tempdb), use RAID1 whenever possible.
- Use the VMware Paravirtual SCSI (PVSCSI) controller type. This controller type has higher throughput with lower latency and CPU utilization and supports up to 254 queues per controller.