We do not recommend using the vNVMe controller due to known performance issues. In general, you achieve better throughput with lower CPU utilization using the PVSCI controller type.
- Object space reservation
It’s often recommended to set the advanced setting for Object Space Reservation (OSR) (https://core.vmware.com/blog/demystifying-capacity-reporting-vsan) to 100%. However, this setting does not influence the performance of your workload; it’s just a reservation. A VMDK with 100% OSR or 0% OSR is able to sustain the same amount of IOPS.
Figure 11.3 – Object space reservation setting
The difference between setting the OSR to 0% and 100% is with OSR set to 100%, your workloads are protected if a vSAN datastore becomes full, which might lead to critical conditions for your workload. On VMware Cloud on AWS, due to Elastic DRS and the elastic compute capacity, an SDDC can never reach over 80% vSAN datastore utilization. Using the default 0% setting of OSR (thin provisioning) will help you to lower the cost and still maintain the same performance level as with OSR set to 100%.
Note
For clusters with shared disks on VMware Cloud on AWS, you can use VMDKs with OSR set to 0%.
- I/O alignment
vSAN implementation on VMware Cloud on AWS uses physical NVMe drives with 4Kn physical drive formatting – 4,096 bytes per physical sector size. Due to the specifics of the VMDK implementation, a logical disk presented to a VM always has a physical sector size of 512 bytes. This discrepancy might cause severe (up to 300%) performance degradation caused by unaligned I/Os. This especially affects small I/O, between 512 and 4,096 bytes.
While an application can choose to use an I/O size as increments of 512, vSAN on the backend always operates with 4,096 bytes. For an I/O smaller than 4,096 bytes, a so-called read-modify-write (RMW) condition might arise, causing three I/Os in the backend for a single guest I/O. You can learn more at https://blogs.vmware.com/apps/2021/12/enhancing-performance-vmc-on-aws-sql-server-trace-flag-1800.html.
In most cases, proper alignment can be achieved by selecting an NTF block size to be 4,096 bytes or an increment of 4,096 bytes when formatting the logical disk within the Guest OS. However, some applications (SQL Server, Oracle) featuring their own I/O subsystem may be affected and require additional in-application tuning to mitigate possible performance issues.
Networking
Networking inside SDDC is powered by NSX. For your workload, it does not mean a lot of changes in terms of performance; however, it might have a much broader influence on the traffic flow, security principles, and so on.
One of the key changes our organizations notice from the network perspective is the change to the maximum transmission unit (MTU). On VMware Cloud on AWS SDDC (https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws-networking-security/GUID-1B51A82F-1AB5-4D35-A170-1044A3A85913.html), all east-west traffic supports an MTU of up to 8,950 bytes, while north-south traffic depends on the communication channel chosen. When using DX, you can set MTU up to 8,900 bytes, and when using a VPN connection, the only option is 1,500 bytes.
We recommend retesting the MTU after migration to make sure packet fragmentation is not happening. Don’t set different MTU sizes within a network!
Note
Due to the specifics of the overlay network, the “classic” jumbo frame size of 9,100 bytes is not supported.