vMotion Failing at 21% with error ""The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct."

vMotion Failing at 21% with error ""The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct."

Scenario

We built a new ESXi 6.7 Cluster and we couldn't make vMotions work there

Troubleshooting steps

1) Make sure there is no IP address conflict.

2) SSH to ESXi and do a VMK Ping check

For Default TCP/IP Stack

vmkping 10.11.7.188

Or If you are using Multi NIC vMotion, you might want to specify which VMK interface to use
vmkping -I vmk<vmkinterfacenumber> <Destination VMK IP to Ping>

Eg -: vmkping -I vmk3 192.168.1.1

For vMotion Stack

If you try above vmkping command on a ESXi host with VMK interfaces on VMK stack, they will fail with an error 
Unknown interface 'vmk': Invalid argument
Because the command is looking for TCP/IP stack by default and this VMK wont be listed there. So you need to specify that.

vmkping -I vmk<vmkinterfacenumber> -S vmotion <Destination VMK IP to Ping>

Eg-: vmkping -I vmk3 -S vmotion 192.168.1.1

If  the Ping test fails, check 
a) vMotion port group settings
b) ETXi Host Configuration -> Networking -> VMKernal Adapters and make sure vmotion is not enabled on the current VMK and only on the correct one. 

3) Check vMotion network port status on ESXi hosts to see if it's listening
We can use netcat utility for this test, similar to telnet test we do in Windows.

nc -z <Destination VMK IP> 8000

In the first try it took me you to the next line immedetly, connection is not establicked.
If it connects, it will stay running like I have in the second try for a while before it stops the connection. 

 

Another way to know that is to check network port status. Below is the equivalent to netstat command.
So what you need to do is, while above command is runnig and still active, open a new ssh session to same ESXi and also to Destination ESXi and look at the listening ports

esxcli network ip connection list | grep -i 8000

If  the connection test fails, revisit ESXi firewall rules using web client