Jumbo frames in virtual FreeNAS 8.3.1 on ESXi 5.X

... and how to setup vmxnet3 vNIC and Jumbo frames

Problem:

When I set jumbo frames (MTU=9000) on em0 vNIC in virtual FreeNAS I can't get frames bigger than 1500 between the peers, even configuring mtu=9000 in all network elements between end points. Do not be fooled if you see this:

[root@freenas] ~# ifconfig em0 mtu 9000
[root@freenas] ~# ifconfig em0 em0: flags=8843 metric 0 mtu 9000 options=9b
ether 00:50:56:87:xx:xx options=9b ether 00:50:56:87:xx:xx
.....

I did some unsuccessful tests with em0 (in my case vnic is recognized as intel 82545EM) and MTU=9K. Eventually I could not get real jumbo frames between FreeNAS and other vm even inside vSphere host. Probably exception is being in the E1000 driver used by FreeNAS virtual machine though this also depends of physical NICs. em(4) supports variations of the hardware - some do jumbo frames, some don't. It could easily get a lot quirks in a VM, depending on how the VM emulates the card and the mode of networking used. This should show which of the Intel cards is being emulated.

# pciconf -lv | grep -A3 em0


Resolution:
FreeNAS 8.3.1 comes with preinstalled 3rd-party/independent vmtools. Unfortunately VMXNET3 isn’t included. If you want to make the most of your virtual network adapter, there is a way to set up vmxnet3 in your freenas.

- mount the vmware tools CD image to the vm freenas




Go to shell and:

- Create mount point for CD

# mkdir /mnt/cd
# mount -t cd9660 /dev/cd0 /mnt/cd
# ls /mnt/cd
./ manifest.txt ../ vmware-freebsd-tools.tar.gz

- Mount root rw (read/write)

# mount -urw /

- Copy file vmware-freebsd-tools.tar.gz to /tmp , change file permission and gunzip

# cd /mnt/cd
#cp vmware-freebsd-tools.tar.gz /tmp
#chmod 777 /tmp/vmware-freebsd-tools.tar.gz
# gunzip vmware-freebsd-tools.tar.gz/tmp
# tar -xf vmware-freebsd-tools.tar

Look for vmxnet3.ko file in FreeBSD8.1 subdirectory (path below) and copy to /boot/modules

# ls /tmp/vmware-tools-distrib/lib/modules/binary/FreeBSD8.1-amd64
./ ../ vmblock.ko vmmemctl.ko vmxnet.ko vmxnet3.ko

- Add vmxnet3 driver in loader.conf

For loading additional kernel modules (such as drivers) - GUI Tunable is preferred method because your changes will persist across upgrades.
GUI - System - Tunables - Add tunable

... or by manually edit /boot/loader.conf (but you risk losing your config if you upgrade)

# nano /boot/loader.conf

GNU nano 2.2.6 File: /boot/loader.conf
#
# Boot loader file for FreeNAS. This relies on a hacked beastie.4th.
#
autoboot_delay="2"
loader_logo="freenas"
#Fix booting from USB device bug
kern.cam.boot_delay="30000"
#VMXNET3 driver
vmxnet3_load="YES"


Save the corrections by Ctrl+O
Exit nano - Ctrl+X


Shutdown FreeNAS and goto vSphere Client - Edit settings - and add VMXNET3 adapter (I’ve added 2 more)



Start FreeNAS and login - Network Interfaces - Add Interface




Now you can utilize VMXNET3 adapters from drop down menu and well functioning 9K jumbo frames!