FreeBSD VLAN issues

Technology
1

So I’m using vm-bhyve to manage my VMs on FreeBSD 12.2, and trying to get a VM with a VLAN to receive the DHCP response to get an IP. The host is plugged into a trunk switch port, and the virtual switches (bridges) are these:

NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     igb0
iot     standard  vm-iot     -        no       -    30    igb0

The VM (named home-assistant, hence HA for short) is configured to use both the public and iot VLANs. The MAC for the iot interface is 58:9c:fc:5f:71:50.

The 2 switch/bridge interfaces are configured as such:

vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether e6:a0:04:c4:8d:9c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
vm-iot: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2a:bb:d5:7e:58:6e
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: igb0.30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 55
        groups: bridge vm-switch viid-97c20@
        nd6 options=1<PERFORMNUD>

With the hardware interfaces being these 2, the 2nd being the VLAN interface.

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether d0:50:99:dd:49:c9
        inet 192.168.2.5 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-iot-igb0.30
        options=200001<RXCSUM,RXCSUM_IPV6>
        ether d0:50:99:dd:49:c9
        inet6 fe80::d250:99ff:fedd:49c9%igb0.30 prefixlen 64 scopeid 0x7
        groups: vlan vm-vlan viid-45833@
        vlan: 30 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

When I boot it up, HomeAssistant (Linux base) it seems both interfaces, ends up getting an interface from the interface with no VLAN (so VLAN 0), but doesn’t get an IP for the VLAN interface.

Using TCPDump on the hardware interface tcpdump -c 1000 -i igb0 -e "ether host 58:9c:fc:5f:71:50" I see the DHCP request, and the DHCP response from the router

58:9c:fc:5f:71:50 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 343: vlan 30, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 58:9c:fc:5f:71:50 (oui Unknown), length 297
40:62:31:12:6b:95 (oui Unknown) > 58:9c:fc:5f:71:50 (oui Unknown), ethertype 802.1Q (0x8100), length 346: vlan 30, p 0, ethertype IPv4, darkrouter.home.darkobjects.net.bootps > homeassistant.home.darkobjects.net.bootpc: BOOTP/DHCP, Reply, length 300

So far so good, looks like it is, indeed, tagged VLAN 30 as I’d expect. When I run it on the hardware VLAN interface (tcpdump -c 1000 -i igb0.30 -e "ether host 58:9c:fc:5f:71:50"), I only see the outbound DHCP request, but never the DHCP response.

Anyone have any thoughts? Anything I should try? Can’t figure out what’s happening, why the DHCP Response isn’t at least making it to igb0.30, seems to be stopped at igb0. Unless I’m doing something wrong with configuring igb0 and it ends up stripping/blocking the VLAN tags and preventing igb0.30 from picking it up.

2

Why does igb0 not have an ipv6 and igb0.30 have an autoconf one? Seems suspicious to me.

How can the DHCP server distinguish between the two interfaces if they have the same MAC address? Wouldn’t the first eat it?

3

That’s not suspicious, I have IPv6 completely turned off on igb0, just don’t have it turned off as default. Was trying to troubleshoot/debug and just tried turning off IPv6 entirely, as much as possible.

And not sure that that’s an issue, as you see it’s responding to the MAC address of the client (VM), rather than MAC of the igb0/igb0.30 address.

4

Are there firewalls anywhere? A packet disappears and I suspect something is dropping it - and DHCP isn’t tcp or udp so a rule may need to be added.

Or NAT but it looks like the interface is on the bridge directly - if you remove the vlan tagging does it work

5

Nope, no firewalls active at all. None of the 3 from the FreeBSD Handbook are active, triple checked each to ensure that it was turned off.

And vm-bhyve won’t create a switch with the igb0 for some reason, BRGADD says it’s already busy. Might be because it’s already in the public v-switch. So no, can’t do it without the VLAN, exactly.

# vm switch create -i igb0 iot
/usr/local/sbin/vm: ERROR: failed to add member igb0 to the virtual switch iot

# ifconfig  vm-iot addm igb0
ifconfig: BRDGADD igb0: Device busy