I am setting up a new Sophos UTM (Business Essentials), in addition to other licensed UTMs. This new one hangs off a licensed one as a VM guest. This is the second install of this kind I have done. The first one works fine. This one is having an issue.
The only difference between the two setups is that this new one has only a single bridged interface for the guest, the intention is to VLAN interfaces within Sophos.
I configured a subnet behind the UTM, and response packets from external hosts with length less than 1000 route though okay. When the packet length is above 1500, the first few packets do not make it from the external interface to the subnet route. But the last packet, that is under length of 1000 makes it.
I performed a tcpdump on both the external interface and internal subnet interface, and have verified this is true. Packets make it to the external interface, but the ones with a length of 1500 do not route to the internal subnet interface.
I am not having this issue on the first UTM setup that is similar to this topology. The configuration is the same between them, except they are on different remote networks, and newest setup uses a single untagged (KVM) virtio network card, where as the prior uses multiple bridged VLAN tagged interfaces.
MTU is 1500 on all interfaces.
After watching the logs, I see this error appearing, and subsequent discussion on the Internet point to a similar issue as I am experiencing, and question a possible bug in the
kernel.
2014:09:04-13:41:35 net301ima kernel: [ 2375.772054] ------------[ cut here ]------------
2014:09:04-13:41:35 net301ima kernel: [ 2375.772067] WARNING: at net/core/dev.c:2033 skb_warn_bad_offload+0xb8/0xc0()
2014:09:04-13:41:35 net301ima kernel: [ 2375.772070] Hardware name: KVM
2014:09:04-13:41:35 net301ima kernel: [ 2375.772074] : caps=(0x0000000000005020, 0x0000000000000000) len=1500 data_len=1390 gso_size=1448 gso_type=5 ip_summed=1
2014:09:04-13:41:35 net301ima kernel: [ 2375.772076] Modules linked in: sd_mod xt_connmark xt_tcpudp xt_multiport xt_set xt_addrtype ip_set_hash_net ip_set_hash_ip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_ftp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_ftp af_packet 8021q ebtable_filter ebtables bridge stp llc redv2_netlink ip6table_ips ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_ips iptable_mangle iptable_nat nf_nat_ipv4 nf_nat xt_NFLOG xt_condition(O) xt_logmark xt_confirmed xt_owner ip6t_REJECT ipt_REJECT xt_state ip_set red2 ip_scheduler red nfnetlink_log mperf nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_raw nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter iptable_raw xt_CT nf_conntrack_netlink nfnetlink nf_conntrack ip6_tables ip_tables x_tables ipv6 loop sg button virtio_net rtc_cmos sr_mod cdrom pcspkr i2c_piix4 evdev virtio_balloon microcode uhci_hcd ehci_hcd virtio_blk processor thermal_sys hwmon pata_acpi ata_generic edd ata_piix libata scsi_mod virtio_pci virtio_ring virtio hid_generic usbhid
2014:09:04-13:41:35 net301ima kernel: [ 2375.772166] Pid: 0, comm: swapper/0 Tainted: G W O 3.8.13.15-110.g4be5643-smp #1
2014:09:04-13:41:35 net301ima kernel: [ 2375.772168] Call Trace:
2014:09:04-13:41:35 net301ima kernel: [ 2375.772174] [<c1209fde>] ? skb_warn_bad_offload+0xb8/0xc0
2014:09:04-13:41:35 net301ima kernel: [ 2375.772179] [<c102877a>] ? warn_slowpath_common+0x7b/0x90
2014:09:04-13:41:35 net301ima kernel: [ 2375.772183] [<c1209fde>] ? skb_warn_bad_offload+0xb8/0xc0
2014:09:04-13:41:35 net301ima kernel: [ 2375.772187] [<c1028817>] ? warn_slowpath_fmt+0x33/0x37
2014:09:04-13:41:35 net301ima kernel: [ 2375.772191] [<c1209fde>] ? skb_warn_bad_offload+0xb8/0xc0
2014:09:04-13:41:35 net301ima kernel: [ 2375.772195] [<c120ad2b>] ? skb_gso_segment+0x9b/0x1d9
2014:09:04-13:41:35 net301ima kernel: [ 2375.772199] [<c120b28b>] ? dev_hard_start_xmit+0x1d4/0x37f
2014:09:04-13:41:35 net301ima kernel: [ 2375.772203] [<c120b606>] ? dev_queue_xmit+0x1d0/0x263
2014:09:04-13:41:35 net301ima kernel: [ 2375.772209] [<c122fd64>] ? ip_finish_output2+0x27a/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772212] [<c122c3f3>] ? skb_dst+0x7/0x7
2014:09:04-13:41:35 net301ima kernel: [ 2375.772216] [<c122fdad>] ? ip_finish_output2+0x2c3/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772220] [<c122f06e>] ? NF_HOOK_COND+0x4f/0x56
2014:09:04-13:41:35 net301ima kernel: [ 2375.772224] [<c122fdad>] ? ip_finish_output2+0x2c3/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772227] [<c122f0f7>] ? ip_output+0x82/0x88
2014:09:04-13:41:35 net301ima kernel: [ 2375.772231] [<c122fdad>] ? ip_finish_output2+0x2c3/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772235] [<c120fd89>] ? dst_output+0x9/0xa
2014:09:04-13:41:35 net301ima kernel: [ 2375.772239] [<c122b39a>] ? ip_rcv_finish+0x27d/0x293
2014:09:04-13:41:35 net301ima kernel: [ 2375.772243] [<c122b11d>] ? ip_rcv+0x277/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772246] [<c122ae9f>] ? NF_HOOK+0x48/0x4f
2014:09:04-13:41:35 net301ima kernel: [ 2375.772263] [<c122b11d>] ? ip_rcv+0x277/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772272] [<c122b0e6>] ? ip_rcv+0x240/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772275] [<c122b11d>] ? ip_rcv+0x277/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772279] [<c1208d32>] ? __netif_receive_skb+0x424/0x475
2014:09:04-13:41:35 net301ima kernel: [ 2375.772283] [<c1202d5a>] ? build_skb+0x27/0xb5
2014:09:04-13:41:35 net301ima kernel: [ 2375.772287] [<c120a3b9>] ? netif_receive_skb+0x63/0x68
2014:09:04-13:41:35 net301ima kernel: [ 2375.772297] [<f83c0ff4>] ? virtnet_poll+0x47d/0x563 [virtio_net]
2014:09:04-13:41:35 net301ima kernel: [ 2375.772303] [<c120a854>] ? net_rx_action+0x91/0x1b1
2014:09:04-13:41:35 net301ima kernel: [ 2375.772308] [<c102e0f0>] ? __do_softirq+0x84/0x143
2014:09:04-13:41:35 net301ima kernel: [ 2375.772312] [<c102e06c>] ? irq_enter+0x4d/0x4d
2014:09:04-13:41:35 net301ima kernel: [ 2375.772314] <IRQ> [<c102df40>] ? irq_exit+0x2f/0x92
2014:09:04-13:41:35 net301ima kernel: [ 2375.772321] [<c100307f>] ? do_IRQ+0x81/0x95
2014:09:04-13:41:35 net301ima kernel: [ 2375.772324] [<c102dfa2>] ? irq_exit+0x91/0x92
2014:09:04-13:41:35 net301ima kernel: [ 2375.772329] [<c101afee>] ? smp_apic_timer_interrupt+0x6f/0x7b
2014:09:04-13:41:35 net301ima kernel: [ 2375.772334] [<c127b80c>] ? common_interrupt+0x2c/0x31
2014:09:04-13:41:35 net301ima kernel: [ 2375.772339] [<c10205e9>] ? native_safe_halt+0x2/0x3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772343] [<c1007773>] ? default_idle+0x1c/0x31
2014:09:04-13:41:35 net301ima kernel: [ 2375.772346] [<c100797a>] ? cpu_idle+0x52/0x71
2014:09:04-13:41:35 net301ima kernel: [ 2375.772350] [<c1378a63>] ? start_kernel+0x31d/0x322
2014:09:04-13:41:35 net301ima kernel: [ 2375.772354] [<c137858b>] ? repair_env_string+0x4f/0x4f
2014:09:04-13:41:35 net301ima kernel: [ 2375.772357] ---[ end trace 49019babd7ff281d ]---
The only difference between the two setups is that this new one has only a single bridged interface for the guest, the intention is to VLAN interfaces within Sophos.
I configured a subnet behind the UTM, and response packets from external hosts with length less than 1000 route though okay. When the packet length is above 1500, the first few packets do not make it from the external interface to the subnet route. But the last packet, that is under length of 1000 makes it.
I performed a tcpdump on both the external interface and internal subnet interface, and have verified this is true. Packets make it to the external interface, but the ones with a length of 1500 do not route to the internal subnet interface.
I am not having this issue on the first UTM setup that is similar to this topology. The configuration is the same between them, except they are on different remote networks, and newest setup uses a single untagged (KVM) virtio network card, where as the prior uses multiple bridged VLAN tagged interfaces.
MTU is 1500 on all interfaces.
After watching the logs, I see this error appearing, and subsequent discussion on the Internet point to a similar issue as I am experiencing, and question a possible bug in the
kernel.
2014:09:04-13:41:35 net301ima kernel: [ 2375.772054] ------------[ cut here ]------------
2014:09:04-13:41:35 net301ima kernel: [ 2375.772067] WARNING: at net/core/dev.c:2033 skb_warn_bad_offload+0xb8/0xc0()
2014:09:04-13:41:35 net301ima kernel: [ 2375.772070] Hardware name: KVM
2014:09:04-13:41:35 net301ima kernel: [ 2375.772074] : caps=(0x0000000000005020, 0x0000000000000000) len=1500 data_len=1390 gso_size=1448 gso_type=5 ip_summed=1
2014:09:04-13:41:35 net301ima kernel: [ 2375.772076] Modules linked in: sd_mod xt_connmark xt_tcpudp xt_multiport xt_set xt_addrtype ip_set_hash_net ip_set_hash_ip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_ftp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_ftp af_packet 8021q ebtable_filter ebtables bridge stp llc redv2_netlink ip6table_ips ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_ips iptable_mangle iptable_nat nf_nat_ipv4 nf_nat xt_NFLOG xt_condition(O) xt_logmark xt_confirmed xt_owner ip6t_REJECT ipt_REJECT xt_state ip_set red2 ip_scheduler red nfnetlink_log mperf nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_raw nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack iptable_filter iptable_raw xt_CT nf_conntrack_netlink nfnetlink nf_conntrack ip6_tables ip_tables x_tables ipv6 loop sg button virtio_net rtc_cmos sr_mod cdrom pcspkr i2c_piix4 evdev virtio_balloon microcode uhci_hcd ehci_hcd virtio_blk processor thermal_sys hwmon pata_acpi ata_generic edd ata_piix libata scsi_mod virtio_pci virtio_ring virtio hid_generic usbhid
2014:09:04-13:41:35 net301ima kernel: [ 2375.772166] Pid: 0, comm: swapper/0 Tainted: G W O 3.8.13.15-110.g4be5643-smp #1
2014:09:04-13:41:35 net301ima kernel: [ 2375.772168] Call Trace:
2014:09:04-13:41:35 net301ima kernel: [ 2375.772174] [<c1209fde>] ? skb_warn_bad_offload+0xb8/0xc0
2014:09:04-13:41:35 net301ima kernel: [ 2375.772179] [<c102877a>] ? warn_slowpath_common+0x7b/0x90
2014:09:04-13:41:35 net301ima kernel: [ 2375.772183] [<c1209fde>] ? skb_warn_bad_offload+0xb8/0xc0
2014:09:04-13:41:35 net301ima kernel: [ 2375.772187] [<c1028817>] ? warn_slowpath_fmt+0x33/0x37
2014:09:04-13:41:35 net301ima kernel: [ 2375.772191] [<c1209fde>] ? skb_warn_bad_offload+0xb8/0xc0
2014:09:04-13:41:35 net301ima kernel: [ 2375.772195] [<c120ad2b>] ? skb_gso_segment+0x9b/0x1d9
2014:09:04-13:41:35 net301ima kernel: [ 2375.772199] [<c120b28b>] ? dev_hard_start_xmit+0x1d4/0x37f
2014:09:04-13:41:35 net301ima kernel: [ 2375.772203] [<c120b606>] ? dev_queue_xmit+0x1d0/0x263
2014:09:04-13:41:35 net301ima kernel: [ 2375.772209] [<c122fd64>] ? ip_finish_output2+0x27a/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772212] [<c122c3f3>] ? skb_dst+0x7/0x7
2014:09:04-13:41:35 net301ima kernel: [ 2375.772216] [<c122fdad>] ? ip_finish_output2+0x2c3/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772220] [<c122f06e>] ? NF_HOOK_COND+0x4f/0x56
2014:09:04-13:41:35 net301ima kernel: [ 2375.772224] [<c122fdad>] ? ip_finish_output2+0x2c3/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772227] [<c122f0f7>] ? ip_output+0x82/0x88
2014:09:04-13:41:35 net301ima kernel: [ 2375.772231] [<c122fdad>] ? ip_finish_output2+0x2c3/0x2c3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772235] [<c120fd89>] ? dst_output+0x9/0xa
2014:09:04-13:41:35 net301ima kernel: [ 2375.772239] [<c122b39a>] ? ip_rcv_finish+0x27d/0x293
2014:09:04-13:41:35 net301ima kernel: [ 2375.772243] [<c122b11d>] ? ip_rcv+0x277/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772246] [<c122ae9f>] ? NF_HOOK+0x48/0x4f
2014:09:04-13:41:35 net301ima kernel: [ 2375.772263] [<c122b11d>] ? ip_rcv+0x277/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772272] [<c122b0e6>] ? ip_rcv+0x240/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772275] [<c122b11d>] ? ip_rcv+0x277/0x277
2014:09:04-13:41:35 net301ima kernel: [ 2375.772279] [<c1208d32>] ? __netif_receive_skb+0x424/0x475
2014:09:04-13:41:35 net301ima kernel: [ 2375.772283] [<c1202d5a>] ? build_skb+0x27/0xb5
2014:09:04-13:41:35 net301ima kernel: [ 2375.772287] [<c120a3b9>] ? netif_receive_skb+0x63/0x68
2014:09:04-13:41:35 net301ima kernel: [ 2375.772297] [<f83c0ff4>] ? virtnet_poll+0x47d/0x563 [virtio_net]
2014:09:04-13:41:35 net301ima kernel: [ 2375.772303] [<c120a854>] ? net_rx_action+0x91/0x1b1
2014:09:04-13:41:35 net301ima kernel: [ 2375.772308] [<c102e0f0>] ? __do_softirq+0x84/0x143
2014:09:04-13:41:35 net301ima kernel: [ 2375.772312] [<c102e06c>] ? irq_enter+0x4d/0x4d
2014:09:04-13:41:35 net301ima kernel: [ 2375.772314] <IRQ> [<c102df40>] ? irq_exit+0x2f/0x92
2014:09:04-13:41:35 net301ima kernel: [ 2375.772321] [<c100307f>] ? do_IRQ+0x81/0x95
2014:09:04-13:41:35 net301ima kernel: [ 2375.772324] [<c102dfa2>] ? irq_exit+0x91/0x92
2014:09:04-13:41:35 net301ima kernel: [ 2375.772329] [<c101afee>] ? smp_apic_timer_interrupt+0x6f/0x7b
2014:09:04-13:41:35 net301ima kernel: [ 2375.772334] [<c127b80c>] ? common_interrupt+0x2c/0x31
2014:09:04-13:41:35 net301ima kernel: [ 2375.772339] [<c10205e9>] ? native_safe_halt+0x2/0x3
2014:09:04-13:41:35 net301ima kernel: [ 2375.772343] [<c1007773>] ? default_idle+0x1c/0x31
2014:09:04-13:41:35 net301ima kernel: [ 2375.772346] [<c100797a>] ? cpu_idle+0x52/0x71
2014:09:04-13:41:35 net301ima kernel: [ 2375.772350] [<c1378a63>] ? start_kernel+0x31d/0x322
2014:09:04-13:41:35 net301ima kernel: [ 2375.772354] [<c137858b>] ? repair_env_string+0x4f/0x4f
2014:09:04-13:41:35 net301ima kernel: [ 2375.772357] ---[ end trace 49019babd7ff281d ]---