When Networking Doesn't Work

Source: Hacker News

The Journey

1. First attempts – Windows XP VM + TSO

• Ran Tyan’s old TSO (Tyan System Operator) software in a Windows XP VM (NAT networking).
• The software installed cleanly but could not discover any IPMI‑enabled servers.
• Manual configuration of the IPMI address also failed.

2. Native Windows tools

• Tried ipmiutil (https://ipmiutil.sourceforge.net/).
• Same result – the SMDC never responded.

3. Windows 10 test box

• Repeated the test on an old PC running Windows 10.
• ipmiutil still refused to talk to the SMDC.

4. Switch to Linux

• Rebooted the same hardware into Linux and ran ipmitool.

• Success! ipmitool could communicate with the SMDC.

• Set up a Windows XP VM on the Linux host – the TSO software now worked fine.

5. Firewalls?

• Disabled Windows firewalls on all Windows hosts – no change.

6. Wireshark sniffing on Windows 11

• Confirmed outbound UDP traffic to port 623 and incoming replies from the SMDC.
• The replies were captured, but no application saw them.

7. PktMon investigation

• Ran PktMon to see how traffic traversed the Windows networking stack.
• Log entry of interest:

DropReason INET: checksum is invalid

• The packet reached the NIC, passed a few filter layers, then the TCP/IP stack dropped it because it thought the UDP checksum was bad.
• Wireshark, however, reported the IP and UDP checksums as valid.

8. Driver tweaking

• NIC: Intel I211 (driver e1r68x64.sys).
• Disabled UDP receive checksum offloading for IPv4.

Result: ipmiutil started working, and VMs on the host could talk to the SMDC.

• The same fix (disable IPv4 UDP Rx checksum offload) solved the problem on a Windows 10 machine with an Intel 82579LM (e1i65x64.sys).
• A Windows 10 laptop with a Killer Wireless 1525 adapter (no offloading options) was never affected.

What Is Going On Here?

• UDP Rx checksum offloading is generally reliable; otherwise DHCP, DNS, etc., would break.
• The SMDC’s UDP packets look perfectly normal (≈ 76 bytes, no VLAN tags, no tunneling).
• Linux (even on the same Intel NIC) accepts them.
• When the Windows TCP/IP stack must validate the checksum itself, it also says they’re fine.

How Rx checksum offloading works

1. Hardware validates the checksum of incoming packets and sets a valid/invalid flag in the receive descriptor.
2. The driver reports that flag to the higher‑level software.
3. The TCP/IP stack trusts the flag and does not recompute the checksum.

Therefore, the only plausible explanation is that the Intel NIC or its driver incorrectly reports the UDP checksum as invalid, causing the stack to drop otherwise valid packets.

Am I the Only One?

• Diagnosing this issue took a lot of time.
• Initial suspicion fell on the SMDC (old IPMI implementations can be flaky), but Wireshark proved the remote side was sending proper replies.
• PktMon was the key: it exposed the “checksum is invalid” drop reason, prompting the driver‑offload tweak.

What I found online

• Only vague references to checksum offloading causing mysterious problems.
• Some recommendations suggest disabling offloading entirely, arguing that the debugging cost outweighs any performance gain.

TL;DR

• Problem: Windows 10/11 machines with Intel NICs dropped valid IPMI UDP replies because the NIC driver reported an incorrect UDP Rx checksum status.
• Fix: Disable IPv4 UDP receive checksum offload in the NIC driver (e.g., via Device Manager → Advanced → UDP Checksum Offload → Disabled).
• Result: IPMI tools (ipmiutil, TSO, VMs) work correctly with the Tyan SMDC.

If you run into similar “no response” issues with legacy IPMI devices on Windows, try toggling the NIC’s checksum offload settings first.

Possible Cause

Without disassembling and tracing the Intel Windows drivers (something I don’t feel like doing), I can only speculate about the cause. I suspect it has something to do with the “Packet ID” field in the IP header. I even learned that there is a 2013 RFC 6864 which redefines how the ID field should be used and interpreted—apparently drafted after someone realized that it was being used in a manner that wasn’t compliant with the older RFCs (in short, the ID was getting recycled far too quickly).

The TCP/IP stack on the Tyan SMDC is very simplistic and seems to use the incoming packet as a template for the outgoing one. As a result, the incoming and outgoing packets (from the SMDC’s point of view) have the same ID. However, the outgoing packet is not marked as Do Not Fragment, which could potentially cause trouble. In practice it’s effectively guaranteed not to get fragmented, but that may be what’s confusing the UDP checksum offloading in the Intel Windows drivers, which is sensitive to fragmentation.

Since I can’t fix the Windows drivers anyway, turning off IPv4 UDP Rx checksum offloading is a perfectly workable solution for me. Except I really shouldn’t need to do that…