8

I am relatively new to networking, so please bear with me. For a purely educational project, I've set up a network where the Raspberry Pi 3 running Raspbian Stretch is to function both as a router and a DHCP server. The intention is to have a private network that shares an internet connection. Here's the network topology:

 +----------+
 | Internet |
 +-----+----+
       | (wlan0)
 +-----+----------+
 | Raspberry Pi 3 | (router + DHCP server)
 +-----+----------+
       | (eth0)
 +-----+--+
 | Switch |  (TP-Link 5-Port Desktop Switch TL-SF1005D)
 +-----+--+
       |
 +-----+-------+
 | My Computer |
 +-------------+

The private network is to be 10.0.0.0/24, with the Raspberry Pi (which is the router and DHCP server) having the 10.0.0.1 address (through eth0).

Routing Configuration on Raspberry Pi

For Raspberry Pi to route packets between wlan0 and eth0, I did sudo sysctl -w net.ipv4.ip_forward=1 (and edited /etc/sysctl.conf to make the change permanent).

Next, on initially empty iptables tables, I did:

# iptables -P INPUT ACCEPT
# iptables -P OUTPUT ACCEPT
# iptables -P FORWARD DROP
# iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
# iptables -A FORWARD -i wlan0 -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT

Network Interface Configuration on Raspberry Pi

Because Raspbian Stretch uses /etc/dhcpcd.conf instead of /etc/network/interfaces to automatically configure network interfaces, I editied /etc/dhcpcd.conf to look like this (see the MY SETTINGS section):

# /etc/dhcpcd.conf on Raspberry Pi (router + DHCP server).

# Inform the DHCP server of our hostname for DDNS.
hostname

# Use the hardware address of the interface for the Client ID.
clientid

# Persist interface configuration when dhcpcd exits.
persistent

# Rapid commit support.
option rapid_commit

# A list of options to request from the DHCP server.
option domain_name_servers, domain_name, domain_search, host_name
option classless_static_routes
# Most distributions have NTP support.
option ntp_servers
# Respect the network MTU. This is applied to DHCP routes.
option interface_mtu

# A ServerID is required by RFC2131.
require dhcp_server_identifier

# Generate Stable Private IPv6 Addresses instead of hardware based ones
slaac private

# MY SETTINGS:
interface eth0
static ip_address=10.0.0.1/24
static broadcast_address=10.0.0.255
static domain_name_servers=8.8.8.8 8.8.4.4

DHCP Configuration on Raspberry Pi

I've installed isc-dhcp-server and configured it like this:

# /etc/dhcp/dhcpd.conf on Raspberry Pi (router + DHCP server).
# Configuration file for ISC dhcpd

# Use Google Public DNS.
option domain-name-servers 8.8.4.4, 8.8.8.8;

default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;

subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.10 10.0.0.254;
    option routers 10.0.0.1;
    option broadcast-address 10.0.0.255;
}

Configuration for serving eth0 only:

# /etc/default/isc-dhcp-server on Raspberry Pi (router + DHCP server).
INTERFACESv4="eth0"
INTERFACESv6=""

Then I rebooted the Raspberry Pi, and ensured that isc-dhcp-server is started.

An Observation of the Problem

Excited about the new network, I took a computer (My Computer in the network diagram above), and connected an ethernet cable to its ethernet port. The computer does not connect, so I take a look at /var/log/syslog on the computer. There, I see these lines:

home dhclient[14783]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 3 (xid=0xcab1222)
home dhclient[14783]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 5 (xid=0xcab1222)
home dhclient[14783]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 14 (xid=0xcab1222)
home dhclient[14783]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 16 (xid=0xcab1222)
## etc.

Curious about whether or not the Raspberry Pi (router + DHCP server) is receiving these requests, I looked at /var/log/syslog on the Raspberry Pi:

rpi dhcpd[9685]: DHCPDISCOVER from f4:8e:38:e9:88:d7 (home) via eth0
rpi dhcpd[9685]: DHCPOFFER on 10.0.0.11 to f4:8e:38:e9:88:d7 (home) via eth0
rpi dhcpd[9685]: DHCPDISCOVER from f4:8e:38:e9:88:d7 (home) via eth0
rpi dhcpd[9685]: DHCPOFFER on 10.0.0.11 to f4:8e:38:e9:88:d7 (home) via eth0
rpi dhcpd[9685]: DHCPDISCOVER from f4:8e:38:e9:88:d7 (home) via eth0
rpi dhcpd[9685]: DHCPOFFER on 10.0.0.11 to f4:8e:38:e9:88:d7 (home) via eth0
## etc.

Indeed the Raspberry Pi is sending DHCPOFFERs but the client computer is somehow not receiving them (or ignoring them).

A Strange Observation

Wanting to probe further, I ran tcpdump on My Computer. After connecting the ethernet cable to My Computer, I ran sudo tcpdump -n udp port 68 -v. And... it connects! My Computer gets an IP address (10.0.0.11) from the DHCP server. And I can use the connection to successfully browse stackoverflow.com without any problems.

Curious, I unplugged and replugged the ethernet cable, and then ran tcpdump slightly differently: sudo tcpdump -n udp port 68 --no-promiscuous-mode -v. Unfortunately, back to square one: no connection, no assigned IP address.

So it seems that My Computer can receive the DHCPOFFER, but only in promiscuous mode? I don't think the NIC is to blame; I am able to use wired ethernet successfully outside this experiment.

Another observation: I tried to connect My Computer to the network again, but this time without the switch (i.e. direct ethernet connection from My Computer to the RPi). Exact same problem. Exact same observation in the logs and with the tcpdump behavior.

Questions

  1. Is my network topology correct?
  2. Is my network hardware correct?
  3. What is wrong with the network configuration?

I've been trying to solve this problem by reading up a lot, and I've learned a lot from that, but I think it's time for me to get some pointers.

Flux
  • 269
  • 3
  • 9
  • 1.) Any topology is ok. You design the topology whatever you like it to be. 2.) We can't check your hardware. You need to do that yourself. Check each component individually, then plug them together as defined by your topology plan. 3.) I guess that's the real question – Thomas Weller Jan 31 '19 at 15:46

2 Answers2

3

I've solved the problem.

UPDATE: The main issue appears to be in the kernel drivers. Problem solved by upgrading the kernel from 4.14.50-v7+ to the latest stable version (currently 4.14.79-v7+) using sudo apt-get install raspberrypi-kernel, and rebooting. The steps outlined below are no longer necessary. (reference).

UPDATE 2: Bad drivers on the client side (e.g. My Computer in the network diagram above) could also be a problem. If the client's Ethernet Network Interface Card is Realtek RTL8111,8168,8411 PCI Express Gigabit Ethernet Controller (to check, run: lscpi | egrep -i 'ethernet'), Debian and Ubuntu both provide a better driver than the one already included in the kernel of a stock install. Install a better driver: sudo apt-get install r8168-dkms. I believe you will need to reboot for the new driver to be effective.

Observations

First, I unplugged the enternet cable from My Computer. On My Computer, I ran sudo tcpdump -n udp port 68 -e -vv.

Simultaneously, on the Raspberry Pi, I ran sudo tcpdump -n udp port 67 -e -vv.

Then I reconnected the ethernet cable to My Computer, and observed all the packets shown by tcpdump on both machines. I noticed that the tcpdump output on the Raspberry Pi would occasionally say that the udp checksum is bad. My guess is that this bad checksum is the reason why My Computer did not accept the DHCPOFFER packet from the Raspberry Pi.

To diagnose the issue further, I installed ethtool on the Raspberry Pi, and ran ethtool -k eth0:

Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
-- snip --

Apparently, the checksumming of outgoing packets is handled by the network interface itself (see tx-checksum-ipv4: on). I learned that this is called 'offloading' i.e. the calculation of checksums is 'offloaded' to the network interface so that the CPU does not have to do the work. Based on the problem I'm having, the network interface is doing a bad job at producing correct checksums.

The Solution

So I decide that the network interface should not checksum the outgoing packets and instead let the CPU handle that. i.e. I decide to disable offloading. To stop the network interface from checksumming outgoing packets:

sudo ethtool --offload eth0 tx off

So now ethtool -k eth0 shows this:

Features for eth0:
rx-checksumming: on
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
-- snip --

Et voilà! Now everything works as expected! Almost immediately after I connect the ethernet cable to My Computer, the DHCP process succeeds. And I can unplug, and replug the ethernet cable, and it still works.

Further Issues

The only issue with the solution is that the effect of sudo ethtool --offload eth0 tx off does not persist accross reboots. To solve this problem, I created a file /etc/dhcpcd.enter-hook to let a dhcpcd hook handle the issue:

# /etc/dhcpd.enter-hook
# See 'man dhcpcd-run-hooks' for more information.

if [ "${reason}" = "PREINIT" ] && [ "${interface}" = "eth0" ]; then
    ethtool --offload eth0 tx off
fi

If anyone has a better way of running ethtool --offload eth0 tx off that can persist across reboots, please share it.

References

These Raspberry Pi bug reports are probably relevant:

I've also opened an issue on GitHub for this problem: https://github.com/raspberrypi/linux/issues/2844

Flux
  • 269
  • 3
  • 9
  • +100, I'm sorry that I can give only one upvote. I will check my issues against this bug. But what I still does not understand: why does it work with promiscuous on? – Ingo Jan 31 '19 at 21:41
  • 1
    @Ingo Scouring the internet, I found that I wasn't the only one with this problem. There are also others scratching their heads about why it only works with promiscuous mode on. Most probably caused by buggy drivers. – Flux May 06 '19 at 21:40
0

Very well done work. Your network topology is correct. Also the hardware seems to be OK. Nothing is wrong with the network configuration but I haven't checked all details for a typo or so. But is seems to be OK because you can get an ip address from the DHCP server.

I have also seen this behavior with promiscuous mode. It was a sophisticated problem with mac addresses on a virtual interface ap0 linked with wlan0 on a wifi repeater. If you are interested in details search for promiscuous on that site.

As far as I see it is not a problem with your network. It is a problem on your client computer. You do not always have to fiddle with tcpdump. You can just enable/disable promiscuous mode on the interface with

pc ~$ sudo ip link set enp3s0 promisc on|off

I don't know something about the network settings on the client computer but my problem was that the network stack does not see the mac address f4:8e:38:e9:88:d7 to be member of the broadcast domain 10.0.0.0/24. If you enable promiscuous mode the stack will listen to all incomming ip packages not only to that are addressed to enp3s0 of the client computer. This way it will see the DHCPOFFER even it means it does not belong to the interface.

I suggest to strip down the network configuration on the client computer to a very simple setup with only one interface enp3s0 for testing.

Ingo
  • 40,606
  • 15
  • 76
  • 189
  • I have solved the problem: https://raspberrypi.stackexchange.com/a/93738/46753. The problem was on the Raspberry Pi. – Flux Jan 31 '19 at 20:44