0

I am running a small server with 6 docker services on a Raspberry Pi Modell B. After a short time of inactivity (and with that I mean no being connected via ssh) of approximately 20-30 minutes, my services can not be reached anymore. I have assigned a static IP address to my Pi. Pinging it yields in no response. I can of course not login to SSH.

This has not happened before. It has run rock solid for the last 6 months 24/7 without any shutdown. I am unsure how to approach this problem and I have no idea how to debug it.

I have tried to find something suspicious in /var/log/syslog/ but I had no success. There is no "gap" in the timestamps of the log file which looks like that the Pi hasn't crashed or anything related to that. The red power LED was on the whole time.

Question: How can I debug this kind of problem? What are things to search for in the syslog file? (I would share my syslog file here, but I am not sure if this would be a security concern. If it isn't, let me know.)

Remarks: My problem might be related to this question but it seems to be slightly different.

This is my setup.

  • Raspberry Pi Model B
  • Raspbian GNU/Linux 10 (buster)
  • Connected via LAN (no Wifi)
  • Official recommended power supply I had no problems for the last six month. I have not updated anything on my Pi. The problem suddenly appeared.
David Scholz
  • 121
  • 6
  • 1
    Have you disabled power management on your WiFi adapter? https://raspberrypi.stackexchange.com/a/96608/94397 – Dougie Mar 23 '22 at 20:51
  • @Dougie yes, I have done that. Sadly, this haven't made a difference. – David Scholz Mar 24 '22 at 01:05
  • 1
    Does it happen when the 6 docker services are not running? – Katu Mar 24 '22 at 09:43
  • Don't know if you've resolved this issue or not, but FWIW, there is a [related thread on GitHub](https://github.com/raspberrypi/linux/issues/4092) where this same issue is discussed. – Seamus Apr 04 '22 at 18:15
  • 1
    Thank you @Seamus for the follow up. The issue was solved with the steps provided in my answer, though I have no idea *why* it has solved it since the root cause is still unknown to me. Your answer has given me great insights on aspects that I haven't known until then, and it could be very much possible that the issue discussed on the provided GitHub thread has something to do with it as well. As usual, the time for debugging is rare on my side at the moment, so I'm happy that it "just works' somehow right now. Thank you again for your help! – David Scholz Apr 04 '22 at 19:41
  • 1
    Good enough then... if you do get around to debugging at some point, a follow-up here would be much appreciated. – Seamus Apr 04 '22 at 19:46

2 Answers2

2

I am not quite sure I follow your explanation of things in the answer you've provided. Your answer indicates you have resolved your problem for now, but your closing sentence suggests you are not quite sure what happened or why. I will try to address that here.

An Introductory:

As a point of clarification for readers who may be unfamiliar, please note the following:

  1. DHCP is the dynamic host configuration protocol. This protocol was designed to simplify IP address provisioning in networks by avoiding the error-prone task of manual provisioning.

  2. dhcpcd stands for DHCP client daemon. Its job is to obtain - from a DHCP server - all the required information needed by the RPi to conduct network operations: an IP address, a default route, and a link to DNS services - as a minimum. dhcpcd is currently the default networking software for RPi. It was chosen as the default DHCP client app for RPi OS by the powers that be in the Raspberry Pi organization, but you of course are free to change that.

  3. dhcpd stands for DHCP daemon. It is the server-side app for DHCP. Unless you're using your RPi as a DHCP server itself, you will not need to install it. Being a DHCP server is ordinarily a full-time job. Networks may become uncooperative when their DHCP server goes on sabbatical!

  4. "Routing" and "routers" refer the process and systems used to move network traffic between their sources and their destinations. It's often the case in smaller networks that routers and DHCP servers cohabit a single box or appliance. Accordingly references to routers sometimes include the DHCP server function as well. The only point here is that routing and DHCP are separate and distinct network functions, but they may be housed together.

I do not claim any particular expertise in DHCP. But I have at least read the system documentation for dhcpcd (man dhcpcd.conf & man dhcpcd), and the Internet Standard for DHCP - RFC 2131. I had occasion to review the protocol some time ago in relation to this old Q&A - which may be relevant to your experience?

FWIW - My take on your experience:

You stated:

  1. The router assigns a static IP address to the pi's mac address.

I am not sure why you say this because the router/DHCP server does not assign static IP addresses! Perhaps you meant that you - as the router/DHCP administrator - made a "static mapping" assignment manually through your router/DHCP server's configuration interface? I'll emphasize this point as it seems to confuse many: Creating a static IP for your RPi in dhcpcd.conf does not induce your router or your DHCP server to do anything at all.

In fact, the static IP configuration in dhcpcd.conf will prevent your RPi's dhcpcd from communicating with the DHCP server. dhcpcd is a client of the DHCP server; it sees a static ip declaration in dhcpcd.conf as the system administrator (you in this case) saying, "I don't need you to act as a client; I'm providing the DHCP server's information for you here." Put another way, dhcpcd will never communicate the static ip configuration you've made in dhcpcd.conf to your router/DHCP server.

But there is a line in man dhcpcd.conf that's very relevant to both DHCP client and server wrt a static IP:

For IPv4, you should use the inform ipaddress option instead of setting a static address.

The syntax for the inform option is given by example below for the wlan0 interface in IPv4:

wlan0
inform 192.168.1.51/24    # '/24' is the CIDR unless you've created a subnet

What does that inform option mean?

I'll call your attention to para 3.4 of RFC 2131. This passage explains the recommendation above for using the inform option: The inform option in dhcpcd.conf provides the mechanism needed to let your router/DHCP server know that you have assigned a static IP address to your RPi. Hopefully it is clear that the router/DHCP server needs this information.

The inform option "informs" the DHCP server to bind the client's MAC addr to the designated IP address, and prompts the server to complete the provisioning process by supplying the default route, DNS hosts and any other data the DHCP server ordinarily sends to clients (e.g. NTP server address for time services). Note however this passage from para 3.4:

The server SHOULD check the network address in a DHCPINFORM message for consistency, but MUST NOT check for an existing lease.

This passage emphasizes that it is the administrator's responsibility to avoid address collisions for static IP assignments. Put another way, the inform option does not give the administrator carte blanche authority to assign addresses willy-nilly without consequence.

Hopefully, it is now clear that the correct approach to static ip usage includes configuration of the DHCP server so that it will never assign an IP address designated in the inform option. How this is done will depend on the characteristics of the DHCP server in use. The exception to this of course is if your entire network is static addresses; in this case you have no need for a DHCP server (or dhcpcd for that matter).

In summary, there are two points:

  1. If you have a DHCP server on your network, placing a static ip configuration in dhcpcd.conf is unnecessary at best, and a source of chaos in your network at worst. This due to the fact that a static ip configuration in dhcpcd is never communicated to the DHCP server. Once again, this is why the author of dhcpcd included the admonition against it in his documentation, and his recommendation to use the inform option.

  2. I don't use docker services, and I'm not familiar with how they obtain their networking information. However, quickly skimming this article from the docker docs suggests that your docker services may be completely dependent on your RPi's network configuration! If that's the case, broken networking on your RPi implies broken networking in your docker apps.

    I hope that understanding how DHCP clients & servers interact (or don't interact), provides you with some useful clues for your analysis. I also hope this "answer" will dissuade you (and others) from using the static ip configuration shown in your answer, and to use the inform (or request) option instead.

Seamus
  • 18,728
  • 2
  • 27
  • 57
  • Thank you for these insights. I made the assignment `static ip address -> mac address` manually in the router itself. I haven't set a static IP address inside the Pis dhcpcd.conf (before this issue; I haven't touched this file ever). The Pi asks the router for an IP address and the router assigns the static IP address which I have configured in the DHCP server which run on the router. What I am not sure about is, why setting the static ip adress in the dhcpd.conf as well solves the issue. This shouldn't be necessary from my current understanding. – David Scholz Mar 24 '22 at 16:51
  • That said, I'm not sure why a device should inform the router about a static IP address. The router's DHCP server does the assignment anyway. – David Scholz Mar 24 '22 at 16:52
  • The static ip address specified in dhcpcd.conf should be outside the range managed by the dhcp server in the router so surely the dhcp server doesn't assign any dhcp address to a device which already has a preset static ip address ? – Drdrm46 Mar 24 '22 at 17:21
  • 1
    It shouldn't be necessary to use dhcpd.conf at all, if the router manages the static IP assignment... – David Scholz Mar 24 '22 at 17:26
  • 1
    I agree it shouldn't be necessary but assigning a static ip address in dhcpcd.conf works very reliably in my experience – Drdrm46 Mar 24 '22 at 17:50
  • Apparently I did a poor job of explaining. – Seamus Mar 24 '22 at 22:05
  • I have an issue with the *the italicized part* of the following statement: "If you did manually configure your DHCP server to reserve the same IP address for which you made the static assignment on your RPi, *and that value was outside the "pool" of addresses allocated for assignment* (emphasis added)...". My experience with home routers is that the MAC based IP address reservation in the DHCP settings of home routers do not allow reserve an address outside the "pool". Or rather one can do it, but it has no effect, i.e. no reservation. This is my limited experience. – user68186 Mar 25 '22 at 15:58
  • @user68186: I can't comment on your experience except to say it's not my experience. I use the [OPNsense](https://opnsense.org/) router/fw. Its DHCP configuration page allows one to declare "**DHCP Static Mappings**", and they definitely must be outside the "pool" of addresses available for assignment to DHCP clients. Typically, I think most home/SB users divide their /24 subnet into two sets: a "DHCP pool" for dynamic clients, and a range of addresses that will not be assigned to dynamic clients. This latter range of addresses are available for "**DHCP Static Mappings**". – Seamus Mar 26 '22 at 18:41
  • Thanks for sharing your experience. – user68186 Mar 26 '22 at 22:56
  • I up voted this answer as it clarifies lots of concepts new users of Pi may not be familiar with. I understand your position about opnsense pfsense and the DHCP server being separate from the router. However, I use (like many) a consumer grade router. I have no control over its software. I can't do much if the router does not listen to `inform` or `request` short of throwing it away and building my own. :) This is why I find `static ip_address=` and the 2 next lines useful. With these the Pi does not have to negotiate with the router's implantation of DHCP server. – user68186 Mar 27 '22 at 03:10
  • @user68186 You can do testing. As far as a static ip, you get no shelter from chaos if the DHCP server assigns your static ip to another host - which will eventually happen. DHCP needs cooperation from all hosts if it's to work. I'll guess that many of these units will surprise you. The RFC dates back 25 years - to 1997 - so it seems likely that they would support inform - almost certainly request. If not, some can be outfitted with other software - OpenWRT for example. Check the [list of supported hardware](https://openwrt.org/supported_devices). – Seamus Mar 27 '22 at 05:03
  • 1
    @DavidScholz: `It shouldn't be necessary to use dhcpd.conf at all, if the router manages the static IP assignment...` That's correct. In "olden days" - before DHCP showed up - every host on the network was manually configured. It worked, but could be brittle. I wonder how many RPi would ever be used today if all networks were manually configured? – Seamus Mar 27 '22 at 19:02
1

I have figured it out, but it is still confuses me why this has not happened before. In short: it seems like the Pi assigns a different IP address to itself after a while. I have monitored the network traffic in my firewall and the static IP address became inactive (as expected), but the DHCP server of my router did not assign a new address to any device in my current network, which led me to the conclusion that the Pi itself does it (which the router does not get notified about of course).

My current setup does the following:

  1. The router assigns a static IP address to the pi's mac address.
  2. The IP addresses of each docker service is mapped to the same static IP address as usual.
  3. The DNS of my router assigns an URL for each docker service to the same IP address of the pi.
  4. My reverse proxy forwards the requests for the specific URLs to the correct service.

The IP address is correctly assigned to the Pi which I have verified by executing the following command.

ifconfig

I have also executed the following command on my Pi.

hostname -I

The static IP address assigned by the router was correctly shown here as well as the IP addresses of the docker services, but I have noticed 8 more IP addresses which I still don't know where they are comming from (and I don't even know if this is related to the problem).

Nevertheless, here is what I did to resolve it. I assigned the static IP address from my router directly in /etc/dhcpcd.conf as follows.

Get the current IP address of the router

ip r | grep default

which yields in something like the following output.

default via 192.168.22.1 dev eth0 proto dhcp src 192.168.22.102 metric 202

The first one is the IP address of the router. The IP address of the currently used DNS is located in /etc/resolv.conf, which matches with my router's ip address in my case.

Assign a static IP address in /etc/dhcpcd.conf

Next, edit the file /etc/dhcpcd.conf, uncomment the following lines and add the information that we have gathered above.

# Example static IP configuration:
interface eth0
static ip_address=192.168.22.100/24
static routers=192.168.22.1
static domain_name_servers=192.168.22.1

In the field static ip_address, we must of course put the static IP address which the router assigns to the Pi as well.

Reboot via sudo reboot. SSH back into the Pi and run hostname -I again. The 8 unknown IP addresses from above have disappeared. The static IP address from the Pi as well as the IP addresses from my docker services remain.

I will leave this question open for a bit longer with the hope that someone has more insights on why this happened.

David Scholz
  • 121
  • 6
  • Just to say that I have found modifying dhcpcd.conf as the reliable method of ensuring a static ip address. – Drdrm46 Mar 24 '22 at 16:28
  • You don't need to assign static IP address in both router and the `dhcpcd.conf` if you assign a static IP address in `dhcpcd.conf` that is outside the routers pool of DHCP address assignment. for example, if your router assigns addresses in the range of `192.168.22.100-250` then you can assign a static address of say `192.168.22.31` as 31 is less than 100. Assuming you have not assigned `31` to another device yourself. as @Seamus said, the `inform` statement should also work. – user68186 Mar 24 '22 at 20:06
  • I actually want the router to decide which IP address to use. I haven't used `dhcpcd.conf` at all prior to my issue and everything worked as expected. If I use the same static IP address in `dhcpcd.conf` which I have assigned to the Pi in the routers DHCP server, then it suddenly worked again. I have no idea why this is the case. – David Scholz Mar 24 '22 at 20:10
  • 1
    I agree what you experienced should not happen. Sometimes routers drop the connection and the Pi (or rather dhcpcd) says "let me ask for another IP address". Whatever the reason, since you want a static IP address and you are messing with `dhcpcd.conf` you might as well get one outside the dynamic address pool of the router. That way the router will have more addresses to dynamically assign to more devices in your network. – user68186 Mar 24 '22 at 20:18
  • This is a good hint, I will play around with this a bit more. Thank you! – David Scholz Mar 24 '22 at 20:20
  • 1
    @Seamus said elsewhere: The `request ` line in [dhcpcd.conf](https://manpages.debian.org/buster/dhcpcd5/dhcpcd.conf.5.en.html) should do what you are trying to do. Request a specific IP address from within the dynamic address pool of the router. – user68186 Mar 24 '22 at 20:24
  • 1
    Here is [an answer](https://raspberrypi.stackexchange.com/questions/37920/how-do-i-set-up-networking-wifi-static-ip-address-on-raspbian-raspberry-pi-os/74428#74428) that recommends the `request` statement in `dhcpcd.conf`. – user68186 Mar 24 '22 at 20:37
  • @user68186, @DavidScholz: I'm fuzzy on the exact difference between the `request` option, and the `inform` option for the situation here. But I've learned recently that some errant code in dhcpcd is causing the `inform` option to create a massive load of log messages! The author of `dhcpcd` (Roy Marples) has reportedly fixed this, but it may take some time before we see that via `apt`. Until then, the `request` option may serve, or leaving the `static ip` config in dhcpcd.conf may be benign as long as you have an entry in your DHCP server consistent with that. – Seamus Mar 26 '22 at 23:48
  • @Seamus My reading of the man page is: `inform` does the same thing as `static ip` and is supposed to be used to assign a static IP outside the dynamic pool of the router. The `request` on the other hand requests for a specific IP address from the dynamic pool. The request may or may not be honored by the router and the router may decide to assign a different IP address ignoring the request particularly if that specific IP address is already assigned to another device. – user68186 Mar 26 '22 at 23:59
  • @Seamus one difference between `inform` and `static ip_address=` is that the `static` statement is followed by 2 other `static` statements, while `inform` does not seem to need them. The dhcpcd is supposed to use the "default values" if one uses inform. I got this by [playing with the GUI network settings](https://raspberrypi.stackexchange.com/questions/133424/how-to-set-up-static-ip-address-via-the-desktop-gui-in-buster-bullseye/133426#133426), and then looking at what changed in the `dhcpcd.conf` file. – user68186 Mar 27 '22 at 00:09
  • 1
    @user68186: You're on the right track. I revised my answer earlier today in an effort to clarify some of this, but yeah - the `static` option communicates nothing to the DHCP server. – Seamus Mar 27 '22 at 02:31
  • Hello again David. I ran across an [interesting article on docker networking](https://docs.docker.com/network/host/) today. I've not read it for detail, but was curious to learn if your dockers use `host networking`? – Seamus Apr 22 '22 at 00:37