4

[EDIT 6-december-2020] In my network I experience the same issue also in a raspberry pi v4, with the Raspberry OS image date December 2nd 2020

** Original post **

I am using Raspbian Buster on a raspberry pi 3b v2 (and a raspberry pi 4b v1.2) and I followed the @Ingo answer in this post. The version of the hardware is

Raspberry Pi 3 Model B Rev 1.2.

The kernel version is

Linux openhab 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux.

The version of the operating system is

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster

The failover works correctly, as far as the gateway is concerned. In normal conditions, i.e. when both ethernet and WLAN0 interfaces are up, the raspberry can reach the the gateway (i.e receives response to ping). When the ethernet link is down, the raspberry can still reach the gateway through WLAN0.

The raspberry cannot reach other hosts, if they are connected through WiFi to the same access point.

More precisely, in normal conditions, i.e. when both ethernet and WLAN interfaces are up, the raspberry cannot ping these hosts (and it cannot receive packets from these nodes). As soon as only one interface remains up (it can be either ethernet or WLAN0) the communication between the raspberry and the "problematic" hosts is re-established.

This behavior occurs with a fritzbox 7590 (fritzOS 7.13) but also with other acces points as well (technicolor tg789vac or Dlink DVA5992 or technicolor DGA4130).

[EDIT 26-april-2020]

Thanks to hints by @Ingo in the comments, I nailed down the issue to the ARP packet traffic exchange, by using tcpdump on the raspberry. I do not understand if this is an issue with the router or with the bonding module. To simplify the description let me add a scheme of the network connections.

   +-(WLAN0)---(WLANfritz)-+   +---(WLANfritz)---Device2(192.168.27.16)
   |                       |   |
Raspberry(192.168.27.2) FritzBox(192.168.27.1)
   |                       |   |
   +-(ETH0)---------(LAN1)-+   +-(LAN2)----------Device1(192.168.27.30)

The raspberry can always ping 192.168.27.30 while it can ping 192.168.27.16 only when only ONE interface (it can be ETH0 or WLAN0) is up (or in other conditions that I will detail later).
It turns out that the ARP Reply packet sent back by 192.168.27.16 to the raspberry in response to its ARP request always reaches the raspberry physical interface (either ETH0 or WLAN0) but it does not always reach the BOND0 interface.

To trigger an ARP exchange I flush the arp cache with the command

sudo ip -s -s neigh flush 192.168.27.xx

after that I give the usual command

ping 192.168.27.xx

At the same time in three different terminals I give the following commands

    sudo tcpdump -i bond0 -vv arp -nn host 192.168.27.xx 
    sudo tcpdump -i eth0  -vv arp -nn host 192.168.27.xx
    sudo tcpdump -i wlan0 -vv arp -nn host 192.168.27.xx

Let me consider the connection with the reachable device first (192.168.27.40).

Both Interfaces Up

(wlan0) tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
(bond0) tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
(eth0)  tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
(bond0) 10:50:50.905705 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.40 tell 192.168.27.2, length 28
(eth0)  10:50:50.905757 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.40 tell 192.168.27.2, length 28
(eth0)  10:50:50.913403 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.40 is-at b4:43:0d:dc:8b:e7, length 112
(bond0) 10:50:50.913403 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.40 is-at b4:43:0d:dc:8b:e7, length 112
No packet reaches wlan0

The request is first sent to bond0, then passed to eth0. The reply is received by eth0 and simultaneously received by bond0.

Now let's consider the non reachable device (192.168.27.16)

(bond0) 10:56:53.731660 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(eth0)  10:56:53.731710 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(wlan0) 10:56:53.827315 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 28

This sequences continues for a long time while the ping command gives

From 192.168.27.2 icmp_seq=1 Destination Host Unreachable

The raspberry sends the request to bond0, it passes through eth0 but the answer arrives through wlan0 and it is not passed to bond0 and it is therefore discarded.

If the wlan0 interface is down, the ARP exchange to 192.168.27.40 is the same as before, while for 192.168.27.16 it is as follows

(bond0) 11:05:06.019184 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(eth0)  11:05:06.019241 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(eth0)  11:05:06.062927 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 46
(bond0) 11:05:06.062927 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 46

The request is first sent to bond0, then passed to eth0. The reply is received by eth0 and simultaneously received by bond0. The raspberry can therefore ping 192.168.27.16.

If the eth0 interface is down (and the wlan0 interface is up, of course)

(bond0) 11:13:10.508756 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(wlan0) 11:13:10.508808 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(wlan0) 11:13:10.526858 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 28
(bond0) 11:13:10.526858 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 28

If, for debugging purposes, I deselect the router security option "Active Wireless device can communicate with each other" (which is not something I want, because I need a wireless connected PC to print to a wireless connected printer), when both interfaces are up the raspberry can communicate with 192.168.27.16. The traffic is as follows

(bond0) 11:22:01.371787 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.2 tell 192.168.27.16, length 46
(eth0)  11:22:01.371787 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.2 tell 192.168.27.16, length 46
(eth0)  11:22:01.371881 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.2 is-at 7a:79:3c:31:5d:e3, length 28
(bond0) 11:22:01.371894 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.2 is-at 7a:79:3c:31:5d:e3, length 28

Obviously, with this router configuration, when eth0 goes down and only wlan0 remains up, the raspberry cannot reach 192.168.27.16. In this case the ARP reply packet never returns back.

In summary, the raspberry cannot communicate with hosts when the reply packet arrives from the wlan0 interface while the request was sent on eth0, because, I think, bond0 only receive packets from the active interface and discards others.

Is this a bug of the bonding module? Does this happen also for the raspberry pi v4? Is it a configuration problem of the router (but how can I deal with it)? Would it be possible to tweak the bonding kernel parameters (I tried with AllSlavesActive options, but I've never seen it used in various bonding examples. Even if I activate it in the netdev file, it does not seem to be really active by looking at the content of /sys/class/net/bond0/bonding/all_slaves_active). Should I try to force such a parameter directly in /etc/modeprobe.d/?

Some details of the configuration are given in the following.

When both interfaces are up this is the status of bonding

openhabian@openhab:~$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 500
Up Delay (ms): 1000
Down Delay (ms): 1000

Slave Interface: eth0
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: b8:27:eb:b3:1c:7d
Slave queue ID: 0

Slave Interface: wlan0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 2
Permanent HW addr: b8:27:eb:e6:49:28
Slave queue ID: 0

The ip address in this case is

 openhabian@openhab:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 7a:79:3c:31:5d:e3 brd ff:ff:ff:ff:ff:ff
3: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:79:3c:31:5d:e3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.27.2/24 brd 192.168.27.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fd6f:17f9:753c:0:7879:3cff:fe31:5de3/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::7879:3cff:fe31:5de3/64 scope link
       valid_lft forever preferred_lft forever
4: wlan0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 7a:79:3c:31:5d:e3 brd ff:ff:ff:ff:ff:ff

The ARP cache reflects the fact the the communication with the problematic host cannot be established. Even if I delete the host from the ARP cache in order to force an ARP search, the problem remains

openhabian@openhab:~$ ip neighbor
192.168.27.1 dev bond0 lladdr 08:60:6e:eb:58:f8 REACHABLE
192.168.27.54 dev bond0 lladdr 50:46:5d:b5:80:bf REACHABLE
192.168.27.30 dev bond0 lladdr 34:f3:9a:cd:e5:4c REACHABLE
*192.168.27.16 dev bond0  FAILED*

openhabian@openhab:~$ sudo ip neighbor delete 192.168.27.16 dev bond0

ip neighbor
192.168.27.1 dev bond0 lladdr 08:60:6e:eb:58:f8 REACHABLE
192.168.27.54 dev bond0 lladdr 50:46:5d:b5:80:bf REACHABLE
192.168.27.30 dev bond0 lladdr 34:f3:9a:cd:e5:4c REACHABLE
*192.168.27.16 dev bond0  FAILED*

As soon as one of the interfaces goes down communication with the node restarts without need to delete the ARP cache:

openhabian@openhab:~$ sudo ip link set wlan0 down
openhabian@openhab:~$ ip neighbor
192.168.27.1 dev bond0 lladdr 08:60:6e:eb:58:f8 REACHABLE
192.168.27.54 dev bond0 lladdr 50:46:5d:b5:80:bf REACHABLE
192.168.27.30 dev bond0 lladdr 34:f3:9a:cd:e5:4c REACHABLE
*192.168.27.16 dev bond0 lladdr 38:1a:52:47:32:35 REACHABLE*

As soon as two interfaces are up the communication fails again.

LionHe
  • 71
  • 11
  • Configuration looks good so far. If you set **eth0** down and **wlan0** up, can you then ping 192.168.27.16? Do you can ping all other ip addresses no matter what interface is up or down? – Ingo Feb 09 '20 at 16:06
  • Yes, If I set eth0 down and wlan0 up I can ping that problematic host and all others. – LionHe Feb 09 '20 at 17:17
  • I substantially updated the initial question with tcpdump outputs in various conditions. @Ingo: It would be interesting to see if this behavior occurs also in your case. – LionHe Apr 26 '20 at 10:14

2 Answers2

3

It appears that it was sufficient to add the AllSlaveActive=1 option to the 02-bond0.netdev file described in @Ingo answer. In this way the ARP packet sent back by the router through the wlan0 interface is forwarded to the bond0 one and the ARP table is always populated. In order to set this parameter the restart of systemd-networkd.service was not enough, I had to reboot the raspberry. After that it was possible to ping all devices in all cases.

root@raspberrypi:~ # cat >/etc/systemd/network/02-bond0.netdev <<EOF
[NetDev]
# status: cat /proc/net/bonding/bond0
Name=bond
Kind=bond
[Bond]
Mode=active-backup
# primary slave is defined in *eth.network
MIIMonitorSec=500ms
MinLinks=1
AllSlavesActive=1
EOF

I do not know if there are any negative side-effects of this parameter. I also wonder why this is not necessary for the Raspberry pi v4 case described in @Ingo's post (or perhaps it depends on how the router manages the mac addresses of the wired and wlan connections).

EDIT 07/06/2020

Actually I experienced some negative side effects: sometimes the raspberry was not reachable anymore until I disconnected the ethernet cable. I therefore figured out a less elegant solution. I restored AllSlavesActive=0 and implemented the following simple script

#!/bin/bash

SLEEP=5
echo "starting wlan control"
date
while : ; do
    ETH0_STATUS=$( cat /sys/class/net/eth0/operstate)
    WLAN0_STATUS=$( cat /sys/class/net/wlan0/operstate)

    if [ $ETH0_STATUS == "up" ] ; then
        if [ $WLAN0_STATUS == "up" ] ; then
            echo "eth0 ON: wlan0 ON: setting wlan0 DOWN"
            sudo ip link set wlan0 down
            date
#        else
#            echo "eth0 ON: wlan0 OFF: doing nothing"
        fi
    else
        if [ $WLAN0_STATUS == "down" ] ; then
#           echo "eth0 OFF: wlan0 ON: doing nothing "
#           iwconfig
#        else
            echo "eth0 OFF: wlan0 OFF: setting wlan0 ON"
            sudo ip link set wlan0 up
            date
#           iwconfig
        fi
    fi
    sleep $SLEEP
done

to run in a window opened with tmux (so that it remains active at logout)

LionHe
  • 71
  • 11
  • Woow! What an issue. But I don't understand it in theory. I will try to reproduce it. The only important difference I see is the Raspberry Pi version. You use a RPi 3B (not 3B+, right?) and I tested with a RPi 4B. Do you see another difference? – Ingo Apr 27 '20 at 10:52
  • I modified the main question with more details on the raspberry pi version, the kernel version and the raspbian version, so that we can compare. But I think that the behavior that I have observed is not really anomalous. According to this [post] (https://networkengineering.stackexchange.com/questions/34754/same-mac-on-different-interfaces) when two interfaces of an host share the same MAC on a LAN, the switches in the network will always use only one MAC in an unpredictable way. Probably professional devices deal with this bonding configuration better than the consumer ones. – LionHe Apr 27 '20 at 12:30
  • @Ingo I recently got a raspberry pi v4 v1.2 and I found exactly the same problem. I started from clean images in parallel both with my raspberry pi v3b and v4b. The kernel was newer and the same on both. As you said that you also have a fritzbox I would be really curios to know what is the cause of this misbehavior. I would have added a comment to your original post on failover but I have not enough reputation. I think that also some other users may experience a similar problem with failover. – LionHe Dec 06 '20 at 13:15
  • My problem is that I can't test it because I don't see the error. – Ingo Dec 06 '20 at 14:27
1

According to the commands output you have given about the configuration, bonding looks good so far. That all interfaces have the same mac address is due to bonding and will not confuse the fritz box router. Quite contrary it ensures that it does not sense the change of an interface on the fly because the mac address doesn't change. That's the trick of bonding.

You have one (or more?) devices that you only can ping if only the wlan0 interface is up as noted in a comment. If you have only the eth0 interface up you cannot reach the problematic device. This is also the case if both interfaces are up. Then eth0 is always used if available because of Primary Slave: eth0 (primary_reselect always). So we can say, if using the wired connection (eth0) then the problematic device cannot be connected. Because everything is working as expected with bonding I don't believe its a problem with it. You should look at your network configuration and/or at the fritz box router what could be the reason that only the WiFi connection works in this case.

Ingo
  • 40,606
  • 15
  • 76
  • 189
  • @LionHe All the same, look at your network. I cannot saying much more about it with the (not) available information about your network. What have I to do to reproduce the error? Haven't seen it before. – Ingo Feb 09 '20 at 20:19
  • I have a few devices that I cannot ping when eth0 and wlan0 are both up. If eth0 is down (unplugging the chord of with sudo ip link set eth0 down) then I can ping them again. If I plug the ethernet chord or give the sudo ip link set eth0 up then the devices are not reachable. If I switch off wlan0 with sudo ip link wlan0 down then the devices are reachable again. I can post the sequence of commands in the first post if it is useful to clarify this. In any case, thanks for your help, I'll try to reproduce the issue in more reproducible conditions (i.e. switching off all my network) – LionHe Feb 09 '20 at 20:20
  • Yes, please [edit](https://raspberrypi.stackexchange.com/posts/108191/edit) your question and add it there. To summarize: if only one interface is up, no matter if **eth0** or **wlan0**, then you can ping all devices on the network. If both interfaces are up, then there are some devices that you cannot ping but the other devices you can ping. Can you confirm this? Do you see any differences between these devices belonging to network connection? Are they always the same failing devices? – Ingo Feb 09 '20 at 21:07
  • In the fritz-box do you have enabled "*Wireless -> Security -> The active wireless devices displayed below may communicate with each other*"? – Ingo Feb 09 '20 at 21:07
  • I modified the question. Meanwhile I found that when both interfaces are up, the Raspberry can't connect to the hosts (not only smart devices, but also android phones or tablets) that are connected to the same access point to which wlan0 is associated. This occurs on the fritzbox but also on another access point (dga4132). As soon as eth0 or wlan0 go down every connection work. – LionHe Feb 10 '20 at 20:26
  • @LionHe That is really curious. There is another access point involved? Can you test with a simple setup only with the fritz-box-router? My Fritz!Box is working, receiving bonded connections. In the next step I would use **tcpdump** (`sudo apt install tcpdump`) and look at the RasPi and on the failing device why they doesn't reply to ping. – Ingo Feb 10 '20 at 21:00
  • I isolated part of my network, keeping only the fritzbox, a tg789vac acting as a switch and access point and the raspberry (I excluded a managed switch and two other access points). The behavior remains the same as before, ruling out some interference by the managed switch. The test with the fritzbox and Raspberry alone will require some more time but it is the next on my to do list. – LionHe Feb 11 '20 at 22:09
  • Even when only the raspberry is directly connected to the fritzbox (I rebooted both after connections) and everything else is disconnected the behavior remains the same. I'll try next with a fresh install of raspbian buster lite. – LionHe Feb 13 '20 at 18:45
  • @LionHe Really weird. To be sure I will reactive an old spare Fritz!Box and verify the configuration. The current used Fritz!Box is needed for my network. – Ingo Feb 13 '20 at 23:07
  • @LionHe I have just tested the setup only with one Fritz!Box and one RPi 4B connected wired and wireless. No other connections existing (except power). All combinations up and down interfaces are working as expected. Also when both interfaces are up. There was a wrong option `mode=RSN WPA` set in `wpa_supplicant-wlan0.conf`, must be `proto=RSN WPA`. I have corrected it. With the latest Raspbian Lite version 2020-02-05 you have also wlan0 to `rfkill unblock 0` (have added it to the doku). – Ingo Feb 14 '20 at 01:35
  • I started from scratch from a fresh image of raspbian buster. In my network there is only the fritzbox 7590 (FritzOS 7.13), the raspberry pi v3b connected by the ethernet chord and a few wifi devices connected to the fritzbox . I followed your updated instructions, but I had to give 'rfkill unblock 1' instead of 0, otherwise wlan would not come up. The behavior is always the same: when both interfaces are up the raspberry cannot communicate with the other wifi devices connected to the fritzbox router. – LionHe Feb 15 '20 at 08:57
  • looking at dmesg I find the following lines '[ 428.579027] bond0: link status down for backup interface wlan0, disabling it in 1000 ms [ 428.579053] bond0: invalid new link 1 on slave wlan0' . Are These link related to the output of the ip link command? – LionHe Feb 15 '20 at 09:27
  • May be something changed in the new versions of FritzOS? Which version did you use? I can try with a different modem router in order to replace the fritzbox, but that's not a great solution. As a workaround I wll add a new access point in my network dedicated to the raspberry only. Should I update the question in order to reflect this limitation of failover on the raspberry pi v3b? – LionHe Feb 15 '20 at 09:45
  • I replaced the fritbox router with a DVA5992 modem router, with only the raspberry connected. The behavior is even worse: the raspberry can only ping the router and never the devices connected to WiFi. I think that we can conclude that failover bonding on buster for raspberry pi v3b does not completely work, differently from raspberry pi v4. – LionHe Feb 15 '20 at 14:19
  • @LionHe That you cannot see other devices on the WiFi is a well known security issue on the router. It does not allow this by default. You have to enable it on the routers settings. – Ingo Feb 15 '20 at 23:52
  • I have played with the setting on the fritzbox "Clients can communicate with each other". When both interfaces are up the raspberry can connect to the wifi connected hosts only if the options **IS NOT** activated. But when the eth0 interface is down, then the raspberry can communicate with the other wifi connected hosts only if the option **IS** activated. I do not know if this can help – LionHe Mar 07 '20 at 19:28
  • @LionHe If both interfaces are up, then only eht0 is used, never wlan0. That's by definition. So the RasPi connected to the fritzbox by eth0. **What is the fritzbox doing with wired connections to connect to its wireless clients?** When eth0 is down, then the RasPi uses only wlan0 to connect to the access point of the fritzbox. Communication to other wifi devices with option enabled is as expected. – Ingo Mar 07 '20 at 20:36
  • @LionHe Just a speculation: maybe the fritzbox is detecting something like a network loop: eth0 -> (inbound) fritzbox (outbound) -> wlan0 and will disable this loop. It works if you disable the routing option??? Test with `sudo systemctl stop wpa_supplicant@wlan0.service`. Then you have no wifi signal from the RasPi at all. – Ingo Mar 07 '20 at 20:54
  • @Ingo.I do not understand your question: therefore let me clarify. If the raspberry is connected through LAN to fritzbox and both eth0 and wlan0 interfaces are up it can ping other wireless client (i.e. my PC connected through wireless to the fritzbox) only if the fritzbox option "clients can communicate between themselves" is **deselected** OR if the wlan0 interface is down. If I disconnect the eth cable then the raspberry can ping the other clients only if the fritzbox option is **selected**. – LionHe Mar 07 '20 at 20:55
  • Isn't disabling the wpa_supplicant the same as disabling the interface? I any case I'll test again. I suspect that the rasperri pi3b kernel does not correctly respond to ARP packets in the case of bonding. – LionHe Mar 07 '20 at 21:00
  • @LionHe Yes, I understand. If eth0 and wlan0 are up and router option enabled then the fritzbox detects a loop (maybe) and blocks connection. If eth0 and wlan0 are up but router option disabled then there cannot be a loop detected and the fritzbox routes traffic from wired to wireless as it always do (maybe). The option is set in the wireless section of the fritzbox and is only valid for wireless clients (maybe). – Ingo Mar 07 '20 at 21:05
  • @LionHe I think shutdown wpa_supplicant isn't the same than disabling interface wlan0. Check with `sudo ip link set wlan0 down` and `sudo iw dev wlan0 info`. You will see that the wifi is still up (type managed, txpower) but only not connected. – Ingo Mar 07 '20 at 21:21