How do I fix incorrect timestamps in my bash script that launches on startup?

Question

I'll try to explain this the best that I can. I am creating a simple bash script that runs on startup. This script is supposed to create an archive based on the current date, run a python program that generates an audio file (pyaudio), and then move the .wav files to the archive previously created.

Here is the code:

    #!/bin/bash

    timestamp=$(date +"%Y-%m-%d %H:%M")

    date=$(date +"%Y-%m-%d")

    log=/home/pi/bar/log.txt

    # archive=/media/pi/ARCHIVE/$date

    archive=/home/pi/bar/$date

    echo "[ $timestamp ] Script Started" >> $log

    if [ ! -d $archive ]; then
        mkdir -m 777 -p $archive;
        echo "[ $timestamp ] Archive Created Successfully..." >> $log
    else
        echo "[ $timestamp ] Archive Already Exists! No New Folder Created." >> $log
    fi

    sudo -H -u pi python3 /home/pi/bar/record_audio.py >> $log

    filecheck=$(ls /home/pi/*.wav 2> /dev/null | wc -l)

    if [ "$filecheck" != "0" ]; then
        mv *.wav $archive
        echo "[ $timestamp ] Files Moved." >> $log
    else
        echo "[ $timestamp ] No Files Found." >> $log
    fi

It works fine if I run the bash script manually through the command line, but if I allow it to run automatically at startup, the timestamps for files and archives are completely wrong. I'm not sure where to start with troubleshooting this issue.

score 2 · Answer 1 · answered Jun 04 '19 at 00:24

When the Pi boots it has no time reference.

Raspbian (default installation) will restore the time from the last saved by fake-hwclock, which should be within 1 hour of the time os shutdown.

Depending on how you run the script it should use this saved time until synchronised by NTP.

You could install a RTC or write a systemd service which waits for synchronisation.

You could check the output of timedatectl before your script gets time,

This will show something like:-

      Local time: Tue 2019-06-04 10:23:02 AEST
  Universal time: Tue 2019-06-04 00:23:02 UTC
        RTC time: n/a
       Time zone: Australia/Sydney (AEST, +1000)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

score 1 · Answer 2 · answered Jun 03 '19 at 23:45

1

You are aware the Pi has no built-in real time clock?

You either have to buy an RTC module for the Pi (DS3232 based preferred) or have to start your script after the Pi has got a valid time through NTP, from your Internet router for example.

answered Jun 03 '19 at 23:45

Janka

1,653
6
10

Ah, you should not wait for the slow and stupid Pi to update the time when he thinks fit You are the master, you should order the lazy and stupid Pi to update the clock first thing after booting up. Every time I boot up Rpi, I notice that the little clock icon at the top right corner of the GUI desktop is not right. I become angry and hardly hit/click the clock icon to order it to update the newtwork time IMMEDIATELY. Of course the slave listens to the master. – tlfong01 Jun 05 '19 at 13:45

score 0 · Answer 3 · edited Mar 07 '20 at 13:16

I had the same issue while trying to run some startup script on my CoreELEC box, I was trying get a time stamp and external IP straight after reboot. Time was incorrect and failed to get an external IP. I guessed that's due to the services not having started yet. I putted a "sleep 5" before running everything else and problems fixed!

Time stamp right after reboot

Box rebooted on 01-01-2015, 11:00:26 AM

Get time stamp again after 5 seconds

Box rebooted on 03-07-2020, 02:24:01 PM

score 0 · Answer 4 · answered Jun 05 '19 at 10:21

There are several ways to wait until the system clock is synchronized with time servers after boot up. You can prepend a check in a loop to your script that will test with timedatectl and sleep if the the clock is synchronized. If you do that it is important to run the script in the background as service until it terminates successful.

You can also try to start your script as a oneshot service and if it fails because the time isn't synchronized you can restart it again after a pause set in the systemd unit file for the service.

I have made a service that will look into the journal for a logged message that the clock is synchronized and then execute the script. How to do it you can look at How can I delay the startup of systemd services until the datetime is set (no RTC on the Raspberry Pi). The unit file there does not fit exactly your needs. Instead you can use this one:

[Unit]
Description=Archive wav files
After=time-sync.target

[Service]
Restart=on-failure
RestartSec=30
ExecStartPre=/bin/bash -c '/bin/journalctl -b -u systemd-timesyncd | /bin/grep -q "systemd-timesyncd.* Synchronized to time server"'
ExecStart=/home/pi/myscript.py
WorkingDirectory=/home/pi
User=pi

[Install]
WantedBy=multi-user.target

You have to change myscript.py and maybe the WorkingDirectory.

tlfong01 · Answer 5 · 2019-06-07T14:28:55.483

Question

Run [bash/python] automatically at startup, timestamp is wrong, ...

Answer

Or you can try the python datetime thing.

Update 2019jun05hkt2124

The OP says the following:

It works fine if I run the bash script manually through the command line, but if I allow it to run automatically at startup, the timestamps for files and archives are completely wrong. I'm not sure where to start with troubleshooting this issue.

I apologize that I suggested a solution but did not "troubleshoot" or explain why the OP gets the wrong time stamp.

The root cause is that linux does not update the user clock immediately after booting, and its reason is that it has too many higher priority (and less time consuming than getting network time) things to do (in the background) after booting. In other words, it is an engineering trade off and user experience/satisfaction vs system performance.

This is a quick and dirty update. Perhaps I can try a better explanation later.

Update 2019jun06hkt0857

Ideally, linux should keep a record (with the user's authorization) of what the unique user has been doing in the past. Say, if Rpi discovers the angry user, ie, me, hits the little real time clock icon angry (many times in 0.1 second), it knows I want network time update ASAP (of course also checking my Rpi is connected online every time, otherwise it is a waste of time).

Actually MS has been doing research in the past 20+ years for their UI/UE, to make their booting as fast as possible. They from time to time invite users to join their real time online survey observing and collecting statistics on what the gereral/average users do, immediately after booting, and after which job, etc, ... I always agree on their invitation as a volunteer, for the benefit of my user pleasure, and general user's pleasure. Below is an article I read yesterday. Of course you can easily google for many more such articles on UI/UE things. In the past 20+ years or so, I have been on alert on many enggr computing thing, including "Code Project Insider Developer News" where this boot user survey article is coming from.

Microsoft wants to know how you feel about Windows 10’s Start menu - Code Project Insider Developer News Darren Allan 2019jun06

Update 2019jun07hkt2226

pi-wake-on-rtc - bablokb 2018mar09

Introduction

This project implements a wake-on-rtc solution for the Raspberry Pi and other small computers based on the DS3231 real-time-clock.

Overview

The solution consists of two parts: a circuit doing the actual power switching and secondly some programs to control the rtc.

The circuit reacts to three events:

a push of the power-on button: this will switch power on and boot the pi

an alarm of the rtc: this will also turn power on

a gpio-interrupt from the pi after shutdown: this will switch off power

The software-part takes care to program the alarm of the rtc before shutdown, so the system will boot at the requested time.

This does not answer the question as to *why* the timestamp generated by the OP's script might be incorrect when run on startup but correct when run manually a little later on. — Roger Jones, Jun 05 '19 at 11:28
Ah, many thanks for pointing out my careless mistake. I focused on solving the problem of wrong timestamping, but did not pay attention why the system is so slow to update the system time. Actually I noticed the slow update for a couple of years. Every time I boot up to GUI desktop, I noticed that the system time shown at the top right corner is not updated to Inernet time. I do notice that the update will come "a bit later". I used to think that when booting up, the system has too many higher priority things to do, so updating the time is low priority. And user can update at once / ... — tlfong01, Jun 05 '19 at 12:42
/ continued from above. My Chinese Windows 10 has the same problem. When booting up, some of the things I wish to do immediately is not yet ready, I need to wait for perhaps 30 seconds or more. I have not complained, because I know what I want to do immediately is low priority for most other users. I think you are right that I missed a point., Let me see how I can improve my answer. — tlfong01, Jun 05 '19 at 12:44
One short comment. I browsed other answers and was very surprised to find suggestions like "check", "wait", or use an RTC, but no one seems to be proactive to "push". — tlfong01, Jun 05 '19 at 13:49
@Roger Jones Just now I updated my answer with an article reference on how MicroSoft's effort to improve their user booting satisfaction. Many thanks again for pointing out my careless mistake. Cheers. — tlfong01, Jun 07 '19 at 01:17
I seem to recall that there's a way to change the `systemd` boot up to wait for the network before going on. This is usually disabled so as to get to the GUI quicker but with some network services still starting up in the background. Not sure where the OPs script comes in that flow but might be a way to make sure that the network is up & NTP is running before trying to run the script. I might add that I think that adding an RTC is a proactive "push" as it gets you the correct system time much faster in the boot sequence than having to wait for the network. — Roger Jones, Jun 07 '19 at 07:43
Yes, I agree. There are three methods to get the correct time stamp: (1) change systemd to wait for network, but this is not proactive, (2) use python get network time function. I think this is protective. (3) Use I2C RTC DS3231. Actually I have been using RTC for one year. But the code is very messy and I have not documented it properly, so it is newbie scary. (I use DS3231 but not DS3232, because no 3232 module is available. So I am waiting for DS3232 module. I know we can use DS3231 DT overlay, but DTO is not OS version invariant, so I always refuse to use it. Just a short reply, .. — tlfong01, Jun 07 '19 at 08:47
My wild guess is that the OP is making an alarm clock which wakes him up every morning, say 7am sharp. The alarm clock uses WAV music or voice to wake him up. Actually I have tried DS3231 hardware interrupt to switch on/enable a low power standby LM2586/LM2741 voltage regulator to power on Rpi. You see, I can write my answer 10 times long, but StackExchane say we should be "laser sharp focus on the OP's topic. When I started answering one year ago, I always got the mods' "off topic" message, and my answer got "on hold", and then closed or removed, I frgt. I also hs a humn voice alrm clk. — tlfong01, Jun 07 '19 at 08:58

How do I fix incorrect timestamps in my bash script that launches on startup?

5 Answers5

Linked