6

I have a Raspberry Pi B+ installed on a local store and when power goes randomly off, SD card tends to get corrupted:

***Error in check_disks:could not repair filesystem, dropping to debug shell, try to run 'fsck' manually:***
### Starting debugging shell... Type exit to quit ###
Sh: can't access tty; job control turned off

This is easily repaired by manually running,

# fsck -fy /dev/mmcblk0p2 
# reboot

Is there a way to avoid this, and let Raspberry do that automatically on that case?

EDIT: It's on running OpenELEC 5.0.3

whitenoisedb
  • 367
  • 1
  • 5
  • 13

2 Answers2

3

Some people seem to have a lot of problems with corruption whereas others do not, which is very puzzling. E.g., I got my first pi more than two years ago, I now have 4, and at least one has been on that whole time. While I usually shut them down properly, I do regularly end up having to kill the power, there have been occasional outages, and I sometimes use a battery that I just let run out.

One of my theories about this is that some small percentage of pis are defective and screwing up cards due to under/over voltage either when power is removed or cut. There have been a number of people here reporting peristent SD card corruption even with a read-only filesystem (#1 -- #2 -- pretty sure there is at least one more I cannot find). That means it is not due to the power being cut with the filesystem out of sync; in other words, shutting down properly would probably not prevent this.

But I have a new theory, after trying to compile a 3.18.x kernel with a cross-compiler I built some time ago. It used gcc 4.8.1, which was released in 2013. Apparently about 7 months ago (looking at the git commits on the kernel), a little bit was added to an ARM specific file in the source such that when I tried to compile it, I got this error:

Your compiler is too buggy; it is known to miscompile kernels

The file this comes from contains a note:

 * GCC 4.8.0-4.8.2: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58854
 *            miscompiles find_get_entry(), and can result in EXT3 and EXT4
 *            filesystem corruption (possibly other FS too).
              ^^^^^^^^^^^^^^^^^^^^^ LOOK

I never had any problems, but one thing I can pretty much guarantee is that for some number of months a lot of raspberry pi distro kernels were built using gcc 4.8.0 - 4.8.2.

I'm sure they are not anymore, so if you have not upgraded your kernel recently, do. You can check the date your kernel was compiled with uname -v. If it was anytime before December 2014, this may be your problem.

goldilocks
  • 56,430
  • 17
  • 109
  • 217
  • wow, deep research goldilocks! are that kind of miscompilings unoticiable? maybe just warnings? I can't believe OpenELEC team would have missed that, but it could have happened. I'll check it. I'm not sure if I can upgrade my kernel in OpenELEC or if I should update my whole OpenELEC version to achieve that. – whitenoisedb May 12 '15 at 02:26
  • This rp b+ is running OpenELEC 5.0.2: `RaspTV2:~ # cat /proc/version` `Linux version 3.17.8 (stephan@buildserver.fritz.box) (gcc version 4.9.2 (GCC) ) #1 PREEMPT Mon Feb 9 00:14:40 CET 2015` – whitenoisedb May 12 '15 at 06:11
  • And today my rp2 had SD card corruption so I cannot access via SSH to print its version, but I guess it's running OpenELEC 5.0.3 so it may be similar output – whitenoisedb May 12 '15 at 06:15
  • Just a note: Last month, I've added a crontab to reboot all mondays at 8am (maybe it has no sense, but I though it could help on the overall performance) – whitenoisedb May 12 '15 at 06:21
  • That kernel should be okay (built w/ gcc 4.9.2). – goldilocks May 12 '15 at 08:01
  • yes... Do you think I can solve this problem by replacing the SD card storage with a pendrive? – whitenoisedb May 13 '15 at 00:17
  • It depends on why. If it is because of the OS falling out of sync with the hardware, it won't help -- in that case you have to make sure a fsck is done every boot. If it is because of some lower level problem (e.g. bad voltage), it might, except that no matter what with a pi, you are dependent on the SD card hardware. – goldilocks May 13 '15 at 00:55
  • oh.. I understand your point. I'm just freaking out because I have those two PIs running 24/7 on a local store, and every two weeks this issue happen. Now Pi2 went into a loop after doing `fsck` manually. I'm trying to think solutions knowing that this may not be ever fixed... – whitenoisedb May 13 '15 at 01:12
2

Is there a way to avoid this, and let Raspberry do that automatically on that case?

There's an easy way to do it generally (i.e., every boot) on Raspbian. Add this to /etc/rc.local:

echo "-fy" > /forcefsck

This will create that file with that content. This will be detected at boot and force a run of fsck -fy on the root filesystem. It's then erased, which is why it must be replaced via rc.local.

If the filesystem is fine, that will add about 10-20 seconds to your boot time, depending on the size of the filesystem.


That only works on SysV style init systems; an equivalent for systemd is documented here.

goldilocks
  • 56,430
  • 17
  • 109
  • 217
  • thanks goldilocks! I'll try it. I forgot to say I'm using OpenELEC, but that should work inside `autostart.sh` – whitenoisedb Apr 09 '15 at 18:22
  • 1
    In fact this is a feature of SysV init specifically, which it would seem openELEC doesn't use. There is some documentation around about using a kernel boot parameter, `fsck.mode=force`, but this is actually passed by the kernel to init, in this case systemd (SysV and systemd being two pretty different implementations of [init](http://en.wikipedia.org/wiki/Init). – goldilocks Apr 09 '15 at 18:53
  • ...I think openELEC probably uses `busybox`, which includes yet another init implementation. You might want to ask about this on openELEC's own forum or [Unix & Linux](http://unix.stackexchange.com/). I'm guessing simply putting `fsck` in `autostart.sh` will not work because that is too late in the boot process. – goldilocks Apr 09 '15 at 18:58
  • Yes...maybe autostart.sh runs after boot process. I've posted [here](http://openelec.tv/forum/124-raspberry-pi/61422-fsck-during-boot#137506) in the openELEC forum if someone helps. – whitenoisedb Apr 09 '15 at 19:56
  • ([just for reference](https://github.com/OpenELEC/OpenELEC.tv/blob/master/packages/sysutils/busybox/scripts/init#L443)) – whitenoisedb Apr 09 '15 at 22:55
  • So their init is a shell script...it can be. If you worked an answer out post it here and you can accept it in 24(?) hours. – goldilocks Apr 10 '15 at 02:52