1

Context

I have just found out the hard way that SD cards have a real life attainable write limit.

I use my Rasbperry Pi 4 (amongst other things) to scrape a website 24/7 with a frequency of 1 minute, which results in a ~1 MB file written every minute. I've done this for approximately 1.5 years, and now I notice the following issues:

  • the card always boots read-only
  • fsck tries to fix it and says it succeeded
  • card appears to be mounted read-write
  • after a reboot all information is gone and nothing is written to the card
  • Running fsck from an external system on the card shows "unable to set superblock flags"

From the internet I learned these are all signs of a hardware issue, and that the SD card prevents writes after it has reached a certain number of write cycles. The card I used is a 32GB SanDisk Extreme Pro.

Question

I want to keep using my Raspberry Pi for the same, write intensive, purpose. What can I do, in terms of storage selection or configuration, to extend the writeable life of my Raspberry storage significantly?

What I've looked into

  • I found this question: How can I extend the life of my SD card?. However, it focusses on how you can reduce writes to extend the life of the storage medium. I have no intention of reducing writes (as that is the core of the application).
  • I found this post about moving the filesystem to an external USB SSD. It's unclear to me if this would change anything as:
    1. External SSD's are also flash memory (right?) and might have the same write limitations.
    2. It's not clear to me if the scraped files would still "go through" the SD card when going to the USB mounted storage.
Saaru Lindestøkke
  • 111
  • 2
  • 1
    *"External SSD's are also flash memory (right?)"* -> There's flash memory and then there is flash memory. If you've been reading about this, then you know the issue **write cycles**, and the usual general principal of "pay more, get more" holds. Flash memory with a short life cycle might only be good for a few thousand cycles, while higher quality stuff might be good for hundreds of thousands. Also, SSDs have (more better) circuitry to identify and isolate bad cells. – goldilocks Jul 20 '21 at 12:13
  • @goldilocks So... the answer is get an external SSD and simply follow the post about moving filesystems? Also, "higher quality stuff" is hard to gauge. I thought my Sandisk SD card was higher quality (based on price and reviews), but it appears that's not true. – Saaru Lindestøkke Jul 20 '21 at 12:17
  • I don't think it is possible to put the kind of memory into an SD card that is used in a good quality SSD (but I could be wrong). Anyway, while it is hard to get stats about write cycles for a make/model of SD card, it is usually explicit in the specs for an SSD. So you are writing ~525 GB/year; a 250 GB SSD should be able to do that for a really long time. And you should be able to get metrics from it to tell you when it is wearing out. – goldilocks Jul 20 '21 at 12:24

2 Answers2

4

SSDs have wear levelling technology and quote their expected life usually having at least 60Tb of writes before failure. If my calculations are correct and using your 1Mb per minute estimate that should give you around 120 years before failure.

Bra1n
  • 1,181
  • 6
  • 7
  • 1
    So would you suggest I move my filesystem to an external SSD? Will this really prevent the data to go "through" the SD card? – Saaru Lindestøkke Jul 20 '21 at 08:20
  • I would boot from the ssd and dispense with the sdcard alltogether. Which model of pi are you using ? – Bra1n Jul 20 '21 at 11:14
  • I was not aware that I could fully get rid of the sdcard. I have a raspberry pi 4B with 8GB of RAM. – Saaru Lindestøkke Jul 20 '21 at 11:58
  • 1
    The data will not go "though the SD card" unless you do it intentionally (or naively, eg. if your web server uses it for some sort of quick caching). Data from a network or produced by an application is first placed into RAM -- that's not avoidable. Where it goes after that, if anywhere, is up to the application. If you are saving things to a partition on an SSD, then that's where it will be written to. The OS doesn't secretly copy it to the SD card in between. There is a bit of a caveat here to do with the relationship between RAM and swap, but it still won't amount to that. – goldilocks Jul 20 '21 at 12:18
  • @goldilocks Maybe this is becoming a new question, but say my python interpreter lives on the SD card, but my script lives on the SSD and is directed to write to the SSD, will data pass through the SD card? – Saaru Lindestøkke Jul 20 '21 at 15:11
  • No. Why would it? – goldilocks Jul 20 '21 at 15:49
  • Booting from the SSD is straightforward I run all my Pi 4s this way, just make sure you use a compatible USB3 to SSD adaptor as a bonus you get much better performance and boot times – Bra1n Jul 20 '21 at 17:03
0

There are more urban myths about the Pi and SD Cards than fact.

It is true that they can and do wear out (I have 2 failures in 9 years) but the risk is much over-rated.

If a card does become read-only, this is often a sign that it has worn out. In this rare case it is unrepairable, and the SD Card firmware does this to protect existing data.

Many of the suggestions about extending SD Card life are futile.

There are a few things you can do:-

  1. Use quality SD Card (I currently use SanDisk® High Endurance microSD Card.
  2. Use larger cards; these spread the load and require less frequent block erase.
  3. Backup your system regularly, preferably using a file based backup. Backup image of SD Card
  4. Rotate between backup and cards freshly imaged from backup. (This keeps files defragmented and minimises the need for blocks to be re-written.)

Without knowledge of what you are doing it is difficult to be specific, but there are means of internally buffering data and only writing actual changes (but this is not Pi specific).

You could use external storage; a NAS or disk mounted on the Pi.

NOTE moving the OS to external storage (at least on the Pi4) is likely to reduce performance, but putting your data on an external device is OK for a low demand situation ~1 MB every minute is OK.

Milliways
  • 54,718
  • 26
  • 92
  • 182