How can I avoid wearing out flash memory by frequently changing symlinks?

Question

I have a circular buffer of files in ram disk. A new file is created every second. I want to always refer to the latest file using the same file name for access by a browser and apache. So, each time I update the file, I update a symlink from FIXED_FILE_NAME to VARIABLE_FILE_NAME. My concern is that such a frequent update of the file system wears out the flash memory that stores the system. How can I avoid this wear ? Thank you in advance for your help.

Answer 1

You can't, but it does not matter because it will take more than a decade to wear a card out that way. I've pointed out previously that there is much paranoia about this amongst pi users, perhaps because of reports of filesystem corruption which litter the internet, although these have nothing to do with wearing out cards. I don't think there is the same degree of hysteria amongst camera and smartphone users.

Anyway, let's unsuperstitiously go through the math. Say you have a cheap 4 GB card. It doesn't matter whether you allocate the whole thing or not, because SD cards use virtual addressing and wear leveling. Physically, the whole card is in play no matter what, even if you are only using a 1 GB filesystem.

Let's also say that cheap card has a very crappy lifespan -- 1000 write cycles. Then let's say it only takes half that time before enough failed blocks have been excluded by the card's built-in controller to make using it silly and splurging another $10, as painful as it seems, becomes worthwhile.

So in total, you can write 2 terabytes to the card. Now let's say that changing an inode is 1 kB. It certainly won't really be that much from a filesystem perspective, but we might as well increase it to 4 kB in case there's something about the SD card which requires changing entire blocks of a certain minimal size, such as 4 kB (this is probably the case). If all you use the SD card for is small writes, this means you could do this on average:

2000000000000 / 4000 = 500000000

500 million times. There are 86400 seconds in a day; at 365 days per year that's

500000000 / 31536000 seconds = 15 years

But of course that's not all that is going on with the card. Add on average another 100 MB per day writing to the card. Spread out, that's another ~1157 bytes per second; we don't need to compensate for the minimal block size here presuming the average write is more than 4 kB. Now we have:

2000000000000 / 5117 / 31536000 = 12 years

To be on the safe side, I suggest you replace the card every 2-3 years if you can afford the expense. If you are really strapped, try pushing it to 5-6. You should keep a backup of the system and, if it is crucial, have a spare card at hand anyway.

There is an additional factor in your favour, which is that the OS uses a page cache in free memory. Frequent changes will not all take place on disk, they will take place in the page cache and occasionally get synced to disk -- this is one reason filesystem corruption is possible in the first place, because if you suddenly kill the power when the system is busy, the cache is not completely synced with the disk at that moment (however, the risk of this really causing a problem is low). See also man sync.

Of course, if you reduce the amount of free memory by using a giant ram disk, you reduce the efficacy of the page cache. That may or may not be wise depending on context, but it likely it is not. Try to leave at least 50-100 MB free unless you do not care about the overall performance of the system.¹

Note that the pi, or some pis, seem to be prone to causing corruption even when using a read-only filesystem -- at least, we've had a few separate reports of that here. You cannot guard against this unknown problem, which may have to do with an old kernel bug (if you have a kernel built in the past year, don't worry), or it may be an electrical flaw in some pis.

---

^{1. Note there are two metrics common in reporting memory usage, the total amount of RAM used by userspace processes, and the total amount of RAM used. The difference between the two is mostly because of the page cache; if you look at the output of free, this is indicated on the second line, +/- buffers/cache. If the system has been running for long enough, particularly on a relatively low memory device like the pi, chances are the kernel will have filled all free RAM (as in, not used by userspace processes) with the page cache. This is why the metric that's relevant for most purposes is the first one, i.e., the memory used by userspace process, because the amount of "free" memory available for use by userspace processes (that is, all the programs you run) is the total amount minus the amount they are already using, and not including the page cache. Remember, stuff in the page cache isn't really in use, it's stuff that was in use and is cached in RAM in case something wants to use it again. If instead something needs more memory for some other purpose, chunks of the page cache are discarded to provide it (possibly being synced to disk first).}

^{So for user purposes, you can consider the page cache free RAM. Hence the second line of free. Here's a headless B+ that's been up for a few days:}

>free
             total       used       free     shared    buffers     cached
Mem:        448376     278064     170312       8416      45968     194780
-/+ buffers/cache:      37316     411060

^{On the first line, it looks like I've used 278 MB. But that includes the page cache. If I added up all the memory used by all my running processes, it wouldn't be anywhere near that amount. That's the number on the second line -- I've really only used 37.3 MB. However, some tools, including top, only report the number from the first line (however, top does report numbers for buffers and cached, which you can subtract from that).}

Answer 2

Create symlinks in two levels. When creating a new file, also create a symlink to it and put that symlink in the same directory as the temp file. Then create a second symlink for the webserver which points to the first symlink. The second symlink should never have to be rewritten, only recreate the first symlink.