99

This question answers the question of how I use an external computer to create a backup of my RPi.

I'm wondering whether I can create a backup image of the SD card that is currently in use, and copying it to a file on a USB storage device. Is this possible? If not, is there any way to create a backup of a RPi without involving another computer?

Eric Wilson
  • 1,695
  • 3
  • 15
  • 12
  • 2
    Sure, but skip /tmp, /run, /proc, /sys, /dev, and /mnt. You don't need to create an image, you need a backup you can create or update an image from. So don't use `dd`, look into `rsync`. – goldilocks Mar 14 '13 at 02:13
  • 2
    @goldilocks I would love it if you would flesh this comment out into a more complete answer, explaining the backup & restore process you have in mind. – Eric Wilson Mar 14 '13 at 09:31
  • Done -- sorry it took me a few days to find the time. – goldilocks Mar 17 '13 at 15:33
  • 1
    If your destination volume is big enough, remounting the filesystem read-only and doing a `dd` copy with an appropriate block size will probably be fastest for a "new" copy. Doing a file-by-file copy to flash/SD media is probably a bad idea. – Chris Stratton Dec 11 '15 at 01:41

9 Answers9

106

Here's an intro to using rsync for back-up on the Pi. Once the initial back-up is created, keeping it up to date this way is much much faster than constantly ripping the entire image. You can do this to a local hard drive or over a network.

You actually do not want a complete copy of a running system as a back-up, since some of the stuff ostensibly in the filesystem exists only at runtime. Including that in a backup and then using it to recreate an image later may create problems for you.

There are some other exceptions too. rsync can accept a list of (glob) patterns to exclude, and those can be read from a file, so let's first go thru what should be in such a file. Note that the entries are of the form /directory/* and not /directory. This is because we want them to exist, but we don't want to copy anything in them.

/proc/*
/sys/*

These do not really exist on disk. They're an interface to the kernel, which creates and maintains them in memory. If you copy these and then copy them back into a system and boot it, it will be (at best) meaningless, since the kernel uses those as mount points for the interfaces [If you want to see what happens when you mount a filesystem partition on a directory with data in it, try. It works and won't do any harm, but the stuff that was in the directory is now inaccessible.]

Note that it is critical that the /sys and /proc mount points exist. But they should not contain anything. Next:

/dev/*

The dev directory is not quite the same thing as proc and sys but for our purposes it is. If you believe that you should save this so you can have the same device nodes in your backup or something, you're wrong. Don't bother. Do not copy dev. Once upon a long time ago Linux did work that way, but it doesn't anymore.

/boot/*

This is sort of a special case with the most (perhaps all) of the Pi specific distros such as Raspbian. It's actually a mount point for the first, vfat, partition. We are going to deal with that separately. Whatever you do, don't bother including it here, because again, it's a mount point.

/tmp/*
/run/*

/run is generally not on disk either, it's in memory. Perhaps /tmp could be too (this would save a bit of SD card action), but in any case, as the names imply, these are not places for storing persistent data. Applications which use them expect that they may be deleted at each boot.

/mnt/*
/media/*

These are important particularly if you are planning to back up to a hard drive or USB stick and the device is in /mnt or /media (automounting tends to use the latter), because if you don't exclude the location of those devices in the filesystem you will create a loop backing up the content of the drive to itself, until it runs out of space. I think rsync might be smart enough to spot something that dumb but try to avoid testing the premise.

On to the actual backing up: Create a directory to back up to on the locally mounted harddrive, USB thing, etc. -- e.g. "pi_backup". You can alternately backup to a remote location via ssh (see below) or using a network mounted filesystem, but this will probably take a while the first time.

If the file containing the list to exclude is /rsync-exclude.txt1 and your drive is /mnt/usbhd, to do the actual backup:

rsync -aHlAXNv --delete --exclude-from=/rsync-exclude.txt / /mnt/usbhd/pi_backup/

Notice that there is a trailing slash on pi_backup/. You may want to have a look at what all the switches mean in man rsync.

This will take a while and produce a lot of output (if you want to examine that in a log instead, append > rsync.log). --delete is meaningless the first time, but for keeping the backup updated use it. This ensures that stuff you've later deleted on the Pi also gets removed from your backup. The a sets recursion into directories and makes sure all the file attributes match. -H is to preserve hard links2, v is for verbose which is why you get some output (otherwise rsync is quiet). See man rsync for more.

There is a shortcut whereby you can skip the --exclude-from file. If you are sure that all of the things you don't want to copy (/tmp etc.) are on separate filesystems, you can just use:

rsync -axHlAXNv --delete-during / /mnt/usbhd/pi_backup/

-x has been inserted. This is the short form of --one-file-system, which tells rsync not to cross filesystem boundaries. Personally I prefer the --exclude-from, but on e.g., default Raspbian, --one-file-system will work fine. You can use both if you want to be -xtra careful :D

That's not quite a complete backup. It's enough if you haven't put anything in boot and you are fine with using the back up to just restore the system by sticking the card in a computer and running:

rsync -av --delete-during /mnt/usbhd/pi_backup/ /mnt/sdcard_partition2/

You could also do this with a card with a new image on it (presuming it's the same as your base image) although that's a little inefficient if you have to create the image (because you're then going to overwrite most of it). You could also connect another SD card via a USB adapter with such an image on it, and use the above method to maintain a duplicate card.

If you've put stuff in /boot (e.g., a custom kernel), including /boot/config.txt, you'll want to back that up too (pretty simple -- there's not much to it). Just do it separately, and when you restore, that stuff goes in the first partition.

See here if you want to create a blank Raspbian style image which you could then backup into. You can use a similar methodology to create an empty Raspbian style card -- just rather than dealing with an .img file, you'd be dealing with a real device (e.g. /dev/sdb), meaning all you have to do is create the partition table with fdisk and then format /dev/sdb1 and sdb2 (or whatever) with mkfs.

But copying the whole image is easier! Why bother with this?

It's not that hard; I restored to a blank card (formatted as per that last link) in 10 minutes. Yes, just using dd on the whole thing is simpler (if you find stuff like words confusing...), BUT then it takes quite a while every time you want to update your backup because you must do 100% of it every time. Using rsync, once a backup exists, updating it is much much faster, so you can set this up to happen painlessly everyday via cron. Over a network even. Every six hours. The more often you do it, the less time it will take.

rsync via ssh

Here's an example:

rsync [options] --rsh="ssh [ssh options]" root@[the pi ip]:/ /backup/rpi/

"Options" would be, eg, -av --delete --exclude-from=/rsync-exclude.txt and "ssh options" is whatever you normally use (if anything). You must have root access via ssh to do this for the purposes of a system backup (set PermitRootLogin=yes in /etc/ssh/sshd_config and restart the server).


1 You should keep this file. You can put comments in it on lines beginning with # or ;. This could include the actual rsync command, which can be copy pasted later so you don't have to remember it each time.

2 Thanks to Kris for pointing out rsync doesn't do this automatically.

goldilocks
  • 56,430
  • 17
  • 109
  • 217
  • 1
    Goldilocks. That looks like a great use of rysync. Any chance of rolling it into a script for us? – totalitarian Aug 27 '13 at 07:47
  • Instead of manually excluding all mountpoints, why not `mkdir /tmp/backupable && mount --bind / /tmp/backupable` and rsync that? That also has the advantage of backing up any data stored in places that are "shadowed" by something mounted there. – n.st Dec 18 '14 at 05:11
  • @n.st Good idea (lol)! I've edited the suggestion into the question, although I still think using `--exclude-from` is a better idea. If you have time, you could write this up as a separate answer, you have my vote, and I can reference that. This answer is long winded enough. – goldilocks Dec 18 '14 at 21:09
  • 1
    Great answer. I've used it setup my rpi automated backup and it works great. I noticed though that when I restored my 'clone' it actually took up more space on the card than the original. So I wonder if it maybe a good idea to include the -H option in the rsynch command. (rsynch manual says -a doesn't imply -H). The -H options tries to preserve 'hardlinks'. – Kris Feb 01 '15 at 20:17
  • @Kris Sorry for the tardy reply. You're right -- I did not realize rsync didn't preserve hardlinks (why wouldn't it?). Anyway, I've incorporated `-H`. – goldilocks Apr 17 '15 at 11:06
  • Hi All, I was trying to follow this but for hidden files starting with `.` doesnt get copy and gives error for example `rsync: chown "/media/pi/VIKAS/pi_backup_29_Aug_2016/home/kodi/.kodi/addons/metadata.tvdb.com/resources/language/Malay/.strings.po.2MezKR" failed: Operation not permitted (1)` – Ciasto piekarz Aug 29 '16 at 08:59
  • this solved my problem `shopt -s dotglob` – Ciasto piekarz Aug 29 '16 at 09:17
  • I'm trying to understand - does your above command create an **.img** file? If not, then how can we do that so that we can easily write the **.img** file to an SD card. – IgorGanapolsky Jan 25 '17 at 15:13
  • 2
    @IgorGanapolsky The intention is to *not* create an image (read the **"But copying the whole image is easier! Why bother with this?"** part). In addition to being easier and faster to maintain once created, this method is generally more flexible. If you want to use it later to create an `.img` you can; [this](http://raspberrypi.stackexchange.com/a/13138/5538) and [this](http://raspberrypi.stackexchange.com/a/29952/5538) should help to explain how they are structured and can be created. – goldilocks Jan 25 '17 at 15:23
  • @goldilocks So once I have these copies backed up into a **pi_backup** dir, like your script suggests, how do I actually use this new directory to restore my Raspberry Pi when it crashes? I am a bit confused, since I am using an external SSD to make your script write these files into a **pi_backup** directory. – IgorGanapolsky Jan 25 '17 at 15:33
  • 1
    See the paragraph that begins, *"That's not quite a complete backup..."*. It's basically the exact same thing in reverse. [This may help](http://raspberrypi.stackexchange.com/a/51100/5538) with some concepts that people are commonly confused by/about. – goldilocks Jan 25 '17 at 15:38
  • shouldn't I insert the SD card on my laptop? I am not sure how I should execute the above command – dearn44 Feb 01 '18 at 13:22
  • @dearn44 You can do it that way if you have `rsync` on your laptop, and it can read ext4 filesystems. You don't have to do it that way though, you can do it from a running system as long as you make sure to use the "exclude from" list. And that you are not backing up to a location that's being backed up. Or on the SD card, since there probably will not be room. – goldilocks Feb 01 '18 at 14:20
  • Is it possible to restore the backup on the running Raspberry? This would be faster, e.g. in case you broke something and don't want to put the SD Card into another PC to do the restoring. – wewa Apr 26 '20 at 07:19
  • It's a bad idea and I would bet against it working in many or most cases (if barely anything is different, obviously it could be done). And what you are risking is being left with a completely unusable card that has to be properly restored -- so it might be worth a try depending on the circumstances, eg., if you think the system is not usuable for you anyway. – goldilocks Apr 26 '20 at 13:20
  • I have started using rsync to backup my data on the Pi. It works great for documents, images, scripts, etc. However, I am struggling using this approach to create a duplicate card using "a card with a new image on it (presuming it's the same as your base image) .. and use the above method to maintain a duplicate card." The Pi did not boot up successfully from the duplicate card thus created. I followed the instructions exactly, even using an SD adapter as recommended. I will post more shortly. – Sun Bee May 07 '20 at 04:16
  • If you restoring to a card created from a new image, note that the contents of the boot partition may then not match -- the major issue there is if the kernel doesn't have an appropriate `/lib/modules/x.x.x`. It should still boot in that case though, just some things will not work. – goldilocks May 07 '20 at 15:32
  • Thank your for this description, it helps me a lot. Maybe it would be a good idea to mention that also `/boot` needs to be backed up because otherwise there may be errors if kernel is updated (like we worked out here: https://raspberrypi.stackexchange.com/questions/112075/failed-to-start-load-kernel-modules-after-rsync-restore). – wewa May 07 '20 at 18:05
  • See the paragraph above that begins, *"If you've put stuff in /boot (e.g., a custom kernel)..."*, and the part before that referencing the `/boot/*` ("This is sort of a special case..."). Do not include it in the regular backup unless you are only going to use it to restore a running system with the first partition mounted. – goldilocks May 07 '20 at 18:33
  • ...okay I see in your other Q&A you have got that. – goldilocks May 07 '20 at 18:35
26

A working script from the Raspberry Community made by a member there.

You can reuse and tweak the code how ever you like.It is well documented and self explanatory.

#!/bin/bash

# Setting up directories
SUBDIR=raspberrypi_backups
DIR=/hdd/$SUBDIR

echo "Starting RaspberryPI backup process!"

# First check if pv package is installed, if not, install it first
PACKAGESTATUS=`dpkg -s pv | grep Status`;

if [[ $PACKAGESTATUS == S* ]]
   then
      echo "Package 'pv' is installed."
   else
      echo "Package 'pv' is NOT installed."
      echo "Installing package 'pv'. Please wait..."
      apt-get -y install pv
fi

# Check if backup directory exists
if [ ! -d "$DIR" ];
   then
      echo "Backup directory $DIR doesn't exist, creating it now!"
      mkdir $DIR
fi

# Create a filename with datestamp for our current backup (without .img suffix)
OFILE="$DIR/backup_$(date +%Y%m%d_%H%M%S)"

# Create final filename, with suffix
OFILEFINAL=$OFILE.img

# First sync disks
sync; sync

# Shut down some services before starting backup process
echo "Stopping some services before backup."
service apache2 stop
service mysql stop
service cron stop

# Begin the backup process, should take about 1 hour from 8Gb SD card to HDD
echo "Backing up SD card to USB HDD."
echo "This will take some time depending on your SD card size and read performance. Please wait..."
SDSIZE=`blockdev --getsize64 /dev/mmcblk0`;
pv -tpreb /dev/mmcblk0 -s $SDSIZE | dd of=$OFILE bs=1M conv=sync,noerror iflag=fullblock

# Wait for DD to finish and catch result
RESULT=$?

# Start services again that where shutdown before backup process
echo "Start the stopped services again."
service apache2 start
service mysql start
service cron start

# If command has completed successfully, delete previous backups and exit
if [ $RESULT = 0 ];
   then
      echo "Successful backup, previous backup files will be deleted."
      rm -f $DIR/backup_*.tar.gz
      mv $OFILE $OFILEFINAL
      echo "Backup is being tarred. Please wait..."
      tar zcf $OFILEFINAL.tar.gz $OFILEFINAL
      rm -rf $OFILEFINAL
      echo "RaspberryPI backup process completed! FILE: $OFILEFINAL.tar.gz"
      exit 0
# Else remove attempted backup file
   else
      echo "Backup failed! Previous backup files untouched."
      echo "Please check there is sufficient space on the HDD."
      rm -f $OFILE
      echo "RaspberryPI backup process failed!"
      exit 1
fi

Consider adding comments to the original forum or post your own version to help mature the content. Take a little give a little.

*And thank you for giving back AndersW (Click for GIT script)

goldilocks
  • 56,430
  • 17
  • 109
  • 217
Piotr Kula
  • 17,168
  • 6
  • 63
  • 103
  • 2
    What if the filesystem (file deletion, new files added) changes in the time while the pi is backuping? – keiki Mar 14 '13 at 13:47
  • (dd)[http://linux.about.com/od/commands/l/blcmdl1_dd.htm] - Is a file level copier. If something changes after it has been copied it will be missed. That is why its important to take the Pi offline. The script stops mysql,apache, cron but possibly you need to stop samba,ftp and anything else. Stop low privilege users from login into SSH and only re enable everything later. This is the only downfall to this script it takes long and has to disable allot of stuff. If you dont want that you need to look at snapshots. – Piotr Kula Mar 14 '13 at 14:26
  • 2
    I back up several disks while they are running with rsync, and I have frequently been able to get exactly what I need from these file backups. However, in general, a unix file system can not be copied perfectly (with every bit in place and correct) while the file system is mounted (*). A copy made while the system was mounted is sometimes called a "dirty copy". Several measures can be taken to improve the quality of a dirty copy (as the script above does, shutting off cron and mysql) but it can't be perfect. Cheers! * -- I am wrong about this, it depends on the file system. – Tai Viinikka Mar 14 '13 at 14:28
  • 1
    [You can look at the Debian recommended backup utilities](http://wiki.debian.org/BackupAndRecovery) and see if Pi has a port of them. `rsnapshot` sounds promoising – Piotr Kula Mar 14 '13 at 14:29
  • 1
    @TaiViinikka You don't need a perfect copy. You need a *partial* copy that can be (quickly and easily) re-imposed on the original base image. `rsync` is the way to go; when I have time tomorrow I'll add an answer. `rsnapshot` is also worth investigating. – goldilocks Mar 14 '13 at 18:12
  • 3
    Based on ppumkins answer above, I synced up the 'dd' script with the latest comments in the original thread and added some minor improvements myself. The end result is available here: . Please don't hesitate to add improvements and sending me pull requests. – AndersW Sep 04 '13 at 21:36
  • 1
    Ideas about how to actually *use* the backup file produced are here: [Restore of Raspberry Backup](http://raspberrypi.stackexchange.com/questions/13129/restore-of-raspberry-backup/13139). – goldilocks Jan 21 '14 at 14:56
  • 1
    I have a 16GB SD card but the system uses only about 5GB and the gzipped image is over 11GB instead of 5 or less. I shrinked the second partition to 6GB and added a script for shrinking the image. This way the backup file is about 3GB: SECTORSIZE=`blockdev --getss /dev/mmcblk0`; ENDSECTOR=`fdisk -l $OFILEFINAL | tail -n 2 | awk '{print $3}'`; truncate --size=$[$SECTORSIZE * ($ENDSECTOR +1)] $OFILEFINAL – pistipanko Dec 25 '16 at 15:17
14

I have adapted @goldilocks answer on rsync for back-up on the pi. I backup to an ext4 partition on a HDD mounted on the Pi. If the HDD is not mounted, rsync would copy to the mount directory (until the SD Card is full). If the HDD is not mounted in rw mode copious error messages are produced. Neither of these is desirable, so I check that my partition is is mounted in rw mode before proceeding.

NOTE 2015-03-03 I modified my answer to accurately copy hardlinks. The original worked, but converted many hardlinks into files. In addition to wasting space, this compromises many uses which assume the hardlinks are in place. (My current image has 869 links, many in Raspbian itself.)

My script to do this follows. (My partition is PiData, mounted on /mnt/PiData

#!/bin/bash
# script to synchronise Pi files to backup
BACKUP_MOUNTED=$(mount | awk '/PiData/ {print $6}' | grep "rw")
if [ $BACKUP_MOUNTED ]; then
    echo $BACKUP_MOUNTED
    echo "Commencing Backup"
    rsync -avH --delete-during --delete-excluded --exclude-from=/usr/bin/rsync-exclude.txt / /mnt/PiData/PiBackup/
else
    echo "Backup drive not available or not writable"
fi

Restore (or update another Pi) with the following:-

sudo rsync -avH /mnt/PiData/PiBackup/ /

I have enhanced the rsync-exclude.txt to eliminate unnecessary files.

The first group are the directories documented by @goldilocks https://raspberrypi.stackexchange.com/users/5538/

The second group are the files and directories created by OS X when I access my Pi using AFP (Apple Filing Protocol). (These are normally invisible on OS X, but not on Raspbian. In any event, there is no need to backup.) Even if you never use AFP, these will do no harm.

The third group are files which do not need to be backed up (and certainly not copied to another Pi). Examples fake-hwclock.data, RPi-Monitor reports. You will probably have others.

/proc/*
/sys/*
/dev/*
/boot/*
/tmp/*
/run/*
/mnt/*

.Trashes
._.Trashes
.fseventsd
.Spotlight-V100
.DS_Store
.AppleDesktop
.AppleDB
Network Trash Folder
Temporary Items

.bash_history
/etc/fake-hwclock.data
/var/lib/rpimonitor/stat/
Milliways
  • 54,718
  • 26
  • 92
  • 182
  • 1
    Is there a way to make that output an **.img** file? – IgorGanapolsky Jan 25 '17 at 15:24
  • @IgorGanapolsky Well, seeing as all of the essential files are there (excepting boot files), it is obviously possible, but if you want an image make an image. You should ask any new question in a new post, not comments. – Milliways Jan 26 '17 at 03:42
  • @Milliways why not should we use "sudo rsync ..." ? There will be some files that may can't be synchonized ? – Smilia Nov 04 '18 at 09:45
7

It's now 2022; the original question here, and some of the answers, are about 9 years old now. I'm adding this answer to provide an update, and to connect this question, and its answers, with a more recent question asked (and answered) here. I've also included a section on Restoration, which is of course the end objective; the Backup is only the means to that end.

Backup:

The image-backup script provides an answer to both questions:

  • RPi can make a backup of itself while running,
  • the backup is contained in an image file (.img) for quick and easy restoration,
  • the image file does not contain unused space, and typically restores to a smaller SD card,
  • the image file is created quickly, (ref. details)
  • the script is extremely easy-to-use , prompting for all required options
  • the image file contains the entire SD card, incl. the /boot partition
  • the backup image file may be updated to maintain a current snapshot of your system
  • the script uses rsync, and therefore avoids most potential issues associated with dd in creating image backups
  • all of the rsync features may be applied to the backup: links, include &/or exclude files & directories, etc.

The image-backup script is part of a set of scripts called image-utils that was posted under the heading of Image File Utilities created by "RonR". You will find a link to download a .zip file there containing all the scripts. The author (RonR) provides technical support through his forum postings.

For those who prefer git, I've forked a GitHub repo of image-backup. I'm making an effort to keep it current with a small script I've written. If you'd like to use the git repo to download direct to your RPi, here's a recipe for that:

  1. If you haven't installed git, you'll need to do so from your RPi CLI:

    $ sudo apt install git
    
  2. Clone the GitHub repo of image-utils:

    $ cd
    $ git clone https://github.com/seamusdemora/RonR-RPi-image-utils
    
  3. Make the scripts executable:

    You now have a folder which is also a local git repo: ~/RonR-RPi-image-utils. And you have a decision to make: preserve your local git repo, or use it as your "working" folder.

    The simplest thing to do is use this folder as a working folder; only slightly more complicated is preserving your local repo & creating a working folder. Here's how I preserve my local repo; you can simply apply the chmod against your repo folder to use it as your working folder:

    $ mkdir ~/imgbkup && cp ~/RonR-RPi-image-utils/image-* ~/imgbkup/
    $ chmod 755 ~/imgbkup/image-*
    

    As an alternative, you could install the script files in /usr/local/sbin & make then executable in a single step:

    $ sudo install --mode=755 --group=pi --owner=pi /home/pi/RonR-RPi-image-utils/image-* /usr/local/sbin/
    # see 'man install' for details
    

    This (usr/local/sbin location) has the advantage of being on your (user pi) PATH; refs. Also note that (e.g.) image-backup must still be run w/ sudo - this is a requirement imposed in the script - not by permissions or ownership.

Restoration:

We create and maintain backups only because they enable us to restore our system to a prior state. Perhaps that prior state is one that existed before our SD card failed, or more frequently, a state that existed before an operator error occurred which accidentally deleted the ~/.ssh folder in our home directory, or /etc, /boot/config.txt, etc, etc. Mistakes do happen, and maintaining a current backup is the key to quickly restoring our system to its pre-disaster machine state.

In choosing a backup solution, one must consider not only the accuracy and efficiency of the backup process, but equally important is the question: What is the restoration process?

The following attempts to answer that question under two scenarios:

Complete system restoration:

This is the simplest scenario as it involves only writing the complete .img backup file to a new SD card, removing the old SD card, inserting the new one in its place and re-booting. The entire restoration process may require only a few minutes. If the backup .img file is up-to-date there should be little or no recovery time required to update configuration files, re-install packages, etc. The key is having a current image.

Partial restoration:

But perhaps you've made an error performing some administrative task; for example, I recently deleted my ~/.ssh folder... I can't replace that! Fortunately, this was also an easy restortion - though involving a bit more work than a full restore:

You can mount an .img file as a "loop device", effectively treating it as another block device - just as you do with your SD, HDD or USB. Only the mounting syntax is different. Without belaboring the point, here's how that might go:

  1. In this case, I was able to restore my RPi while still connected via SSH. The balance of this example is based on that scenario.

  2. However, the same basic procedure applies if you've lost access to your RPi; the difference being that you'll need another Linux machine - a restoration system. The restoration system can be any system that can mount ext4 & FAT filesystems; e.g. another RPi, Ubuntu laptop, etc. The restoration system must have access to the SD card of the "broken" system, and the *.img file created by image-backup.

  3. From the damaged RPi: Mount the *.img file (20220530_Pi4B_imagebackup.img) as follows:

$ sudo losetup -Pf 20220530_Pi4B_imagebackup.img
  1. Find the loop device(s) used:
$ ls -l /dev | grep loop
brw-rw---- 1 root disk      7,   0 Jun 18 04:32 loop0
brw-rw---- 1 root disk    259,   0 Jun 18 04:32 loop0p1
brw-rw---- 1 root disk    259,   1 Jun 18 04:32 loop0p2
... 
  1. Create mount point(s) for the partitions you need (i.e. /, /boot, or both) & mount the loop device/backup image:
$ sudo mkdir /mnt/part1 /mnt/part2     # create mount point
$ sudo mount /dev/loop0p2 /mnt/part2   # mount partition 2
$ sudo mount /dev/loop0p1 /mnt/part1   # mount partition 1
  1. At this point you can browse through the backup image file (20220530_Pi4B_imagebackup.img) partitions, locate the ~/.ssh folder, and verify the contents if you wish.
$ cd /mnt/part2/home/pi
$ ls -lR .ssh
  1. And finally - Perform the restoration:, and you are back in business:
$ cp -ipr .ssh /home/pi/.ssh

Partial restoration using a restoration system

  1. If you are unable to connect to your damaged RPi, you will need a restoration system to perform the restoration. As explained above, the procedure is similar. The only real difference is that in addition to mounting the backup *.img file, you will also need to mount the SD card from the damaged RPi. You will restore from the loop-mounted *.img file to the SD card.

  2. The SD card from the damaged RPi may be mounted via any available USB port using a USB-to-SD card adapter similar to the one shown below. Once the damaged/missing files & folders are copied/rsync'd from the loop-mounted backup to the SD card, simply re-insert the SD card in the RPi and boot.

SD-USB adapter

Seamus
  • 18,728
  • 2
  • 27
  • 57
7

I have three Pis running in my local net and need to backup them on a regular base with cron when they are up and running. That's why I created a script which is able to create dd, tar and rsync backups and to restore them. I prefer to use rsync for my backups but other folks prefer dd or tar. It's used by a lot of people already. Hope it's useful for others also :-) raspibackup - Raspberry creates backups of itself

framp
  • 872
  • 7
  • 15
  • 1
    Nope, sorry: asking user to run (as root!) a script downloaded over HTTP is irresponsible. Please distribute this script over a safe channel. – Clément May 01 '16 at 20:38
  • I see your point. But please note: Initially the user had to download raspiBackup and the sample config file manually and to copy them in /usr/local/bin and /usr/local/etc (That's why root access is required). But then I was asked by users to provide an installation script for their convenience. The instructions how to install everything manually are still there (actually the last installation option). If you want to discuss this issue in more detail feel free to comment on the webpage. I don't think this is the right place for this kind of discussion because it's offtopic. – framp May 07 '16 at 07:36
  • 1
    I don't think it's off topic, and root or not doesn't matter much. The point is that software should be distributed over a secure channel, and your answer is encouraging bad security practices. – Clément May 07 '16 at 21:44
  • How do I have to publish the software so you will feel confident and safe and download it? Just use HTTPS instead of HTTP? – framp May 08 '16 at 11:01
  • 1
    That would be a great step forward, yes :) – Clément May 09 '16 at 03:06
  • 1
    I accept your proposal. The Website is now ssl enabled and the installation script uses https. Unfortunately there is still an open issue with the http->https redirection which I will sort out the next couple of days. If you use [this link](https://www.linux-tips-and-tricks.de/en/backup) you'll be safe. – framp May 09 '16 at 17:24
  • 2
    Just to note that delivery over HTTPS does not in any way add any security in this instance! You are still downloading and running a script from the Internet. The secure process is to download the script (http/https is irrelevant), open the script in an editor and read it top to bottom, check it for oddities and insecurities. Only when you are satisfied should you run it. Framp might be a hacker for all any of us know and delivery over https would only make him smile in that case :) (BTW, that isn't an accusation Framp!) – Julian Knight Jun 15 '16 at 15:02
  • 2
    I agree with you. That's why there are two ways described how to install the script: 1. Use the installerScript 2. Download it manually, check the code and then install it manually – framp Jun 16 '16 at 17:40
  • @JulianKnight Running `apt-get upgrade` is just as insecure: you're telling `apt` to download a bunch of executables from the Internet and run them. Would you also say that the secure process is to download all packages manually, read them from top to bottom and check for insecurities? – Dmitry Grigoryev Jan 09 '19 at 15:45
  • Dimitry, any code repository except one that you control tightly has risks. The larger the number of entries, the higher the risk (in general). If you want a high-security system, you have to validate everything. We ALL take risks like this every day & in general, there are enough eyes on the common repos to keep problems to a min. Not always though & there are plenty of well documented examples of repos being compromised. – Julian Knight Jan 21 '19 at 21:24
3

Here is our stable tool for such purposes: https://github.com/aktos-io/aktos-dcs-tools

This tool is written to make ssh connections, make backup-root, make mount-root from remote places in mind at first, and then local sessions are added. So it supports local backups, direct remote backups, proxy remote backups. Backups are taken incrementally (only diffs are transferred) and backup directories are standalone (just pick a directory/version to restore, any directory has a full backup). Of course, you have versions (backup.last-0 is the newest one). You can interrupt backup process at anytime and continue later.

Here are the instructions for your specific problem:

 ssh to-your-raspberry
 cd /mnt/usb0/my-rpi-backups
 git clone https://github.com/ceremcem/aktos-dcs-tools backup-tools
 ln -s backup-tools/Makefile .

 ./backup-tools/configure # you do not need to make any settings for local sessions, just save the default 

 # just for the first time
 make set-local-session  # a flag file is created
 make init               # snapshots directory is created

 # anytime you want to back up
 make backup-root        # backup with rsync

EDIT

Now there is a new target added: You can create a physical SD Card from your backups with one command:

make create-disk-from-last-backup

Follow the instructions, create your SD Card, boot RaspberryPi with this newly created SD Card.

ceremcem
  • 277
  • 7
  • 16
2

Here is a complete different approach. You can use LVM (Logical Volume Manager) to make consistent backups. Beside other improvements like easy adding, expanding and reducing storage or restore the operating system to an earlier status from a snapshot you can also make backups. You don't have do worry about dynamic changed files during backup, setting file systems read only, exclude specific directories or something else. With LVM you simply create a snapshot, mount this snapshot and backup it with the method you prefer. You can make a copy with cp -a, make a mirror with rsync, make an archive with tar or make an image with dd. Assuming you have mounted a backup device on /mnt/usbhd/pi_backup/ you can do for example:

rpi ~$ sudo lvcreate --snapshot --name rpi_snap --size 1G rpi_vg/root_lv
rpi ~$ sudo mkdir /mnt/snapshot
rpi ~$ sudo mount /dev/mapper/rpi_vg-rpi_snap /mnt/snapshot

# make backups
rpi ~$ sudo cp -a /mnt/snapshot/ /mnt/usbhd/pi_backup/
rpi ~$ sudo rsync -aH --delete /mnt/snapshot/ /mnt/usbhd/pi_backup/
rpi ~$ sudo tar -czf /mnt/usbhd/pi_backup/backup.tar.gz -V "Backup of my Raspberry Pi" -C /mnt/snapshot/ ./
rpi ~$ sudo dd if=/mnt/snapshot/ of=/mnt/usbhd/pi_backup/backup.img bs=4M

rpi ~$ sudo umount /mnt/snapshot/
rpi ~$ sudo lvremove rpi_vg/rpi_snap

It takes only one time a little effort to setup LVM. How to do that you can look at Easy backups and snapshots of a running system with LVM.

Ingo
  • 40,606
  • 15
  • 76
  • 189
  • How would you compare the `LVM approach` to [`rpi-clone`, a Raspberry Pi backup script on github](https://github.com/billw2/rpi-clone)? – Seamus Jan 27 '20 at 19:50
  • 1
    @Seamus **LVM** isn't a script doing things like any other solution I read here and also rpi-clone. It is part of the operating system to manage storage devices, even more it's part of the kernel, It must only invoked. Beside easy taking backups of running systems you can do many other nice things with your storage. E.g. you can boot into different snapshots with different stages of software installation. You can just revert to a snapshot and boot. It runs immediately, no need to fiddle with backup files and with long restore time. I use it for testing; always having a fresh flashed install. – Ingo Jan 27 '20 at 21:48
  • 1
    @Seamus Beside this you can do many other nice things with managing your storage, e.g. improve or shrink space, add additional physical volumes (disks, SSD, etc.) and so on. – Ingo Jan 27 '20 at 21:51
0

I have found a backup tool which makes installable images.

It also has utilities to mount and shrink images.

This may be useful to others

The documentation that comes with it is very brief so I note the following:-

  1. Extract the utilities into any directory and make scripts executable.
  2. Mount an ext4 formatted partition on your Pi in /mnt or /media (any format which allows large files and is supported by Pi e.g. exFAT or a network drive can be used).
  3. For the initial run you will be prompted for a Backup Image name e.g. /mnt/Image/BusterBackup.img
  4. You will be prompted for a Image ROOT filesystem size (in MB), this can be 0 for smallest possible or blank for full backup.
  5. On subsequent runs enter the path of the Backup Image to incrementally update.
An example of the commands I used:-
# Mount USB
sudo mount /dev/sda1 /mnt/Image/
# Update backup
sudo image-utils/image-backup /mnt/Image/BusterBackup.img
# Mount backup
sudo image-utils/image-mount /mnt/Image/BusterBackup.img  MountedImages
When done, run:
sudo umount MountedImages; sudo losetup -d /dev/loop0
# Compress backup
sudo sh -c "gzip -9c /mnt/Image/BusterBackup.img  > Images/BusterBackup.img.gz"

I have slightly modified the original (to copy mountpoints), to correctly calculate partition offsets and sizes and added a couple of comments.

#!/bin/bash
# Original https://raspberrypi.org/forums/viewtopic.php?p=1528736
# 2019-09-26    Modified to set size of boot sector

trap '{ stty sane; echo ""; errexit "Aborted"; }' SIGINT SIGTERM

ADDBLK=0

# Set BOOT_SIZE_MB to the Desired boot sector size (in MB) - should be multiple of 4MB
BOOT_SIZE_MB=256
BOOTSIZEM=$BOOT_SIZE_MB'M'

BOOTBEG=8192
BOOT_SIZE="$((BOOT_SIZE_MB * 1024 * 1024))"
ROUND_SIZE="$((4 * 1024 * 1024))"
# Ensure root sector starts on an Erase Block Boundary (4MB)
ROOTBEG=$(((BOOT_SIZE + ROUND_SIZE -1) / ROUND_SIZE * ROUND_SIZE / 512 + BOOTBEG))

MNTPATH="/tmp/img-backup-mnt"

ONEMB=$((1024 * 1024))

# create BOOT loop device
mkloop1()
{
  local INFO1=""
  local SIZE1=0
  local START1=0

  sync
  INFO1="$(sfdisk -d "${IMGFILE}")"
  START1=$(grep type=c <<< "${INFO1}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE1=$(grep type=c <<< "${INFO1}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP1="$(losetup -f --show -o $((${START1} * 512)) --sizelimit $((${SIZE1} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create BOOT loop device"
  fi
}

rmloop1()
{
  if [ "${LOOP1}" != "" ]; then
    sync
    losetup -d "${LOOP1}"
    LOOP1=""
 fi
}

# create ROOT loop device
mkloop2()
{
  local INFO2=""
  local SIZE2=0
  local START2=0

  sync
  INFO2="$(sfdisk -d "${IMGFILE}")"
  START2=$(grep type=83 <<< "${INFO2}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE2=$(grep type=83 <<< "${INFO2}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP2="$(losetup -f --show -o $((${START2} * 512)) --sizelimit $((${SIZE2} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create ROOT loop device"
  fi
}

rmloop2()
{
  if [ "${LOOP2}" != "" ]; then
    sync
    losetup -d "${LOOP2}"
    LOOP2=""
  fi
}

# Mount Image partitions
mntimg()
{
  MNTED=TRUE
  if [ ! -d "${MNTPATH}/" ]; then
    mkdir "${MNTPATH}/"
    if [ $? -ne 0 ]; then
      errexit "Unable to make ROOT partition mount point"
    fi
  fi
  mkloop2
  mount "${LOOP2}" "${MNTPATH}/"
  if [ $? -ne 0 ]; then
    errexit "Unable to mount image ROOT partition"
  fi
  if [ ! -d "${MNTPATH}/boot/" ]; then
    mkdir -p "${MNTPATH}/boot/"
    if [ $? -ne 0 ]; then
      errexit "Unable to make BOOT partition mount point"
    fi
  fi
  mkloop1
  mount "${LOOP1}" "${MNTPATH}/boot/"
  if [ $? -ne 0 ]; then
    errexit "Unable to mount image BOOT partition"
  fi
}

umntimg()
{
  umount "${MNTPATH}/boot/"
  if [ $? -ne 0 ]; then
    errexit "Unable to unmount image BOOT partition"
  fi
  rmloop1
  umount "${MNTPATH}/"
  if [ $? -ne 0 ]; then
    errexit "Unable to unmount image ROOT partition"
  fi
  rmloop2
  rm -r "${MNTPATH}/"
  MNTED=FALSE
}

errexit()
{
  echo ""
  echo "$1"
  echo ""
  if [ "${MNTED}" = "TRUE" ]; then
    umount "${MNTPATH}/boot/" &> /dev/null
    umount "${MNTPATH}/" &> /dev/null
    rm -rf "${MNTPATH}/" &> /dev/null
  fi
  rmloop1
  rmloop2
  exit 1
}

LOOP1=""
LOOP2=""
MNTED=FALSE

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

if [ $(id -u) -ne 0 ]; then
  errexit "$0 must be run as root user"
fi

PGMNAME="$(basename $0)"
for PID in $(pidof -x -o %PPID "${PGMNAME}"); do
  if [ ${PID} -ne $$ ]; then
    errexit "${PGMNAME} is already running"
  fi
done

rsync --version &> /dev/null
if [ $? -ne 0 ]; then
  errexit "rsync not installed (run: apt-get install rsync)"
fi

if command -v systemctl > /dev/null && systemctl | grep -q '\-\.mount'; then
  SYSTEMD=1
elif [ -f /etc/init.d/cron ] && [ ! -h /etc/init.d/cron ]; then
  SYSTEMD=0
else
  errexit "Unrecognized init system"
fi

if [ ${SYSTEMD} -eq 1 ]; then
  ROOT_PART="$(mount | sed -n 's|^/dev/\(.*\) on / .*|\1|p')"
else
  if [ ! -h /dev/root ]; then
    errexit "/dev/root does not exist or is not a symlink"
  fi
  ROOT_PART="$(readlink /dev/root)"
fi

ROOT_TYPE=$(blkid "/dev/${ROOT_PART}" | sed -n 's|^.*TYPE="\(\S\+\)".*|\1|p')

ROOT_DEV="${ROOT_PART:0:(${#ROOT_PART} - 1)}"
if [ "${ROOT_DEV}" = "mmcblk0p" ]; then
  ROOT_DEV="${ROOT_DEV:0:(${#ROOT_DEV} - 1)}"
fi

PTUUID="$(blkid "/dev/${ROOT_DEV}" | sed -n 's|^.*PTUUID="\(\S\+\)".*|\1|p')"

DEVSIZE=$(blockdev --getsize64 "/dev/${ROOT_PART}")
BLKSIZE=$(blockdev --getbsz "/dev/${ROOT_PART}")
BLKCNT=$((${DEVSIZE} / ${BLKSIZE}))
INFO="$(df | grep /dev/root)"
DFKSIZE=$(awk '{print $2}' <<< "${INFO}")
DFKFREE=$(awk '{print $4}' <<< "${INFO}")
ROOTSIZE=$((${BLKCNT} * ${BLKSIZE}))
ROOTUSED=$(((${DFKSIZE} - ${DFKFREE}) * 1024))
IRFSMIN=$(((${ROOTUSED} + (${ADDBLK} * ${BLKSIZE}) + (${ONEMB} - 1)) / ${ONEMB}))
IRFSMAX=$(((${ROOTSIZE} + (${ONEMB} - 1)) / ${ONEMB}))

IMGFILE="$1"
if [ "${IMGFILE}" = "" ]; then
# Create Image file
  while :
  do
    echo ""
    read -r -e -i "${IMGFILE}" -p "Image file to create? " IMGFILE
    if [ "${IMGFILE}" = "" ]; then
      continue
    elif [[ ! "${IMGFILE}" =~ ^/mnt/.*$ && ! "${IMGFILE}" =~ ^/media/.*$ ]]; then
      echo ""
      echo "${IMGFILE} does not begin with /mnt/ or /media/"
      continue
    fi
    if [ -d "${IMGFILE}" ]; then
      echo ""
      echo "${IMGFILE} is a directory"
    elif [ -f "${IMGFILE}" ]; then
      echo ""
      echo -n "${IMGFILE} already exists, Ok to delete (y/n)? "
      while read -r -n 1 -s answer; do
        if [[ "${answer}" = [yYnN] ]]; then
          echo "${answer}"
          if [[ "${answer}" = [yY] ]]; then
            break 2
          else
            break 1
          fi
        fi
      done
    else
      break
    fi
  done
  IRFSSIZE=""
  while :
  do
    echo ""
    read -r -e -i "${IRFSSIZE}" -p "Image ROOT filesystem size (MB) [${IRFSMAX}]? " IRFSSIZE
    if [ "${IRFSSIZE}" = "" ]; then
      IRFSSIZE=${IRFSMAX}
      break
    elif [ ${IRFSSIZE} -ge ${IRFSMIN} ]; then
      break
    else
      echo ""
      echo "Requested image ROOT filesystem size (${IRFSSIZE}) is too small (Minimum = ${IRFSMIN})"
      IRFSSIZE=${IRFSMIN}
    fi
  done
  echo ""
  echo -n "Create ${IMGFILE} [${IRFSSIZE} MB] (y/n)? "
  while read -r -n 1 -s answer; do
    if [[ "${answer}" = [yYnN] ]]; then
      echo "${answer}"
      if [[ "${answer}" = [yY] ]]; then
        break
      else
        errexit "Aborted"
      fi
    fi
  done
  if [ -f "${IMGFILE}" ]; then
    rm "${IMGFILE}"
    if [ $? -ne 0 ]; then
      errexit "Unable to delete existing image file"
    fi
  fi
  ROOTEND=$((${ROOTBEG} + ((${IRFSSIZE} * ${ONEMB}) / 512) - 1))
  truncate -s $(((${ROOTEND} + 1) * 512)) "${IMGFILE}"
  if [ $? -ne 0 ]; then
    errexit "Unable to create image file"
  fi
# create image/partitions
  sync
  fdisk "${IMGFILE}" <<EOF > /dev/null
p
n
p
1
${BOOTBEG}
+${BOOTSIZEM}
t
c
p
n
p
2
${ROOTBEG}
${ROOTEND}
p
w
EOF

  mkloop1
  mkloop2
  mkfs.vfat "${LOOP1}" > /dev/null
  if [ $? -ne 0 ]; then
    errexit "Unable to create image BOOT filesystem"
  fi
  dosfsck "${LOOP1}" > /dev/null
  if [ $? -ne 0 ]; then
    errexit "Image BOOT filesystem appears corrupted"
  fi
  if [ "${ROOT_TYPE}" = "f2fs" ]; then
    mkfs.f2fs "${LOOP2}" > /dev/null
  else
    mkfs.ext4 -q -b ${BLKSIZE} "${LOOP2}" > /dev/null
  fi
  if [ $? -ne 0 ]; then
    errexit "Unable to create image ROOT filesystem"
  fi
  rmloop2
  rmloop1
# Initialise image PARTUUID
  fdisk "${IMGFILE}" <<EOF > /dev/null
p
x
i
0x${PTUUID}
r
p
w
EOF
# Create empty directories in image root partition
  mntimg
  mkdir "${MNTPATH}/dev/" "${MNTPATH}/media/" "${MNTPATH}/mnt/" "${MNTPATH}/proc/" "${MNTPATH}/run/" "${MNTPATH}/sys/" "${MNTPATH}/tmp/"
  if [ $? -ne 0 ]; then
    errexit "Unable to create image directories"
  fi
  chmod a+rwxt "${MNTPATH}/tmp/"
  umntimg
  echo ""
  echo "Starting full backup (for incremental backups, run: $0 ${IMGFILE})"
# END of create image/partitions
else

# Check existing Image
  if [[ ! "${IMGFILE}" =~ ^/mnt/.*$ && ! "${IMGFILE}" =~ ^/media/.*$ ]]; then
    errexit "${IMGFILE} does not begin with /mnt/ or /media/"
  fi
  if [ -d "${IMGFILE}" ]; then
    errexit "${IMGFILE} is a directory"
  elif [ ! -f "${IMGFILE}" ]; then
    errexit "${IMGFILE} not found"
  fi
  echo "Starting incremental backup to ${IMGFILE}"
fi

# rsync root partition
mntimg
sync
rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/media' --exclude '/mnt/*/*' --exclude '/proc' --exclude '/run' --exclude '/sys' \
--exclude '/tmp' --exclude 'lost\+found' --exclude '/etc/udev/rules.d/70-persistent-net.rules' --exclude '/var/lib/asterisk/astdb.sqlite3-journal' / "${MNTPATH}/"
if [[ $? -ne 0 && $? -ne 24 ]]; then
  errexit "Unable to create backup"
fi
sync
umntimg
Milliways
  • 54,718
  • 26
  • 92
  • 182
-1

Open terminal and type 'lsblk -f' .
This should show all connected storage devices.
Then type 'dd if=/dev/[the NAME of your sd card] bs=1M'.
This will take a while so you might want to run it in the background.
This is the same exact way you backup your sd card in Linux.

jay jayjay
  • 147
  • 8