How to Backup/Shrink the image of my working Raspbian SD Card

Question

I want to shrink the image to minimise storage and to efficiently distribute this image to others who may have smaller SD Cards.

Answer 1

Many users ask how they can produce a small customised image that can be used to create new SD Cards and/or how they can shrink an image to make a compact backup of a running system.

I have found a backup tool which makes installable images on the Pi itself.

The original tool has changed, and keeps changing, so I can not guarantee the download - particularly as the changes are undocumented.
My script, below is in regular use

It also has utilities to mount and shrink images.

This may be useful to others

The documentation that comes with it is very brief so I note the following:-

1. Extract the utilities into any directory and make scripts executable.
2. Mount an ext4 formatted partition on your Pi in /mnt or /media (any format which allows large files and is supported by Pi e.g. exFAT or a network drive can be used).
3. For the initial run you will be prompted for a Backup Image name e.g. /mnt/Image/BusterBackup.img
4. You will be prompted for a Image ROOT filesystem size (in MB), this can be 0 for smallest possible or blank for full backup.
   For a compact image use the Used from df -H --type=ext4 --output=fstype,used plus 300
5. On subsequent runs enter the path of the Backup Image to incrementally update.

NOTE The image created will not auto resize on boot, and should be maximised in raspi-config.

An example of the commands I used:-

# Mount USB
sudo mount /dev/sda1 /mnt/Image/
# Mount network drive
sudo mount.cifs //10.1.2.107/Images /mnt/Image -o user=UUU
# Update backup
sudo image-utils/image-backup /mnt/Image/BusterBackup.img
# Mount backup
sudo image-utils/image-mount /mnt/Image/BusterBackup.img  MountedImages
When done, run:
sudo umount MountedImages; sudo losetup -d /dev/loop0
# Compress backup
sudo gzip -9kN /mnt/Image/StretchBackup.img

I have slightly modified the original image-backup (to copy mountpoints), to correctly calculate partition offsets and sizes and added a couple of comments.

#!/bin/bash
# Original https://raspberrypi.org/forums/viewtopic.php?p=1528736
# 2019-09-26    Modified to set size of boot sector

trap '{ stty sane; echo ""; errexit "Aborted"; }' SIGINT SIGTERM

ADDBLK=0

# Set BOOT_SIZE_MB to the Desired boot sector size (in MB) - should be multiple of 4MB
BOOT_SIZE_MB=256
BOOTSIZEM=$BOOT_SIZE_MB'M'

BOOTBEG=8192
BOOT_SIZE="$((BOOT_SIZE_MB * 1024 * 1024))"
ROUND_SIZE="$((4 * 1024 * 1024))"
# Ensure root sector starts on an Erase Block Boundary (4MB)
ROOTBEG=$(((BOOT_SIZE + ROUND_SIZE -1) / ROUND_SIZE * ROUND_SIZE / 512 + BOOTBEG))

MNTPATH="/tmp/img-backup-mnt"

ONEMB=$((1024 * 1024))

# create BOOT loop device
mkloop1()
{
  local INFO1=""
  local SIZE1=0
  local START1=0

  sync
  INFO1="$(sfdisk -d "${IMGFILE}")"
  START1=$(grep type=c <<< "${INFO1}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE1=$(grep type=c <<< "${INFO1}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP1="$(losetup -f --show -o $((${START1} * 512)) --sizelimit $((${SIZE1} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create BOOT loop device"
  fi
}

rmloop1()
{
  if [ "${LOOP1}" != "" ]; then
    sync
    losetup -d "${LOOP1}"
    LOOP1=""
 fi
}

# create ROOT loop device
mkloop2()
{
  local INFO2=""
  local SIZE2=0
  local START2=0

  sync
  INFO2="$(sfdisk -d "${IMGFILE}")"
  START2=$(grep type=83 <<< "${INFO2}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE2=$(grep type=83 <<< "${INFO2}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP2="$(losetup -f --show -o $((${START2} * 512)) --sizelimit $((${SIZE2} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create ROOT loop device"
  fi
}

rmloop2()
{
  if [ "${LOOP2}" != "" ]; then
    sync
    losetup -d "${LOOP2}"
    LOOP2=""
  fi
}

# Mount Image partitions
mntimg()
{
  MNTED=TRUE
  if [ ! -d "${MNTPATH}/" ]; then
    mkdir "${MNTPATH}/"
    if [ $? -ne 0 ]; then
      errexit "Unable to make ROOT partition mount point"
    fi
  fi
  mkloop2
  mount "${LOOP2}" "${MNTPATH}/"
  if [ $? -ne 0 ]; then
    errexit "Unable to mount image ROOT partition"
  fi
  if [ ! -d "${MNTPATH}/boot/" ]; then
    mkdir -p "${MNTPATH}/boot/"
    if [ $? -ne 0 ]; then
      errexit "Unable to make BOOT partition mount point"
    fi
  fi
  mkloop1
  mount "${LOOP1}" "${MNTPATH}/boot/"
  if [ $? -ne 0 ]; then
    errexit "Unable to mount image BOOT partition"
  fi
}

umntimg()
{
  umount "${MNTPATH}/boot/"
  if [ $? -ne 0 ]; then
    errexit "Unable to unmount image BOOT partition"
  fi
  rmloop1
  umount "${MNTPATH}/"
  if [ $? -ne 0 ]; then
    errexit "Unable to unmount image ROOT partition"
  fi
  rmloop2
  rm -r "${MNTPATH}/"
  MNTED=FALSE
}

errexit()
{
  echo ""
  echo "$1"
  echo ""
  if [ "${MNTED}" = "TRUE" ]; then
    umount "${MNTPATH}/boot/" &> /dev/null
    umount "${MNTPATH}/" &> /dev/null
    rm -rf "${MNTPATH}/" &> /dev/null
  fi
  rmloop1
  rmloop2
  exit 1
}

LOOP1=""
LOOP2=""
MNTED=FALSE

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

if [ $(id -u) -ne 0 ]; then
  errexit "$0 must be run as root user"
fi

PGMNAME="$(basename $0)"
for PID in $(pidof -x -o %PPID "${PGMNAME}"); do
  if [ ${PID} -ne $$ ]; then
    errexit "${PGMNAME} is already running"
  fi
done

rsync --version &> /dev/null
if [ $? -ne 0 ]; then
  errexit "rsync not installed (run: apt-get install rsync)"
fi

if command -v systemctl > /dev/null && systemctl | grep -q '\-\.mount'; then
  SYSTEMD=1
elif [ -f /etc/init.d/cron ] && [ ! -h /etc/init.d/cron ]; then
  SYSTEMD=0
else
  errexit "Unrecognized init system"
fi

if [ ${SYSTEMD} -eq 1 ]; then
  ROOT_PART="$(mount | sed -n 's|^/dev/\(.*\) on / .*|\1|p')"
else
  if [ ! -h /dev/root ]; then
    errexit "/dev/root does not exist or is not a symlink"
  fi
  ROOT_PART="$(readlink /dev/root)"
fi

ROOT_TYPE=$(blkid "/dev/${ROOT_PART}" | sed -n 's|^.*TYPE="\(\S\+\)".*|\1|p')

ROOT_DEV="${ROOT_PART:0:(${#ROOT_PART} - 1)}"
if [ "${ROOT_DEV}" = "mmcblk0p" ]; then
  ROOT_DEV="${ROOT_DEV:0:(${#ROOT_DEV} - 1)}"
fi

PTUUID="$(blkid "/dev/${ROOT_DEV}" | sed -n 's|^.*PTUUID="\(\S\+\)".*|\1|p')"

DEVSIZE=$(blockdev --getsize64 "/dev/${ROOT_PART}")
BLKSIZE=$(blockdev --getbsz "/dev/${ROOT_PART}")
BLKCNT=$((${DEVSIZE} / ${BLKSIZE}))
INFO="$(df | grep /dev/root)"
DFKSIZE=$(awk '{print $2}' <<< "${INFO}")
DFKFREE=$(awk '{print $4}' <<< "${INFO}")
ROOTSIZE=$((${BLKCNT} * ${BLKSIZE}))
ROOTUSED=$(((${DFKSIZE} - ${DFKFREE}) * 1024))
IRFSMIN=$(((${ROOTUSED} + (${ADDBLK} * ${BLKSIZE}) + (${ONEMB} - 1)) / ${ONEMB}))
IRFSMAX=$(((${ROOTSIZE} + (${ONEMB} - 1)) / ${ONEMB}))

IMGFILE="$1"
if [ "${IMGFILE}" = "" ]; then
# Create Image file
  while :
  do
    echo ""
    read -r -e -i "${IMGFILE}" -p "Image file to create? " IMGFILE
    if [ "${IMGFILE}" = "" ]; then
      continue
    elif [[ ! "${IMGFILE}" =~ ^/mnt/.*$ && ! "${IMGFILE}" =~ ^/media/.*$ ]]; then
      echo ""
      echo "${IMGFILE} does not begin with /mnt/ or /media/"
      continue
    fi
    if [ -d "${IMGFILE}" ]; then
      echo ""
      echo "${IMGFILE} is a directory"
    elif [ -f "${IMGFILE}" ]; then
      echo ""
      echo -n "${IMGFILE} already exists, Ok to delete (y/n)? "
      while read -r -n 1 -s answer; do
        if [[ "${answer}" = [yYnN] ]]; then
          echo "${answer}"
          if [[ "${answer}" = [yY] ]]; then
            break 2
          else
            break 1
          fi
        fi
      done
    else
      break
    fi
  done
  IRFSSIZE=""
  while :
  do
    echo ""
    read -r -e -i "${IRFSSIZE}" -p "Image ROOT filesystem size (MB) [${IRFSMAX}]? " IRFSSIZE
    if [ "${IRFSSIZE}" = "" ]; then
      IRFSSIZE=${IRFSMAX}
      break
    elif [ ${IRFSSIZE} -ge ${IRFSMIN} ]; then
      break
    else
      echo ""
      echo "Requested image ROOT filesystem size (${IRFSSIZE}) is too small (Minimum = ${IRFSMIN})"
      IRFSSIZE=${IRFSMIN}
    fi
  done
  echo ""
  echo -n "Create ${IMGFILE} [${IRFSSIZE} MB] (y/n)? "
  while read -r -n 1 -s answer; do
    if [[ "${answer}" = [yYnN] ]]; then
      echo "${answer}"
      if [[ "${answer}" = [yY] ]]; then
        break
      else
        errexit "Aborted"
      fi
    fi
  done
  if [ -f "${IMGFILE}" ]; then
    rm "${IMGFILE}"
    if [ $? -ne 0 ]; then
      errexit "Unable to delete existing image file"
    fi
  fi
  ROOTEND=$((${ROOTBEG} + ((${IRFSSIZE} * ${ONEMB}) / 512) - 1))
  truncate -s $(((${ROOTEND} + 1) * 512)) "${IMGFILE}"
  if [ $? -ne 0 ]; then
    errexit "Unable to create image file"
  fi
# create image/partitions
  sync
  fdisk "${IMGFILE}" <<EOF > /dev/null
p
n
p
1
${BOOTBEG}
+${BOOTSIZEM}
t
c
p
n
p
2
${ROOTBEG}
${ROOTEND}
p
w
EOF

  mkloop1
  mkloop2
  mkfs.vfat "${LOOP1}" > /dev/null
  if [ $? -ne 0 ]; then
    errexit "Unable to create image BOOT filesystem"
  fi
  dosfsck "${LOOP1}" > /dev/null
  if [ $? -ne 0 ]; then
    errexit "Image BOOT filesystem appears corrupted"
  fi
  if [ "${ROOT_TYPE}" = "f2fs" ]; then
    mkfs.f2fs "${LOOP2}" > /dev/null
  else
    mkfs.ext4 -q -b ${BLKSIZE} "${LOOP2}" > /dev/null
  fi
  if [ $? -ne 0 ]; then
    errexit "Unable to create image ROOT filesystem"
  fi
  rmloop2
  rmloop1
# Initialise image PARTUUID
  fdisk "${IMGFILE}" <<EOF > /dev/null
p
x
i
0x${PTUUID}
r
p
w
EOF
# Create empty directories in image root partition
  mntimg
  mkdir "${MNTPATH}/dev/" "${MNTPATH}/media/" "${MNTPATH}/mnt/" "${MNTPATH}/proc/" "${MNTPATH}/run/" "${MNTPATH}/sys/" "${MNTPATH}/tmp/"
  if [ $? -ne 0 ]; then
    errexit "Unable to create image directories"
  fi
  chmod a+rwxt "${MNTPATH}/tmp/"
  umntimg
  echo ""
  echo "Starting full backup (for incremental backups, run: $0 ${IMGFILE})"
# END of create image/partitions
else

# Check existing Image
  if [[ ! "${IMGFILE}" =~ ^/mnt/.*$ && ! "${IMGFILE}" =~ ^/media/.*$ ]]; then
    errexit "${IMGFILE} does not begin with /mnt/ or /media/"
  fi
  if [ -d "${IMGFILE}" ]; then
    errexit "${IMGFILE} is a directory"
  elif [ ! -f "${IMGFILE}" ]; then
    errexit "${IMGFILE} not found"
  fi
  echo "Starting incremental backup to ${IMGFILE}"
fi

# rsync root partition
mntimg
sync
rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/media' --exclude '/mnt/*/*' --exclude '/proc' --exclude '/run' --exclude '/sys' \
--exclude '/tmp' --exclude 'lost\+found' --exclude '/etc/udev/rules.d/70-persistent-net.rules' --exclude '/var/lib/asterisk/astdb.sqlite3-journal' / "${MNTPATH}/"
if [[ $? -ne 0 && $? -ne 24 ]]; then
  errexit "Unable to create backup"
fi
sync
umntimg