Views
For those of you who were interested in my backup scripts on the #kernel-panic IRC channel, here they are.
They're faily simple, extensible, and easy to use. They require zsh version 4.0 or newer.
In order to use disktotape.zsh you'll need flexbackup. This is optional.
Enjoy. Ping on me for any questions.
-Kelsey
Addendum: It seems as though people want to see these without downloading them. sigh Here goes.
Here's a README file. Everyone likes READMEs?, right? :
dailybackup.zsh, part of:
Backup Exec Agent (c) 2005 Kelsey Hudson. All rights reserved.
This picks up data from either local directories or network hosts and puts it
in another area, presumably for backing up data. The data remains unchanged
(as much as possible). Only one file is added in the case of a standard backup:
at the top-level of the backup output directory a file .00backup-stat is
created which shows various information about the source and destination of
the backup, the start and finish time, etc. Daily backups have placeholder
files (0-length) which are named <day>-is-<date>, for easy identification of
which day each backup references. .00backup-stat is still at the top-level
backup directory, *not* under each day.
Host setup:
Setup is rather easy. The simplest way to set up the software for backup is
to generate an ssh identity key for the root account, then distribute this
key to all the machines which are to be backed up.
To generate an ssh key:
machine% ssh-keygen -t dsa -f id_dsa
ssh-keygen will ask you for a passphrase for the keyfile. Since this
software is meant to be automated it is reccomended that no passphrase be
used; simply hit enter at the prompt.
Place the resulting id_dsa and id_dsa.pub files into ~root/.ssh. Copy the
id_dsa.pub file to each machine you wish to back up and append it to
~root/.ssh/authorized_keys (note: sshd must be configured to allow public
key authentication).
Note that this poses a potential security issue. If some malicious user
compromises the machine running this backup software, they can effectively
gain root access on *ALL* machines to which this key is distributed. To get
around this problem, you can set up rsyncd on machines to be backed up and
instead specify an rsync:// URI on the MACHINE line in the backup profile.
See rsyncd.conf(5) and the section BACKUP PROFILES for more information.
COMMAND LINE USAGE
dailybackup.zsh is simple to use. It takes one argument: the name of a
backup profile to process.
Example:
box# dailybackup.zsh sample-rsync.pfl
will process the backup instructions in sample-rsync.pfl. See the files
sample-*.pfl included with the distribution and read the section BACKUP
PROFILES below for more information.
BACKUP PROFILES
Files ending in .pfl specify backup profiles. Each profile file contains
information necessary to back up a machine.
There are two mandatory fields and two optional fields available in profile
files. These control the various behaviours of the program.
The mandatory fields are as follows:
MACHINE="" * The machine to contact (specify %LOCAL% for local disk-to-disk)
* This can be a proper rsync:// machine specification.
DIR=() * A bash-style array containing a list of directories or rsync
* modules which need to be backed up to this machine.
The optional fields are as follows:
TYPE="" * Backup type to perform. Valid entries are std and daily.
* Standard backups copy files to one directory; daily backups
* copy files to a directory under the output labeled with the
* day's shortened name. Default type is std.
OUT="" * Directory on the local machine in which backups will be stored.
* Default directory is /opt/backup/data/<profile-name>.
/opt/backup/data
This is the default directory for disk backups. It's advisable to mount a
large, LOCAL persistent storage volume here.
disktotape.zsh, part of:
Backup Exec Agent (c) 2005 Kelsey Hudson. All rights reserved.
This script is a front-end to flexbackup, suitable for use from within
cron(8). Thus it is mandatory that flexbackup and friends are installed
and configured prior to installing and attempting to utilize this script.
OPTIONS
There's one option to the program: the type of backup to perform. There
are essentially three types of backups available, full, incremental, and
differential. These are explained in the flexbackup(1) man page. The default
is to perform flexbackup's default.
PARAMETERS
There is one operating parameter inside the disktotape.zsh file: MOUNTPOINT.
In my setup, this is where I have taken an LVM snapshot of the latest copy
of replicated data. This can be any directory where the data you wish to
back up resides.
snaprotate.zsh, part of:
Backup Exec Agent (c) 2005 Kelsey Hudson. All Rights Reserved.
This script takes advantage of the built-in snapshot functionality of Linux
LVM2. A list of snapshot volume names and snapshot mount points is configured
inside the script. The script ages snapshot volumes out each time it is run.
The snapshots are initially mounted read-write to write a file .00snapdate to
the top-level directory on the filesystem. This file contains the date and
time the snapshot was taken. The filesystem is then remounted read-only.
CONFIGURATION OPTIONS
Edit the script with your editor of choice to change the following variables
to match your system's configuration.
VOLGRP="" * The LVM2 volume group on which your logical volumes reside.
STORAGEVOL="" * The logical volume on which your data resides. Snapshots
* will be taken of this volume.
SNAPSHTVOL=() * a bash-style array containing a list of arbitrary rotating
* snapshot volume names. The number of entries in this array
* must match the number of entries in the MOUNTPOINT
* parameter below.
MOUNTPOINT=() * a bash-style array containing a list of arbitrary rotating
* mount points for the snapshot volumes. The number of
* entries in this array must match the number of entries in
* the SNAPSHTVOL parameter above. The first such entry in
* this array is always the most current snapshot, the second
* is the next-most-current, etc.
LVEXTENTS=n * A number of logical volume extents to size the snapshot
* volumes. The number of free extents in your logical volume
* must equal at least this number times the number of
* snapshot volumes configured above. So, if you have three
* snapshot volumes configured, and you allocate 5000 extents
* to each volume, you need at least 15000 free extents on
* your volume group. Ideally, this number should be 10% of
* the size of your primary storage volume.
FILESYSTEM="" * The filesystem type on your primary storage volume.
FSOPTS="" * Extra filesystem options to pass to mount(8). XFS, for
* instance, requires the 'nouuid' option when mounting
* snapshot volumes.
The following is the source to dailybackup.zsh :
#!/bin/zsh
# (c) 2005 khudson: all rights reserved
PROG=`basename $0`
synerr() {
echo ""
echo "usage: $PROG <profile>"
echo ""
echo "Arguments:"
echo " profile | A profile file describing a backup"
echo ""
echo "Profile files (* denotes mandatory; defaults in ()):"
echo " MACHINE=<rsync-path> | *Machine to contact via rsync"
echo " | %LOCAL% denotes local directories"
echo " DIR=<path|(path1 path2 ..)> | *Directory or list of directories to"
echo " | process for this target"
echo " OUT=<path> | Local filesystem path where backups"
echo " | will be stored (/opt/backup/data/<profile>)"
echo " TYPE=<std|daily> | Type of backup to perform (std)"
echo ""
if [ ! -z "$1" ]; then
echo "error: $1";
fi
exit 2
}
waitcomplete() {
if [ -f $1 ]; then
if [ -z "$3" ]; then
echo "backup in progress; waiting $2 seconds" >&2
sleep $2
waitcomplete $1 $2 next
else
case "$3" in
next)
echo "backup still in progress; waiting another $2 seconds" >&2
sleep $2
waitcomplete $1 $2 last
;;
last)
echo "waited twice; bailing out." >&2
exit 2
;;
*)
echo "this shouldn\'t happen" >&2
;;
esac
fi
fi
}
localdir() {
echo $1 | grep -q "%LOCAL%"
return $?
}
rsyncpath() {
echo $1 | grep -q "^[Rr][Ss][Yy][Nn][Cc]://"
return $?
}
[ -z "$1" ] && synerr;
PF=`basename $1 .pfl`
LOCKFILE=/var/lock/dbu.$PF
OUTFILE=/opt/log/dbu.$PF.log
[ -f /opt/bin/functions ] && . /opt/bin/functions || {
echo "$PROG: function library unavailable: cannot continue."
exit 2
}
waitcomplete $LOCKFILE 3600;
trap "rm -f $LOCKFILE; rm -f /tmp/$$.rsync.log" 0 1 2 3 15
touch $LOCKFILE
[ ! -f "$1" ] && synerr "file not found";
# stupid shit
if [ `basename $1` = $1 ]; then
. ./$1
else
. $1
fi
[ ! -z "$MACHINE" ] || synerr "missing MACHINE in profile $1";
[ ! -z "${DIR[*]}" ] || synerr "missing DIR in profile $1";
[ -z "$TYPE" ] && TYPE=std
[ -z "$OUT" ] && OUT=/opt/backup/data/$PF
case "$TYPE" in
std)
;;
daily)
DAY=`date +%a | tr A-Z a-z`;
rm -f ${OUT}/${DAY}-is-*
touch ${OUT}/${DAY}-is-`date +%m-%d`
OUT=$OUT/$DAY
;;
*)
synerr "TYPE must be one of: std daily (default: std)"
;;
esac
mkdir -p $OUT
rm -f $OUT/.00backup-stat
exec 4>$OUT/.00backup-stat
echo "start-time `date`" >&4
if rsyncpath $MACHINE; then
if ! echo $MACHINE | grep '/$'; then
MACHINE=${MACHINE}/
fi
else
MACHINE=$MACHINE:
fi
localdir $MACHINE && unset MACHINE
for i in "${DIR[@]}"; do
rsync -rpltogvv ${MACHINE}${i} $OUT > /tmp/$$.rsync.log 2>&1
if [ $? -ne 0 ]; then
DT=`date "+%b %d %T"`
echo ""
echo "$DT $PROG: problems processing profile $PF, directory $i"
echo ""
cat /tmp/$$.rsync.log
fi
done
echo "end-time `date`" >&4
echo "machine-path $MACHINE" >&4
echo "dirs-processed ${DIR[@]}" >&4
echo "output-directory $OUT" >&4
echo "backup-type $TYPE" >&4
exec 4<&-
exec 2<&-
And, the source to disktotape.zsh :
#!/bin/zsh
MOUNTPOINT="/opt/backup/snap0"
LOCKFILE=/var/lock/disktotape
OUTFILE=/tmp/disk-to-tape.$$
BULOG=/tmp/flex.$$
PROG=`basename $0`
LOCKOURS=0
trap "rm -f $OUTFILE $BULOG; [ $LOCKOURS -gt 0 ] && rm -f $LOCKFILE; exit 2" \
0 1 2 3 15
[ -f /opt/bin/functions ] && . /opt/bin/functions || {
echo "$PROG: function library not available: cannot continue."
}
exec 2>$OUTFILE
if [ -f $LOCKFILE ]; then
echo "error: disk-to-tape backup already running"
fatal "" 2
else
LOCKOURS=1
touch $LOCKFILE
fi
exec 3>$BULOG
case i in
full)
flexbackup -dir $MOUNTPOINT -level full >&3
;;
incr*)
flexbackup -dir $MOUNTPOINT -level incremental >&3
;;
diff*)
flexbackup -dir $MOUNTPOINT -level differential >&3
;;
*)
flexbackup -dir $MOUNTPOINT >&3
;;
esac
STAT=$?
exec 3<&-
if [ $STAT -gt 0 ]; then
recoverable "problems with backup:"
echo "error: log of backup process as follows:"
cat $BULOG
echo "error: end of log"
fi
exit 0
Note: there are more script files, but this should be enough to get you started.