A user at $WORK was running a series of jobs on the cluster -- dozens
at any moment. Other users have their quota set to 60 GB, but this
user was not (long story). His home directory is at 400GB, but it was
closer to a terabyte not so long ago....right when we had a hard drive
and a tape drive fail at the same time on our backup server.
We do backups every night to tape using Bacula. Most backups are
incremental (whatever changed since the last backup, usually the day
before) and are small...maybe tens of GB per day. But backups for
this user, because of the proliferation of logs from his jobs, were
closer to the size of his home directory every day -- simply because
all these log files were being updated as each job progressed.
Ordinarily this wouldn't be a problem, but the cluster of hardware
failures have really fucked things up; they're better now, but I'm
very slowly playing catchup backups. Eating a tape or more every day
is not in my budget right this moment.
I asked him if any of the log files could be excluded from backups
without any great loss. After talking it over with him, we came to
this agreement:
- His home directory would be backed up (obvs)
- but within "projects/output", only files that contained "rep0"
somewhere in the filename would be backed up.
This would exclude lots of other files like "1rep2.foo", "8rep9.log",
etc, and would cut out about 200 GB of useless churn every day.
Bacula has the ability to do this sort of thing...but I found its
methods somewhat counterintuitive, so I want to set down what I did
and how I tested it.
First off, the original, let's-include-everything FileSet looked like
this:
FileSet {
Name = "example"
Include {
File = /home/example
Options {
signature = SHA1
}
}
Exclude {
File = /proc
File = /tmp
File = /.journal
File = /.fsck
File = /.zfs
}
}
We back up everything under /home/example, we keep SHA1 signatures,
and we exclude a handful of directories (most of which are
boilerplate, applied to every FileSet by default).
In order to get Bacula to change the FileSet definition, you have to
get the director to reload its configuration file. But some errors
-- not all -- cause a running bacula-dir process to die. So before I
started fiddling around, I added a Makefile to the /opt/bacula/etc
directory that looked like this:
test:
@/opt/bacula/sbin/bacula-dir -t && echo "bacula-dir.conf looks good" || echo "problem with bacula-dir.conf"
reload: test
echo "reload" | /opt/bacula/sbin/bconsole
Whenever I made a change, I'd run "make reload", which would test the
configuration first; if it failed, bacula would not be reloaded.
(The "@" symbol, in a Makefile, discards standard output.)
Next, I needed a listing of what we were backing up now, before I
started fiddling with things:
echo "estimate job=fileserver-example listing" | bconsole > /tmp/listing-before
The "estimate" command gets Bacula to estimate how big the job is; the
"listing" argument tells it to list the files it'd back up. By
default it gives you the info for a full backup. (You can also append
a joblevel, so you can see how big a Differential or Incremental; I
didn't need that here, but it's worth remembering for next time.)
After that, I made another Makefile that looked like this:
test: estimate shouldwork shouldfail
estimate:
@echo "estimate job=fileserver-example listing" | bconsole > /tmp/listing-after ; wc -l /tmp/listing*
shouldwork: estimate
grep rep0 /tmp/listing-before | grep projects/output | while read i ; do grep -q $$i /tmp/listing-after || exit 1 ; done
shouldfail:
grep rep2 /tmp/listing-before |grep projects/output | while read i ; do grep -q $$i /tmp/listing-after && exit 1 ; done ; true
This is a little hackish, so in detail:
The estimate target gets an updated listing of what Bacula will
back up; the line count lets me eyeball how it compares to the old,
all-inclusive listing.
The shoudwork target gives me a quick way to make sure that all
the files with "rep0" in the name and "projects/output" in the path
are still in that updated listing. We grep for these files in the
new listing; it either works or exits with error code 1, which make
will catch and declare an error.
The shouldfail target is similar, except I'm making sure that
files with "rep2" in the name are excluded from the new listing
and we're short-circuiting the loop if any line is found. The
"true" at the end is there to give make a final success; we only
make it to that command if the entire loop has not found anything,
which is what we want. It's there to make this test a "MUST NOT".
(That's probably not explained very well.)
Anyhow: after each change, I'd run "make reload" as root to make sure
that the syntax worked. After that, I'd run "make test" as an
ordinary user (no need for root privileges) to make sure that I was on
the right track. After a while, I got this:
FileSet {
Name = "example"
Include {
File = /home/example
Include {
Options {
signature = SHA1
Wilddir = /home/example/projects/output
Exclude = yes
}
}
}
Include {
File = /home/example/projects/output
Options {
WildFile = "*rep0*"
Signature = SHA1
}
Options {
Exclude = yes
RegexFile = ".*"
}
}
Exclude {
File = /proc
File = /tmp
File = /.journal
File = /.fsck
File = /.zfs
}
}
Again, this is a little counterintuitive to me, so here's how it works out.
The first "Include" stanza is the same, except that in the "Options"
section we're excluding "/home/example/projects/output". That's what
the "Wilddir" and "Exclude = yes" directives are for.
The second "Include" stanza puts the "/home/example/projects/output"
back in, but modified with two "Options" sections: the first to
include "rep0" (a simple fileglob) and the second to exclude
everything. What ends up being included by this stanza is the union
of those two options: only files named "rep0" in the directory
"/home/example/projects/output".
Last, the third stanza is our standard "Exclude" boilerplate.
After I was confident that I had the right set of files excluded, I
sent the user a list of files to confirm that all was well:
cat /tmp/listing_before | while read i ; do grep -q $i /tmp/listing_after || echo $i ; done > /tmp/excluded
Now, I'm the first to admit that that is ugly. Diff, useless use of
cat...lots of objections to raise. But it's been a long day and I got
what I wanted. I pointed the user at it, made sure it was okay, and
committed the changes.
All in all, this gave me a good loop for testing: it caught fatal
errors before they happened, it let me be sure I was excluding the
right things, and I was able to work in a stepwise fashion to get
where I wanted.
