Hi again,
after a long New Year’s break, I’m back to deliver the promised procedure of backup via rsync when working with Openfiler.
In the previous post I described the hardware situation in our company, so it should be clear by now that I have two Openfiler machines: one in production, and another one in standby.
Before I get into the details, I wanna tell you about some things you may want to consider.

The Openfiler is actually able to do the DRDB, which is a High Availability setup, when the production machine is replicated online to the backup one. Each of them has an IP, plus they have another virtual IP that they pass on to each other in case of failure. The users (and the DNS) actually work with this VIP. So any failure is actually transparent to everybody. It’s a standard HA setup. The only thing I asked myself is what happens if you accidentally delete a file? If it’s going to be replicated immediately to the other machine - what’s the point of having a backup? Actually, I believe I’m not the only one who’s asked that question, so it’s very possible that there are answers and solutions to that, but somehow I decided not to try so hard, especially that we decided that it would be ok to have a backup once a day. YMMV if you need to do it more often. When that was out of the way, it was clear I should go with rsync.

This is where it all begins.
Not to bore you with details, I just wanna mention that I didn’t have any success with rsync’s built-in daemon, as I had a problem with permissions (samba again), but to tell you the truth, it didn’t really matter, because instead of using modules I used the full path. Big deal.

Another major thing I had a problem with, was the same one from the HA part: what happens if you delete a file? After a lot of googling, I came to a conclusion that what I needed was a recycle bin. Surprise surprise - Openfiler doesn’t have one. Why? I don’t know:)

So to sum up, before I post the full solution, the main idea was to write a few scripts, each for one share and run them one by one at night (say, every half hour, so that they let each other finish).
Each script will contain one rsync command.
In addition, we’ll create a recycle bin in every share, so that every user from that share would have access to it.
Then we’ll write another script to find deleted items in the recycle bin that have been there over a week and delete them too. The latter will be run from the production machine.

I had another thought about the last one: I wanted to delete the items in the trash after a week on the production server and after 2 weeks on the backup. To backup the backup:)
But I have to dig some more regarding the excluded files functions in rsync. Or, as an alternative, you can just separate the directories inside the shares in the script, which is a bit stupid:)

So here we go.
First, we need the 2 servers to talk to each other without asking for password and such. I had no problems with it, since they’re both within the same LAN and I’m not paranoid. This is how we do it:
legend:
production server = prod1
backup server = back2

on back2 run these:

ssh-keygen -t rsa
ssh root@prod1 \"mkdir .ssh\"
scp .ssh/id_rsa.pub root@prod1:.ssh/authorized_keys2

I run under root.

Next we have to write the scripts:

nano /root/scripts/test_rsync.sh
#!/bin/bash
MASTER=\"prod1\"
DIR=\"/mnt/vol/disk/share/\"
LDIR=\"/mnt/volbkp/diskbkp/share/\"
SSH=\"/usr/bin/ssh\"
rsync -av --delete --rsh=$SSH $MASTER:$DIR $LDIR

This one will copy everything from the prod1 server’s “system” share to the same share at back2.
It will copy all the permissions also, but you really should read the manual, because there a lot of keys included.
Be aware that it will also delete all the files on the back2 that aren’t on the prod1!!
You may want to include the –dry-run key to see what’s gonna happen before you actually delete the files.

Now we let the cron worry about running it:

crontab -e
a (to insert)
01 1 * * * /root/scripts/test_rsync.sh
\"Esc\" (to stop editing)
\"Shift+zz\" (to save and exit)

I’m being so specific about vi, because personally, I’m not used to work with it, so it may help someone.
This will run the script at 1:01 am every night.
Do this for every share.
It’s also a good idea to “chmod +o” the script to make’em executable.

Now, the recycle bin.
This one is tricky. To just add a recycle bin to a share, you may only edit the smb.conf and add some lines(I’ll tell you later which ones). The biggest problem here is with the same generate.inc file that’s messing up the whole thing each time you restart samba from the web interface.
So we need to dig in once again.

nano /opt/openfiler/var/www/includes/generate.inc

on line 504, after these:

	function ac_recurse_dir($ac_mountpoint, $ac_lv, $ac_vg, $ac_is_snapshot,
				$ac_chmod, $ac_chmod_path, $ac_share_path) {

insert:

	/* -------------------------- */
global $vfs_objects, $recycle_repository, $recycle_keeptree, $recycle_versions,
 $recycle_touch, $recycle_maxsize, $recycle_exclude;
	
		$vfs_objects = \"vfs objects = recycle\";
		$recycle_repository = \"recycle:repository = .recycle\";
		$recycle_keeptree =  \"recycle:keeptree = Yes\";
		$recycle_touch =  \"recycle:touch = Yes\";
		$recycle_versions =  \"recycle:versions = No\";
		$recycle_maxsize =  \"recycle:maxsize = 0\";
		$recycle_exclude = \"recycle:exclude = *.tmp *.temp\";

you may want to edit the “exclude” line to add more extensions.
Note that we “touch” the files when they’re going to recycle bin. This way we can spot them via atime function.
restart samba from webconfig and once you delete your first file, you should see a .recycle directory within that same share. It also preserves the dir tree.

Now, the only thing left is to find these old deleted files and remove them permanently:

nano /root/scripts/recycle_remove.sh
#!/bin/bash
find /mnt/vol/disk/share/.recycle/ -type f -atime +7 -exec rm {} \;

edit the crontab the same way we did it before and let it run every day at 6 am or something.
Note, this one we put on the production server.

Another helpful thing would be to edit /etc/aliases and change the root’s mail to your own, so that you get the log from cron.

Well, that’s about it. Hope it helped.

C ya,
Ozzik.

Update: In order for the recycle bin to work you also have to enter these lines at line 1175, after it says “Experimental” and at line 1586, also after “Experimental”:

 $ac_smb_fp->AddLine($vfs_objects);
                                                        $ac_smb_fp->AddLine($recycle_repository);
                                                        $ac_smb_fp->AddLine($recycle_keeptree);
                                                        $ac_smb_fp->AddLine($recycle_versions);
                                                        $ac_smb_fp->AddLine($recycle_touch);
                                                        $ac_smb_fp->AddLine($recycle_maxsize);
                                                        $ac_smb_fp->AddLine($recycle_exclude);