Posts Tagged by Rsync

rsyncp – provide ssh password to rsync using expect

28-Jul-2011 Posted by Sonia Hamilton under , ,
No Comments

A script I wrote – rsyncp. It allows you to provide a password to rsync over ssh, when you’re unable to use ssh key-based authentication:

% cat ~/bin/rsyncp
#!/usr/bin/expect
eval spawn rsync $argv
expect  "*?assword:" { send "secretstuffr"}
expect eof

You can then use rsyncp in another script, like this:

for h in foo bar ; do
  rsyncp -av --progress srcdir/ $h:dstdir/
done

And before you add a comment saying use ssh keys or keychain, this is totally insecure, I agree with you! Key based authentication is disabled on the target server, and I’m still trying to resolve the politics around this at my current company, and sshpass seems broken <sigh>

fuzzy rsync

03-Mar-2011 Posted by Sonia Hamilton under
No Comments

A nice option I discovered in rsync – the –fuzzy option. If a destination file is missing rsync looks for a similar file in the same directory, and copies/moves it. The current “similar” algorithm is “identical size and modified time”, or “similarly-named”. For example:

rsync foo/ example.com:video/foo/ --fuzzy --delete-delay
...
sent 69712 bytes  received 118992 bytes  75481.60 bytes/sec   <-- small amount of data xferred
total size is 4966997090  speedup is 26321.63                 <-- woohoo! big speed up

Notice the –delete-delay option is fairly much required, as rsync by default does a –delete-before, thus removing the base file before it can be copied/moved.

When to use this? When you’re rsync’ing large binary files, and the only change you’ve made is to copy or rename files within the same directory. Which is often what I’m doing – at work moving around isos or install packages, and at home organising BJJ videos :-)

Addendum

Also, the –partial and –partial-dir options are handy when working with large binary files. Tells rsync not to delete partially tranferred files if your transfer gets interrupted, for example by a network glitch. (Thanks AndyS).

DeltaCopy – rsync for Windows

12-Nov-2009 Posted by Sonia Hamilton under ,
1 Comment

A nice little tool I came across – DeltaCopy – for when you have to work with Windows desktops and want to rsync (and don’t want to install Cygwin). I haven’t had time to play with it yet, but apparently the DeltaCopy client will connect to an rsync server.

Out of interest, there’s also a SourceForge package for installing a minimal ssh server on Windows, rather than the full Cygwin.

rsync notes

14-May-2006 Posted by Sonia Hamilton under ,
2 Comments

Summary

 $ rsync -av -e ssh --delete --force source_host:source_dir/ dest_dir
 $ rsync -av -e ssh --delete --force source dest_host:dest_dir/

eg

 $ rsync -av -e ssh                  sandia:stripes/music/spanish-french/ spanish-french/
 $ rsync -av -e ssh --delete --force sandia:stripes/music/spanish-french/ spanish-french/

Using SSH

Use rsync over ssh with the -e flag:

 $ rsync -av -e ssh source dest_host:dest_dir

To automate rsync over ssh backups, use keychain. For example, for a user called backupuser:


 $ cat ~sonia/bin/rsyncup.sh
 #!/bin/bash
 . /home/backupuser/.keychain/$HOSTNAME-sh
 rsync -av -e ssh  --delete --force /home /etc dest_host:home-etc-backup/
 $ crontab -l
 1 4 * * * ~sonia/bin/rsyncup.sh

Trailing Slash

Summary – these are equivalent:

 rsync $myflags sandia:stripes/books/     books/
 rsync $myflags sandia:stripes/books      .

A trailing slash on the source directory syncs the directory contents, but not the directory itself; without a trailing slash the source directory itself will be synced. The trailing slash has no affect on the destination directory.

For example, with this source directory structure:

 ~/a
 ~/a/dir1
 ~/a/dir2

 $ rsync -av --stats ~/a/ destination_dir
 $ ls -F destination_dir
 dir1/ dir2/                 # ie synced as destination_dir/dir1
 $ rsync -av --stats ~/a destination_dir
 $ ls -F destination_dir
 a/
 $ ls -F destination_dir/a
 dir1/ dir2/                 # ie synced as destination_dir/a/dir1

Limiting Bandwidth

Limit bandwidth used by rsync with the –bwlimit option; the value is in kilobytes/s (many network speeds are measured in kilobits, which are 1/8th of kilobytes).


 $ rsync -av --bwlimit=256 src1 src2 dest

Deleting Files

If files are deleted from the source directory, they will only be deleted from the destination directory if the –delete option is used:

 $ rsync -av --delete ~/a/ destination_dir

File Selection

Files can be included or excluded from syncing using –include-from or –exclude-from; it’s better not to use both as things can get rather confusing…

 $ cat exclude.txt
 *tmp
 /etc/skel
 $ rsync -av src1 src2 --exclude-from=exclude.txt dest_host:dest_dir

Filepaths

You can specify that directories specified in the source path be created at the destination using the -R option (usually not desired):

 $ rsync -av -e ssh src_host:/var/www /dest
 $ ls -F /dest
 www/                     # notice /dest/var not created
 $ rsync -avR -e ssh src_host:/var/www /dest
 $ ls -F /dest
 var/
 $ ls -F /dest/var
 www/                    # ie synced as /dest/var/www, not /dest/www

Rsync on Windows

  • on Windows, install Cygwin, rsync, OpenSSH, vim
  • add c:cygwinbin to the PATH system environment variable
  • either get into a bash shell and run rsync normally, by running cmd then c:cygwincygwin.bat, or
  • run rsync directly from the Windows command prompt:
 c:cygwinbinrsync -av -e ssh src dest_host:dest_dir