Monthly Archives: July 2011
rsyncp – provide ssh password to rsync using expect
| 28-Jul-2011 | Posted by Sonia Hamilton under Expect, Rsync, Ssh |
A script I wrote – rsyncp. It allows you to provide a password to rsync over ssh, when you’re unable to use ssh key-based authentication:
% cat ~/bin/rsyncp
#!/usr/bin/expect
eval spawn rsync $argv
expect "*?assword:" { send "secretstuffr"}
expect eof
You can then use rsyncp in another script, like this:
for h in foo bar ; do rsyncp -av --progress srcdir/ $h:dstdir/ done
And before you add a comment saying use ssh keys or keychain, this is totally insecure, I agree with you! Key based authentication is disabled on the target server, and I’m still trying to resolve the politics around this at my current company, and sshpass seems broken <sigh>
shell – all parameters – $* vs $@
| 19-Jul-2011 | Posted by Sonia Hamilton under Shell |
Both $* and $@ expand to “all positional parameters”; in a nutshell – “$@” is often desired as it handles spaces correctly; see man bash “Special Parameters”.
% ls my* myfoo my foo myqux ---------------- % ./star_test my* $* myfoo myfoo
myqux $@ myfoo my
foo
myqux "$*" myfoo my foo myqux <-- probably not desired
"$@" myfoo my foo
myqux ---------------- % cat star_test #!/bin/bash echo '$*' ./pargs $* echo echo '$@' ./pargs $@ echo echo '"$*"' ./pargs "$*" echo echo '"$@"' ./pargs "$@" ---------------- % cat pargs #!/bin/bash for i ; do echo $i done
git – rebase and push script
| 18-Jul-2011 | Posted by Sonia Hamilton under Git |
For development projects, I use the standard git workflow ie master is my main branch, and I do development in branches (read more about this recommended workflow at A successful Git branching model**).
(** this site also suggests always using –no-ff with merge – a practice I now follow – it certainly makes reading history easier)
When I’m happy with my work in a dev branch I cherry-pick or merge into master. I’ll then want to rebase my other dev branches against master, and push all branches to a “backup” server (ie a bare git repository). On my dev branches I’m frequently rewriting history with git reset, git commit –amend, and git rebase -i, but on my master branch I never rewrite history (ie only ever use git revert). So when I push dev branches I’m going to need to do a git push -f, whereas on the master a git push will suffice.
I’m doing this regularly, and it’s boring and error prone. So I’ve written a script to automate the whole process ie rebase each dev branch against master, and push all local branches somewhere else. If you have branches off (non-master) branches, this script won’t work. But I’ve found in my day-to-day development as a sysadmin (devop?) my branching isn’t that complicated, so this script is adequate.

If you branch off (non-master) branches, don't use this script! (image: Jon Loelger, "Version Control with Git")
% cat rebase.and.push
#!/bin/bash
# vim: ai ts=4 sts=4 et sw=4 ft=sh
# rebase all local branches against master, then force push
# them to supplied remote. Also push master
original_branch=$(git symbolic-ref -q HEAD)
original_branch=${original_branch##refs/heads/}
original_branch=${original_branch:-HEAD}
if [ $original_branch = 'HEAD' ] ; then
echo
echo "On a detached head. Exiting..."
exit 1
fi
remote=${1:-origin}
# find all local branches
locals=`git br | sed 's/^ |* //' | grep -v master`
# rebase against master
for branch in $locals ; do
echo
echo "rebasing against master - $branch ..."
git co $branch
if ! [ $? -eq 0 ] ; then
echo "Unable to checkout $branch. Exiting..."
exit 1
fi
git rebase master
if ! [ $? -eq 0 ] ; then
echo "Exiting. Rebase failed..."
exit 1
fi
git push -f $remote $branch
done
echo
echo "push master..."
git co master
if ! [ $? -eq 0 ] ; then
echo "Unable to checkout master. Exiting..."
exit 1
fi
git push $remote master
echo
echo "Returning to original branch..."
git co $original_branch
ssh, sudo, tty, shell
| 14-Jul-2011 | Posted by Sonia Hamilton under Shell |
A nice little script snippet I like to remember for automating quick stuff across multiple hosts. -t forces tty (so sudo will work); -q quiet – disable banners; -c script (“command”) to su. Presumes sudo doesn’t prompt for a password; otherwise, see Expect and Exploring Expect
for h in $hosts ; do echo -e "nnhost: $hn------------------------------------" ssh -t -q $h "sudo su - -c "magic_stuff" " done
Clustering setup
| 06-Jul-2011 | Posted by Sonia Hamilton under Clustering, Slug |
A nice post from Jake on the SLUG list where he asks about a small clustering setup:
I’m setting up some new hardware for a client of mine. Basically 2 new Dell PowerEdge R210 IIs – nice and small and they seem quiet enough for an office (unlike the original R210). Anyway, quad core xeon, 2x 1Tb 7200RPM drives, 8gb ram (~$1500 each btw). The machines will be used as VM hosts (kvm) for a handfull of guests, file/domain server (ebox), PBX (piaf), mail server web server, “magic windows only application” server etc. As the guests run everything in the ofice some kind of HA is needed.
I’m thinking ganeti is the way to manage it all as it takes care of DRBD and failover giving me a nice HA-ish setup without too much work.
The questions. How should I setup my disks?
I’m tossing up between, (all the below assume a small raid 1 partition for /, this is for the storage pool) RAID 1 with mdadm giving me 1 disk read speed but the ability to read 2 streams. RAID 1 with mdadm and fancyness giving me 2x read speed for a single stream, but normal behaviour for multiple streams and 1x write.
RAID 0 with mdadm 2x read, 2x write and trust DRBD to look after my data ;->
no RAID, set up as 2 separate drives use DRBD to look after data and get “more spindles” into the mix (so say file server lives on one disk and mail server on another)
What deb based distro should I use for my host (or present a *really* compelling argument for something else)? I would use ubuntu as i’m most familiar with it but the recent debacle with unity has really hurt my confidence with them.
Should i go to debian and if so which version?
Networking
each server has 2 gbit ports on it.
should I
team the nics, then run a vlan for drbd and another for regular coms
dedicated nic for drbd and one for general coms.
Any other suggestions or gotchas to look out for?
I have my own answers to these questions, but I’d really like to hear any advice from the crowd (its like “the cloud” but open source and P2P ;->)

Recent Comments