Garrett helped write svnsync as part of
svnsync is a one way replication system for Subversion. It allows you to create a read-only replica of a repository over any RA layer (including http, https, svn, svn+ssh).
First, lets setup the initial sync. We have two repositories, I will skip the details of svnadmin create. For the remote access to the replica repository, I used svnserve, and I added a user with full write access. The destination repository should be completely empty before starting.
So, to make this easier, I am going to put the repository URIs into enviroment variables:
$ export FROMREPO=svn://svn.example.com/ $ export TOREPO=svn://dest.example.com/
Because the svnsync is allowed to rewrite anything on the TOREPO, we need to make sure the commit hooks are configured to allow our ’svnsync’ user to do anything it wants.
On the server hosting TOREPO, I ran this:
$ echo "#!/bin/sh" > hooks/pre-revprop-change $ chmod 755 hooks/pre-revprop-change
Now we are ready to setup the sync:
$ svnsync init ${TOREPO} ${FROMREPO}
This will prompt you for the username, password, and also sets several revision properties on the $TOREPO, for revision zero. It doesn’t actually copy any of the data yet. To list the properties that it created, run:
$ svn proplist --revprop -r 0 ${TOREPO}
svn:sync-from-uuid
svn:sync-last-merged-rev
svn:date
svn:sync-from-url
$ svn propget svn:sync-from-url --revprop -r 0 ${TOREPO}
svn://svn.example.com/
So all the knowledge about what we are syncing from is stored at the destination repository. No state about this sync is stored in the source repository.
We are now ready to begin copying data:
$ svnsync --non-interactive sync ${TOREPO}
And if everything is setup correctly, you will start replicating data.
Except, I suck. And the first thing I did was hit control+c. I figured this is a cool replication system, so I just ran the sync command from above again, and got this:
$ svnsync --non-interactive sync ${TOREPO}
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
Failed to get lock on destination repos, currently held by 'svn.example.com:0e4e0d98-631d-0410-9a00-9320a90920b3'
svnsync: Couldn't get lock on destination repos after 10 attempts
Oh snap. I guess its not so easy to restart after an aborted sync.
I started debugging, and found that svnsync kept its lock state in a special property in revision zero again.
So, To fix this, we can safely just delete this lock:
$ svn propdelete svn:sync-lock --revprop -r 0 ${TOREPO}
Now running sync again works! Hurrah!
After the sync finishes, we will want to keep the replica up to date.
I personally set a ‘live’ sync, but it is also possible to use a crontab or other scheduling method to invoke sync whenever you want.
To setup a live sync, on the FROMREPO server, I appended this to my hooks/post-commit file:
svnsync --non-interactive sync svn://dest.example.com/ &
You will want to make sure that the user-running subversion (and the hook script) has a cached copy of the authentication info for the destination repository.
Unfortunately, the post-commit hook won’t catch everything, so we also need to added this to the post-revprop-change hook:
svnsync --non-interactive copy-revprops svn://dest.example.com/ ${REV} &
This will help propagate things like editing svn:log messages.
And there you go, thats the path I took to mirror one of my repositories onto another machine.
Have you tried svk (http://svk.elixus.org/)?
Did you heard about svn:external?
Yeah, Ive tried SVK, and it can ‘mirror’ a repository, but the biggest flaw with using SVK as a mirroring tool is that it doens’t preserve revision numbers prefectly in all cases.
Now I’ve got it sync’d, what can I do with it?
I’d like to be able to browse the history of a file while offline; is there a way to tell svn that the working copy I have is mirrored locally?
And what are the long range plans? It would make lots of sense to me to delete the text-base originals from my working copy if I’ve got a copy of the repos available.
SVK also fails to produce 100% identical copies in some cases, due to the tricks it needs to play to figure out what’s going on in a revision from the log and diff APIs. Eventually it’ll be upgraded to take advantage of the new replay API we added for svnsync, but for now it’s not going to get you a 100% copy.
please note that on windows, as stated in the svn book, the commit hooks is done creating a blank file in the hooks/ directory named ‘pre-revprop-change.bat’
worrying about having a cached copy of credentials for teh secondary server is not necessary if you specify the –username and –password in the post-commit hook.
Hi,
i tried your Howto, but it failed always, can you help me?
when i go this way like you:
$ touch hooks/pre-revprop-change
$ chmod 755 hooks/pre-revprop-change
i get after the init command the fallowing error:
svnsync: ‘pre-revprop-change’ hook failed with error output:
The output is empty, i think because the file, which is touched is also empty!?
when i copy the hooks/pre-revprop-change .tmpl to hooks/pre-revprop-change and make it executable i get this error, after the init command:
svnsync: ‘pre-revprop-change’ hook failed with error output:
Changing revision properties other than svn:log is prohibited
can you write me an answer if you know it?
thanks for your help!
ben
Wow, I didn’t realise the Subversion guys had done this – I’ve just run across it. It’s something I’d thought about before at a passing glance, but now I’m going to give it a go.
Thanks for the article.
Ben:
Yeah, instead of just making an empty file, put “#!/bin/sh” inside it at the top, and it should fix it.
Someone mentioned to me the other day that he keeps two Subversion repositories in sync by actually checking the repository into a second “meta-repository”. Interesting idea, but surely that is fraught with danger?
btw, if you pass the –verbose flag to svn proplist, it will show you the value of each property.
svn proplist –revprop –verbose -r 0 ${TOREPO}
Thanks for providing this documentation. There isn’t much else to be found. Evidently there will be more with the 1.4.1 release.
oh, and it’s “propdel”, not “propdelete”
I followed the instructions showed in http://
>>> journal.paul.querna.org/articles/2006/09/14/using-svnsync and
>>> http://svn.collab.net/repos/svn/trunk/notes/svnsync.txt and was
>>> able to sync my repo using the following command:
>>>
svnsync init file:///export3/svnrep-bak/repos/svnsync/test file:///export3/svnrep/repos/svnsync/test
BUT, whenever I use URLs instead of file locations I get the error:
svnsync init http://ny390-sf01a:9050/svn/svnsync-bak/test http://ny390-sf01a:9050/svn/svnsync/test
svnsync: DAV request failed; it’s possible that the repository’s pre-revprop-change hook either failed or is non-existent
svnsync: At least one property change failed; repository is unchanged
My apache installations works fine since I am able to do normal repo operations (checkout, commit, update, etc) using URLs; permissions are set properly in destination repo as well.
Does somebody have any clue on what’s wrong ?
[quote]
svnsync –non-interactive copy-revprops svn://dest.example.com/ ${REV} &
[/quote]
Anyone knows what is happening when a revprop is changed on a ${REV} which is not yet synchronized?
@Grégoire: Well, its whatever the old state is. Rev Props them selves are not versioned… so there isn’t a way to keep a ‘history’ of them.
@Chip
Thank you for the reply. I’m setting up a repository mirror with svnsync and I’m a bit worried that the revision properties can change at any time. To be a perfect mirror, the only way to use svnsync is to put it in the post-commit hook and pre-revprop-change hook. Starting svnsync sinc once every x time to synchornize a mirror will most probably miss the revprop changes. And if svnsync copy-revprops is used in the pre-revprop-change, then svnsync sync has to be called from post-commit hook. Otherwise you put your mirror at risk. For instance:
1. svnsync sync is called from a script once every hour, svnsync copy-revprops called from the pre-revprop-change hook:
- svnsync sync is called. Mirror is updated to rev x one hour of delay. During that time:
- commit in master of rev x+1
- revprop change to modify svn:log in master for rev x+1
- svnsync copy-revprops called for rev x+1 <= x+1 is not yet commited in the mirror.
@Chip [...]
2. svnsync copy-revprops is not called from pre-revprop-change
- svnsync sync is called. Mirror is updated to rev x one hour of delay. During that time:
- commit in master of rev x+1
- revprop change to modify svn:log in master for rev x+1
- revprop change to modify svn:log in master for rev x-5
- svnsync sync is called. Mirror is updated to rev x+1 <= revprop change for rev x+1 will be populated correctly, not the revprop change for rev x-5
@Chip [...]
If you have any suggestions, they will be highly appreciated.
Grégoire.
Thanks, that was just what I was looking for!!
Unfortunately, I ran into a problem, I ran the
- svnsync init ${TOREPOS} ${FROMREPOS} which run just fine,
but the
-svnscync sync ${TOREPOS} gave an unknown, at least to me, message:
-svnsync: Server doesn’t support the replay command
-svnsync: Unknown command ‘replay’
Is this maybe because the ${FROMREPOS} is still running svn 1.2.3?
Thanks a lot. I just added a link to this great howto on my blog, because many of my friends are searching for this solution without svk a long time
.
My sychronization run was interrupted by a nightly backup program holding onto one of the files needed. And I had the same problem you described upon restarting the sync job. Thanks for the propdel solution that you gave, except that the svn subcommand that worked for me was ‘propdel’, not ‘propdelete’. I was prompted to do a ’svn help’ and it was quite easy to pick the right subcommand then. I’m on svn 1.4.4, Windows. Just want to share this with you.
I think post-revprop-change needs to have this:
REV=”$2″
in it in order for ${REV} to have the correct value.
Thanks for the how-to!
I used
ln -s `which true` hooks/pre-revprop-change
to create an always-succeeding pre-revprop-change hook.
Hey Paul,
Thanks for the great article but I have a slightly different problem.
I have a destination repository which is available on the public URL (http) but my source directory is in LAN. So the approache I think would not work for me as my destination server will not be able to access the source repository.
Is it possible to mirror the source directory (Within a LAN) by running commands in the source machine as source machine can access the destination machine?
Let me know if something is not clear.
Pankaj
Creating an empty pre-revprop-change.bat file in the hooks directory didn’t work on our windows slave repo. But after adding the following line to the file we succeded:
@exit 0
Thanks Paul, like you I have just hit Ctrl+C, and you helped me fix it quickly.
Cheers! Robin
Hello,
My svn server (visualSVN) post-commit hook required valid username and password such as:
svnsync sync file:///Repositories/mirror –sync-username myuser –sync-password mypassword
exit 0
Difficult for me to find because it worked fine by command line
Paul you are a legend! Thanks!
If you have no access to hooks on destination mirror (like in the case of SourceForge.net SVN repositories, you might find AXE SVN Sync as a valid alternative to svnsync, since it does not require exclusive access to the destination repository).