105

I have a folder a/ and a remote folder A/. I now run something like this on a Makefile:

get-music:
 rsync -avzru server:/media/10001/music/ /media/Incoming/music/

put-music:
 rsync -avzru /media/Incoming/music/ server:/media/10001/music/

sync-music: get-music put-music

when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.

This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.

In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.

If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).

Thing is I have 3 machines in context:

  1. desktop
  2. notebook
  3. home-server

So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.

I guess this is only possible with a database and track of operations :P

Any simpler solutions? Thank you.

4
  • 1
    I have this same problem. Did you find a solution, mwm?
    – mouche
    Commented Feb 12, 2011 at 23:53
  • 7
    i actually did now. i was trying to achieve a dropbox like experience and came up with a solution that involves unison (that uses rsync) and lsyncd for monitoring file changes. i also found other people with this solution and more tricks on top » cerebralmastication.com/2011/04/fast-two-way-sync-in-ubuntu
    – mwm
    Commented Apr 26, 2011 at 1:56
  • nowadays I'm using nextcloud locally on my home server. perfect "dropbox" sollution 100% managed by me. has android/ios/win/macos/linux/web clients.
    – mwm
    Commented Mar 13, 2018 at 16:34
  • 1
    @mwm : as I have already noted at superuser.com/questions/34697/… , this is a misunderstanding : unison does not "use" rsync in any normally meaningful way. It is surprising how many people keep believing this.
    – q.undertow
    Commented May 16, 2022 at 2:12

11 Answers 11

69

Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/

Syntax:

unison dirA/ dirB/

Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:

unison -auto dirA/ dirB/

unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).

Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.

Note 2: I am now currently using Mutagen for some purposes. It still doesn't cover the question's use case completely, but it does implement two-way sync.

14
  • 15
    Unison is pretty easy to use for small directories. But it is excruciatingly SLOW for directories with a large number of files. Where rsync will start copying almost right away, unison sometimes takes over 12 hours to start copying, because it scans every file before copying anything. Commented Jun 16, 2014 at 22:25
  • 1
    It git good at this ?At least git is easy to use and transport is fast.for blob there is an extension developed by github called Large File Storage.
    – wener
    Commented Jun 3, 2015 at 8:58
  • 5
    @wener Don't use a hammer for a screw, or a screwdriver for a nail. :)
    – ADTC
    Commented Aug 22, 2016 at 3:11
  • 3
    If you're synchronizing e.g. a music library where you care about 2-way file deletion but don't care about file content conflicts, you can use Unison -fastercheckUNSAFE flag to considerably speed up Unison. It will not compute the file content hash but just look at timestamp/length and overwite if newer. See the manual
    – KrisWebDev
    Commented Jun 4, 2017 at 10:17
  • 5
    unison is great - but also very unreliable. In about 50% of my use cases it simple has not worked. The most common cause of this is poor inter version capability. I sync sometimes with content on a computer I have no privileges on. If that server happens to have different version than I do, unison fails. In addition, I've recently had it fail between unison clients of the same version, because they were using different compiled versions of some library (this was between an ubuntu computer and a raspberry pi).
    – argentum2f
    Commented Feb 9, 2018 at 22:20
9

I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.

9

Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.

It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.

FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.

  • On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops. I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
  • Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.
2
  • Does not work over a network. Why would I want to sync two local folders?
    – mjs
    Commented Jan 26, 2022 at 14:44
  • @mjs I have added the answer to the above reply Commented Feb 7, 2022 at 3:01
8

You could also try bitpocket: https://github.com/sickill/bitpocket

6

Try this,

get-music:
 rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/

put-music:
 rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/

sync-music: get-music put-music

I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).

5
  • 5
    Won't this delete new local files? -u skips files newer on the destination, but rsync doesn't have a way of tracking the age of a delete, so how could this script differentiate between when a local file is new (and should be kept) or old (and should be deleted)? Or am I misunderstanding? Commented Aug 26, 2013 at 19:08
  • Does --delete-excluded delete any files that exist on destination directories but not source directories? I thought you had to define --exclude=<filename> so won't rsync only delete these excludes? Or does rsync treat all non-source directory files as "excludes"? Commented Sep 17, 2013 at 4:12
  • 2
    Found the answer: In addition to deleting the files on the receiving side that are not on the sending side, this tells rsync to also delete any files on the receiving side that are excluded (see --exclude) rsync.samba.org/ftp/rsync/rsync.html Commented Sep 17, 2013 at 4:53
  • 2
    From what I understand, if I create a file on the destination and then execute the rsync command to copy from source to destination, it will delete the file at the destination, since no time stamps are used. This isn't what you want right?
    – elexhobby
    Commented Feb 11, 2014 at 19:08
  • 10
    @Lübnah: Correct; this solution is not a full 2-way sync: Because rsync with --delete-excluded is run with the server as the source first: files added locally will be deleted during that run (whereas files added to the server are copied to the local target). The only local files synced back to the server are files that also exist on the server and were modified locally (more recently than the server versions) - which is not likely in the OP's scenario. As you hint at, methinks true 2-way syncing can't be done with rsync, because saving state between runs is required.
    – mklement0
    Commented Jun 6, 2014 at 23:56
5

You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.

4

You could try csync, it is the sync engine under the hood of owncloud.

1
1

I'm now using SparkleShare https://www.sparkleshare.org/

works on mac, linux and windows.

1
  • as time goes by, I keep updating my solution. I'm now using nextcloud on my home server. syncing with the nextcloud client on linux, mac, windows, android and ios.
    – mwm
    Commented Aug 8, 2018 at 21:02
1

Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.

3
  • 1
    It does not support two way sync: rclone.org/commands/rclone_sync Commented Dec 18, 2019 at 21:26
  • 2
    Looks like rclone does support two-way (bidirectional) sync now: rclone.org/commands/rclone_bisync Commented Apr 14, 2022 at 15:27
  • Aye, rclone seems to work quite well in its bidirectional mode! I'm a fan of all those tools — rsync, unison, rclone — but lately I've been using rclone most, since I can pretty much do bidirectional sync with basically any protocol out there... Commented Dec 19, 2022 at 12:29
0

I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.

1
  • 2
    The short equivalent of option --recurse is -r, which the OP is using (embedded in compressed-options group -avzru).
    – mklement0
    Commented Jun 7, 2014 at 0:04
0

I've built a lightweight, simple-to-use and easy to configure daemon called livesync, check it out at https://github.com/brauliobo/ruby-livesync

It relies on pyinotify to watch and sync from remote to local, reusing the same persist ssh connection

Not the answer you're looking for? Browse other questions tagged or ask your own question.