70

I'm looking for an rsync-like program which will create any missing parent directories on the remote side.

For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.

The command:

rsync /top/a/b/c/d remote:/top/a/b/c

won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.

This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:

rsync -v -r a dest:dir  \
  --include 'a/b'       \
  --include 'a/b/c'     \
  --include 'a/b/c/d'   \
  --include 'a/b/c/d/e' \
  --exclude 'a/*'       \
  --exclude 'a/b/*'     \
  --exclude 'a/b/c/*'   \
  --exclude 'a/b/c/d/*' 

will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)

Are there any other options?

2

8 Answers 8

31

You may be looking for

rsync -aR

for example:

rsync -a --relative /top/a/b/c/d remote:/

See also this trick in other question.

6
  • 2
    Why include -a too? I'm pretty sure it's not needed. Jan 17, 2015 at 21:42
  • 1
    @KennyEvitt -a uses recursive (-r) but not relative (-R) so -a is still needed sometimes to preserve owner/perms (-aR). You can test it running rsync as root (or another user) to see this behavior.
    – dhaupin
    May 3, 2016 at 19:09
  • 8
    --relative will not create missing directory components on the remote side like in the question.
    – juzzlin
    Oct 10, 2016 at 12:42
  • 10
    Doesn't work for me, complains rsync: mkdir "/home/constantine/Projects/schifra-pkg/pkg/schifra-git/usr/include/schifra" failed: No such file or directory
    – Hi-Angel
    Jun 8, 2018 at 12:51
  • 5
    @juzzlin, according to unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative will create missing directory components on the remote side if you use . to anchor the starting parent directory to create at the destination. See my answer at stackoverflow.com/a/55231772/107158. Mar 19, 2019 at 0:11
21

As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.

From the rsync manual page (man rsync):

--mkpath                 create the destination's path component
17
rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user@remote:/tmp/imaginary/

From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.

it lets you execute arbitrary command to setup the path for rsync executables.

2
  • 1
    Very nice, this is preferable because it doesn't require a separate ssh connection. Forgot the && rsync and was getting cryptic error messages. Dec 14, 2018 at 18:20
  • this is what i tried rsync --archive --remove-source-files --rsync-path='mkdir -p /node1/node3/ && rsync' /xx/node1/node3/file.txt /node1/node3/file.txt' But throws error. node1/node3/ path doesnt exist in destination. Am i missing anything?. I am copying in the same host. So remote is not used.
    – sjd
    Oct 7, 2020 at 9:21
10

i suggest that you enforce the existence manually:

ssh user@remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c

this creates the target folder if it does not exists already.

2
  • 7
    Yes - that is one way to do it. But it seems that my use case is such a useful scenario that there should be a command for it or some options to rsync.
    – ErikR
    Aug 30, 2013 at 22:06
  • 1
    i am not aware of any such flag.
    – mnagel
    Aug 31, 2013 at 8:11
7

According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:

derek@DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek@DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek@DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/

sent 99 bytes  received 28 bytes  254.00 bytes/sec
total size is 0  speedup is 0.00
1
2

--relative does not work for me since I had different setup. Maybe I just didn't understood how --relative works, but I found that the

ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c

is easy to understand and does the job.

3
  • --relative didn't work for you, because it has nothing to do with this :)
    – juzzlin
    Oct 10, 2016 at 12:43
  • @juzzlin Maybe I just didn't understood how --relative works. ;)
    – andrej
    Oct 10, 2016 at 13:24
  • 1
    @juzzlin, according to unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination. See my answer at stackoverflow.com/a/55231772/107158. Mar 18, 2019 at 23:58
2

I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.

Simply use cp as an intermediate step with the --parents option

cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/

cp --parents will create the structure for you.

You can call it from any subfolder if you want only one subset of the parent folders to be copied.

1
  • 1
    cp: with --parents, the destination must be a directory
    – Hi-Angel
    Jun 8, 2018 at 12:53
2

A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).

'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:

ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_

1
  • 1
    I didn't like this solution at first but it looks like it's the best choice for my situation where I'm trying to create directories that don't exist in the source hierarchy. Dec 28, 2020 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.