I use the following bash script to copy only files of certain extension(in this case *.sh), however it still copies over all the files. what's wrong?
from=$1 to=$2 rsync -zarv --include="*.sh" $from $to
Join Stack Overflow to learn, share knowledge, and build your career.
I think --include
is used to include a subset of files that are otherwise excluded by --exclude
, rather than including only those files.
In other words: you have to think about include meaning don't exclude.
Try instead:
rsync -zarv --include "*/" --exclude="*" --include="*.sh" "$from" "$to"
For rsync version 3.0.6 or higher, the order needs to be modified as follows (see comments):
rsync -zarv --include="*/" --include="*.sh" --exclude="*" "$from" "$to"
Adding the -m
flag will avoid creating empty directory structures in the destination. Tested in version 3.1.2.
So if we only want *.sh files we have to exclude all files --exclude="*"
, include all directories --include="*/"
and include all *.sh files --include="*.sh"
.
You can find some good examples in the section Include/Exclude Pattern Rules of the man page
rsync -zarv --include="*/" --include="*.sh" --exclude="*" "$from" "$to"
.
– Bijou Trouvaille
Jun 3 '13 at 9:17
--include=\*.sh
then --exclude=\*
)
– TrueY
Nov 28 '14 at 9:26
The answer by @chepner will copy all the sub-directories irrespective of the fact if it contains the file or not. If you need to exclude the sub-directories that dont contain the file and still retain the directory structure, use
rsync -zarv --prune-empty-dirs --include "*/" --include="*.sh" --exclude="*" "$from" "$to"
Here's the important part from the man page:
As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped.
To summarize:
Also, something ending with a slash is matching directories (like find -type d
would).
Let's pull apart this answer from above.
rsync -zarv --prune-empty-dirs --include "*/" --include="*.sh" --exclude="*" "$from" "$to"
.sh
filesFinally, the --prune-empty-directories
keeps the first rule from making empty directories all over the place.
--include "*/"
allow any file to be sync'd that is in any directory? Or does matching a file require matching on a directory pattern AND matching on a file pattern?
– FlexMcMurphy
Jan 6 at 23:07
a '*' matches any path component, but it stops at slashes.
"
– Jim Hunziker
Jan 7 at 16:55
One more addition: if you need to sync files by its extensions in one dir only (without of recursion) you should use a construction like this:
rsync -auzv --include './' --include '*.ext' --exclude '*' /source/dir/ /destination/dir/
Pay your attention to the dot in the first --include
. --no-r
does not work in this construction.
EDIT:
Thanks to gbyte.co for the valuable comment!
include
and the exclude
options in the order which they were specified in. In addition to this, it stops at the first matched one. So, if we specify the --exclude '*'
at the first place in this example the rsync will do nothing. See the man for more explanations.
– Serge Roussak
Jun 14 '18 at 15:33
-- include './'
is saying include everything in the source directory path? Then the next one ` --include '.ext'` include the specific file in the source path named .ext
and then the exclude says don't send anything else --exclude '*'
? Is that correct?
– Charlie Parker
Jun 14 '18 at 16:23
Wrote this handy function and put in my bash scripts or ~/.bash_aliases
. Tested sync'ing locally on Linux with bash and awk
installed. It works
selrsync(){
# selective rsync to sync only certain filetypes;
# based on: https://stackoverflow.com/a/11111793/588867
# Example: selrsync 'tsv,csv' ./source ./target --dry-run
types="$1"; shift; #accepts comma separated list of types. Must be the first argument.
includes=$(echo $types| awk -F',' \
'BEGIN{OFS=" ";}
{
for (i = 1; i <= NF; i++ ) { if (length($i) > 0) $i="--include=*."$i; } print
}')
restargs="$@"
echo Command: rsync -avz --prune-empty-dirs --include="*/" $includes --exclude="*" "$restargs"
eval rsync -avz --prune-empty-dirs --include="*/" "$includes" --exclude="*" $restargs
}
short handy and extensible when one wants to add more arguments (i.e. --dry-run
).
selrsync 'tsv,csv' ./source ./target --dry-run
If someone looks for this…
I wanted to rsync only specific files and folders and managed to do it with this command: rsync --include-from=rsync-files
With rsync-files:
my-dir/
my-file.txt
- /*
rsync
, this can very well be achieved with the shell internals? – Inian Jun 14 '18 at 4:08-r
is redundant because-a
implies-r
– wisbucky Nov 24 '20 at 4:10