I would like to get files from a remote machine using rsync and SSH (from within a Python program).
How do you start a local instance of rsync and introduce it to the SSH channel I have opened with Paramiko?
Join Stack Overflow to learn, share knowledge, and build your career.
This is an old question, but still the first hit in Google when searching for "rsync over paramiko", and the only up voted item here is a comment that is unrelated to the OP's question (the link in that comment points to using ControlMaster which is not supported by Paramiko).
There is an example of how to setup local port forwarding in one of the Paramiko demos here. There is also an easier to use version of it in a pull request. You can use the local port forwarding in two ways:
rsync
over local port using ssh protocol. This is equivalent to running ssh -L12345:localhost:22 remote-host
followed by rsync --rsh 'ssh -p 12345' sourcedir/file localhost:/destdir/file
.rsync
over local port using rsync protocol. This is similar to running ssh -L12345:localhost:12345
followed by rsync sourcedir/file rsync://localhost:12345/module/destdir/file
, except that you need to setup an ad-hoc rsync daemon running on 12345
with a module name that points to the destdir
on the remote host.I personally prefer the 2nd approach above, though it is slightly more complex because, it skips the local ssh
client and also uses rsync protocol which I think is more efficient than using ssh
.
Using the ForwardServer
from the above pull request, the code would look somewhat like this (depends on Fabric
):
RSYNC_SPEC = """
port=12345
use chroot=false
[homedir]
log file=/tmp/rsync-ad-hoc.log
max verbosity=4
path=/home/{user}/
read only=false
"""
@task
def rsync(local_path, remote_path):
"""
local_path: Absolute path to a local source
remote_path: Relative path (from home directory) to a remote destination
"""
with ForwardServer(0, "localhost", rsync_port, connections[env.host_string].get_transport()) as serv:
local_port = serv.socket.getsockname()[1]
run("killall rsync; rm -f /tmp/rsync-ad-hoc.log /tmp/rsync-ad-hoc.conf; :")
put(local_path=StringIO(RSYNC_SPEC.format(user=env.user)), remote_path="/tmp/rsync-ad-hoc.conf", )
run("rsync --daemon --config /tmp/rsync-ad-hoc.conf")
remote_rsync_path = os.path.join("rsync://localhost:%s/homedir" % local_port, remote_path)
# Rsync expects the root of the destination to exist.
run("mkdir -p /home/{user}/{path}".format(user=env.user, path=remote_path))
logging.info("Attempting rsync from (localhost, %s, %s) to (%s, %s, %s)", local_port, local_path, env.host_string, rsync_port, remote_path)
local("rsync -avzPh --delete %s/ %s/" % (local_path, remote_rsync_path))
You can also have the function take a remote absolute path and generate the directory for the module (instead of assuming that it is relative to the user's home directory).
I had problems with paramiko and rsync that I was not able to work out. I routinely use paramiko successfully with a number other commands (e.g., mkdir, mpstat, service, local python programs, etc), using this typical framework:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(domain, username="root",timeout=5)
stdin, mpstat, stderr = client.exec_command('mpstat')
x=mpstat.readlines()
# process the lines as though it is a file
With respect to rsync, rather than pursue a paramiko solution, I reverted to:
x='''ssh root@%s "rsync -aRzq root@%s:%s /root/Backups/%s/%s " \
'''%(BackupServerIP, ServerIP, file, Service, Server)
os.system(x)
I generally prefer paramiko because it is easy to process its outputs, so I would be interested to know if there is a problem using it with rsync or I just didn't persist long enough.
rsync
withsftp
feature from Paramiko? – Itachi Feb 25 '20 at 14:56