16

I'm using Paramiko to tail -f a file on a remote server.

Previously, we were running this via ssh -t, but that proved flaky, and the -t caused issues with our remote scheduling system.

My question is how to kill tail when the script catches a SIGINT?

My script (based on Long-running ssh commands in python paramiko module (and how to end them))

#!/usr/bin/env python2
import paramiko
import select

client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username='victorhooi', password='blahblah')
transport = client.get_transport()
channel = transport.open_session()

channel.exec_command("tail -f /home/victorhooi/macbeth.txt")
while True:
    try:
        rl, wl, xl = select.select([channel],[],[],0.0)
        if len(rl) > 0:
            # Must be stdout
            print channel.recv(1024)
    except KeyboardInterrupt:
        print("Caught control-C")
        client.close()
        channel.close()
        exit(0)

The script catches my Ctrl-C successfully, and ends. However, it leaves the tail -f process running on the remote system,.

Neither client.close() nor channel.close() seem to terminate it.

What command can I issue in the except block to kill it?

The remote server is running Solaris 10.

8 Answers 8

8

You should use ssh keepalives... the problem you have is that the remote shell has no way of knowing (by default) that your ssh session was killed. Keepalives will enable the remote shell to detect that you killed the session

client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username='victorhooi', password='blahblah')
transport = client.get_transport()
transport.set_keepalive(1)   # <------------------------------
# ... carry on as usual...

Set the keepalive value as low as you like (even 1 second)... after several seconds, the remote shell will see that the ssh login died, and it will terminate any processes that were spawned by it.

4
  • 1
    Hmm, I tried adding the set_keepalive(5) line - still no luck =(. The "tail -f" process still remains after Ctrl-C-ing the Python script, even after waiting for 5 seconds (or longer). Urgh. The remote server is Solaris 10, btw, if that matters at all.
    – victorhooi
    Oct 12, 2011 at 3:00
  • You'll need to wait for several keepalive intervals... set your keepalive to 1, and I'm pretty sure you'll see it terminate within 10 seconds (if Solaris behaves like my BSD boxes) Oct 12, 2011 at 3:01
  • @victorhooi you should edit the question with any relevant details like that.
    – brc
    Oct 12, 2011 at 3:02
  • @victorhooi, you it may make things easier for you if you explicitly invoke a shell with channel.invoke_shell()... although this will require more modifications to your script Oct 12, 2011 at 3:19
7

There is one way to do this. It works like on the shell

ssh -t commandname

The option -t is opening a pseudo pty to help ssh to track how long this process should last. the same can be done via pormiko via

channel.get_pty()

prior to execute_command(...). This will not open a shell like it does with channel.invoke_shell(), it just requests such a pseudo interface to tie all processes to. The effect can also be seen if ps aux is issued on the remote machine, the process is now attached to the sshd with a ptxXY interface.

2
  • Note that this works, but only if the program you're running behaves nicely when given a SIGHUP
    – num1
    Jul 25, 2014 at 0:15
  • Setting get_pty to True works. I can cancel my processes using Ctrl + C. But it introduces another problem. Now exec_command will timeout after about 10minutes. I am unable to run command which are taking more than 10 minutes. Jul 17, 2023 at 14:23
4

I just hit this issue and wasn't in a position to issue a pkill to close the process at the end.

A better solution is to change the command you are running to:

tail -f /path/to/file & { read ; kill %1; }

This will let you run your tail command for as long as you need. As soon as you send a newline to the remote process the kill %1 will execute and stop the tail command you backgrounded. (for reference: %1 is a jobspec and used to describe the first process that has been backgrounded in your session, ie the tail command)

3

Here's a way to obtain the remote process ID:

def execute(channel, command):
    command = 'echo $$; exec ' + command
    stdin, stdout, stderr = channel.exec_command(command)
    pid = int(stdout.readline())
    return pid, stdin, stdout, stderr

And here's how to use it (replace ... with the bits in the original question):

pid, _, _, _ = execute(channel, "tail -f /home/victorhooi/macbeth.txt")
while True:
    try:
        # ...
    except KeyboardInterrupt:
        client.exec_command("kill %d" % pid)
        # ...
2
  • This is actually getting the pid of the remote shell, not the pid of the command that is run in the remote shell, right? May 7, 2014 at 12:08
  • @DavidDoria Same thing. :-) The use of exec means that the executed command inherits the shell PID. May 11, 2014 at 17:13
1

While not the most efficient method, this should work. After you CTRL+C; In the KeyboardInterrupt handler you could exec_command("killall -u %s tail" % uname) like so:

#!/usr/bin/env python2

import paramiko
import select

import time
ltime = time.time()

# Or use random:
# import random
# ltime = random.randint(0, 500)

uname = "victorhooi"
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username=uname, password='blahblah')
transport = client.get_transport()
channel = transport.open_session()

channel.exec_command("tail -%df /home/victorhooi/macbeth.txt" % ltime)
while True:
    try:
        rl, wl, xl = select.select([channel],[],[],0.0)
        if len(rl) > 0:
            # Must be stdout
            print channel.recv(1024)
    except KeyboardInterrupt:
        print("Caught control-C")
        channel.close()
        try:
            # open new socket and kill the proc..
            client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))
        except:
            pass
    
        client.close()
        exit(0)

This would kill any open processes named tail. That may cause issues though if you have tails open that you dont want to close, if thats the case you could grep a ps, get the pid and kill -9 it.

First, set tail to read n lines from end of file before following. set n to a unique nuber like time.time(), since tail doesn't care if that number is larger then the number of lines in the file, the large number from time.time()shouldnt cause issues and will be unique. Then grep for that unique number in the ps:

   client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))
4
  • That might not work for us - we have many other tail processes running on the remote server for other things. Is there perhaps a way we can get the PID of the tail when running exec_command, save it, then kill it that way? (Yes, all these remote tails are somewhat messy - I'm hoping to replace it with a proper bus like RabbitMQ, but util then we have to make do).
    – victorhooi
    Oct 12, 2011 at 3:07
  • See my edits above, the formatting is all weird in this comment ).
    – chown
    Oct 12, 2011 at 3:11
  • Yup, that solution works. As you note however, it is a bit hacky (the whole grepping for PIDs, particularly if multiple people tail the same file). Still it works. I'm surprised there isn't a way to pull the PID via Paramiko, as you normally can with Python's POpen. Strange.
    – victorhooi
    Oct 12, 2011 at 23:37
  • That solution is not a pythonic solution.. You should never have to spawn a process on the remote machine to kill the command you already spawned... What if that command needs to be killed also? What if you are running an ssh server on a machine that doesn't have the kill command? Aug 13, 2013 at 1:01
1

Had the same problem with ssh -t. There is a library called closer - it runs a remote process via ssh and closes automatically for you. Check it out.

0

Specifically for 'tail' you could use the --pid=PID argument and let tail take care of it:

  --pid=PID  with -f, terminate after process ID, PID dies
0

You can use get_pty as described in https://stackoverflow.com/a/38883662/565212.

E.g. scenario - When to call client/channel.close():
Step1: Execute a remote command that writes to the log file.
Step2: Spawn a thread that executes the tail command and blocks in the readline loop
Step3: In main thread, when the command returns, you know there will be no more logs, kill the tail thread.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.