0

I'm trying to use urllib2 over a proxy to scrap a web page that isn't directly available (it's running in the remote server's local network and isn't externally accessible). The proxy I'd prefer is a SSH SOCKS proxy (like you get if you run ssh -D 9090 server), both because I have access to this and because it's fairly secure.

I've had a poke around with paramiko but everything I find points to running a SSH connection out over SOCKS, which is the opposite of what I'm actually trying to accomplish here.

I have seen the Transport class but this only does dumb forwarding and doesn't provide a nice OpenSSH-SOCKS proxy interface that I can latch onto with SocksiPy (et al).

Net::SSH::Socks for Ruby is exactly what I'm looking for in the wrong language. Is there anything available in Python that provides a proxy over SSH?

1 Answer 1

0

I have a workaround that works for scraping. Instead of trying to use the SSH connection, I'm using the remote shell to pull out the data:

from bs4 import BeautifulSoup
import paramiko

ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.connect('example.com', username='Oli', look_for_keys=True, timeout=5)

stdin, stdout, stderr = ssh.exec_command('/usr/bin/wget -qO- "%s"' % url)
soup = BeautifulSoup(stdout)

ssh.close()

This isn't what I was looking for to begin with (and I'd still very much like to see if there's a way of connecting a SOCKS socket in over SSH) but there is some elegance in its simplicity.

1
  • Did you find any other way? I need same functionality as with ssh -D option.
    – BigSack
    Jan 28, 2014 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.