4

I would like to run a small http proxy server on my machine which sends all requests to an upstream / cascading proxy server.

Twisted Web HTTP Proxy -> Squid -> Internet connection

Unfortunately I cannot figure out how to send all ProxyClient requests upstream / cascade them to Squid. Could you give me a hint?

Do I have to use reactor.connectTCP and HttpClientFactory for this and somehow direct the Squid response through?

I have used the following Twisted code sample to get the twisted http proxy up:

from twisted.web import proxy, http
from twisted.internet import reactor
from twisted.python import log
import sys
log.startLogging(sys.stdout)

class ProxyFactory(http.HTTPFactory):
    protocol = proxy.Proxy

reactor.listenTCP(8080, ProxyFactory())
reactor.run()

Thanks,

Mathias

5
  • Is this just a curiosity / learning exercise? Or do you have a specific purpose in mind?
    – MattH
    Commented Jan 10, 2011 at 16:47
  • Hi MattH, I would like to use two proxy servers (running in 2 VirtualBox / 2 VMware - each VM bounded to a single network interface) and combine the DSL download speed of both interfaces. I would like to rotate the proxy on each new connection request (count and rotate IP) Commented Jan 10, 2011 at 18:24
  • Hey MattH, I have also tried the ClearOS distribution, which should support MultiWan, but the connection speed did not double using both interfaces. Moreover I tried to change the routing metric in Windows 7 on each new request but it seems the OS is still selecting a preferred interface. I would like to run a 10 connection download over 2 proxy server / 5 connection each proxy. Many thanks, Mathias Commented Jan 10, 2011 at 18:32
  • This 10 connection download, do you mean to download a single file using 10 slices concurrently?
    – MattH
    Commented Jan 11, 2011 at 9:40
  • Yes, using a download manager where you configure twisted as proxy. Twisted would send the request to an alternate upstream proxy server and the connection should run over several HTTP proxies at the same time. Commented Jan 11, 2011 at 9:52

2 Answers 2

1

I think you're over complicating things. To me, it sounds like what you want to do is round-robin load balance http proxy requests between two squid services.

If it was me, I wouldn't write a byte of code and use something off the shelf.

I'd use HA-Proxy with a configuration something like this:

global
        daemon
        maxconn 256

defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

frontend loadbal
        bind *:3128
        default_backend squids
        option http_proxy

backend squids
        option http_proxy
        server squid1 192.168.1.2:3128
        server squid2 192.168.1.3:3128
        balance roundrobin

You might need to do some playing around, some webservices may require certain types of stickiness, but this application is highly configurable in this regard. It looks like HA-Proxy v 1.4 supports keep alive connections, which may simplify some of the corner cases I can imagine.

Oh, also, if you're trying to get the most out of your two connections, I'd suggest that you configure ICP between your two squid boxes.

2
  • Thanks Matth, I was not aware of the HA-Proxy software. Round robin is exactly what I would like to do. Many thanks Commented Jan 11, 2011 at 12:01
  • @mathias: You can be the second person I point towards this pertinent article about The 3 Basic Tools of Systems Engineering
    – MattH
    Commented Jan 11, 2011 at 16:20
-1

This sounds like something twisted.web.proxy.ReverseProxyResource might be good for.

3
  • Hi Jean-Paul, thank you for your prompt reply. I have tried your example code with the following line: site = server.Site(proxy.ReverseProxyResource('www.google.com',80,'')) Each time google comes back with this error message: 404 Not Found The requested URL /ttp%3A//www.google.com/ was not found on this server. Is this a bug? Commented Jan 10, 2011 at 18:18
  • How can I redirect the request to only load google.com homepage? In case of my proxy would I need to write site = server.Site(proxy.ReverseProxyResource(PROXY_IP,PROXY_PORT,URL_TO_REQUEST_FROM_PROXY))? Many thanks, Mathias Commented Jan 10, 2011 at 18:22
  • The client issuing requests to your proxy wants to be talking to a different kind of proxy, I guess. ReverseProxyResource isn't appropriate after all. Commented Jan 11, 2011 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.