I am trying to figure out a mind-numbing apache issue where requests are going through two layers of apache reverse proxies before hitting upstream services. Most traffic seems to make it through fine. The notable exception is websockets.
In particular, this is a test request
curl -i -H 'Connection: Upgrade' -H 'Upgrade: websocket' localhost:80/test.html
When proxying requests from port 80 to port 8080, I notice (using tcpdump
and Wireshark) that the Upgrade
header has been removed and Connection: Keep-Alive
has been set instead. Moreover, any attempts I have made at resetting the Connection
header to Upgrade and Upgrade: websocket
have been impotent.
Note that the upstream service needs Connection: Upgrade
and Upgrade: websocket
to initiate the websocket (I get a 404 error without those headers).
Why is Apache forcing Connection: Keep-Alive
when proxying to itself? Is there any way to force it to pass along Connection / Upgrade headers or set those values manually? RequestHeader
and friends have unfortunately not been helpful. Without Connection
and Upgrade
making it through the reverse proxy, the upstream service pukes and throws a 404/Not found at the websocket endpoint.
ServerRoot "/usr/local/apache2"
Listen 80
Listen 8080
LogLevel rewrite:trace8
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule mime_module modules/mod_mime.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule auth_mellon_module modules/mod_auth_mellon.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule ssl_module modules/mod_ssl.so
<IfModule unixd_module>
User daemon
Group daemon
</IfModule>
<VirtualHost *:80>
LogLevel rewrite:trace8
ServerName localhost
RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket
RewriteRule /(.*) ws://localhost:8080/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket
RewriteRule /(.*) http://localhost:8080/$1 [P,L]
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
</VirtualHost>
<VirtualHost *:8080>
LogLevel rewrite:trace8
ServerName localhost
UseCanonicalName On
RewriteEngine On
ProxyPass / http://proxy-debug:8080/
ProxyPassReverse / http://proxy-debug:8080/
ProxyRequests Off
</VirtualHost>
ServerAdmin [email protected]
ErrorLog /proc/self/fd/2
DocumentRoot "/"