Our team recently had some problems with Jenkins Plugin Manager installing / updating plugins. After we fixed some problems with our firewall and proxy rules (we are behind a corporate firewall and a proxy), we still had issues downloading some of the plugin files.
Some plugin files were downloaded and installed successfully, others failed with the following error:
Failed to install git-client
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3057)
at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
at java.net.URLConnection.getContentLength(URLConnection.java:485)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)
Caused: java.net.SocketTimeoutException: Read timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)
Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi to /var/lib/jenkins/plugins/git-client.jpi.tmp
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)
Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi (redirected to: http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
at java.lang.Thread.run(Thread.java:748)
To diagnose potential network problems, I tried downloading git-client.hpi directly from the machine hosting the Jenkins server via curl, which was successful:
[root@servername ~]# curl -vk -x http://proxyserver:7689 http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi -o /tmp/git-client.hpi
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to proxy proxyserver port 6789 (#0)
* Trying 10.100.50.100...
* Connected to proxyserver (10.100.50.100) port 6789(#0)
> GET http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ftp-nyc.osuosl.org
> Accept: */*
> Proxy-Connection: Keep-Alive
>
0 0 0 0 0 0 0 0 --:--:-- 0:01:01 --:--:-- 0< HTTP/1.1 200 OK
< Date: Mon, 11 Nov 2019 08:18:55 GMT
< Server: Apache
< Last-Modified: Sat, 02 Nov 2019 12:03:46 GMT
< ETag: "7dd268-5965bdfed67ce"
< Accept-Ranges: bytes
< Content-Type: application/vnd.hp-hpid
< Age: 6915
< Content-Length: 8245864
< Via: 1.1 proxyserver
<
0 8052k 0 0 0 0 0 0 --:--:-- 0:01:01 --:--:-- 0{ [data not shown]
100 8052k 100 8052k 0 0 130k 0 0:01:01 0:01:01 --:--:-- 2132k
* Connection #0 to host proxyserver left intact
Downloading via curl was successful, although I noticed that after successful connection, the download process was stalled at 0% for about a minute. Then it suddenly started, and finished in seconds.
EDIT: this is not an insfrastructure problem, and has nothing to do with INFRA-2393, nor any other problems related to update site infrastructure:
This read timeout was encountered first in October 2019, and could be reproduced in the middle of December 2019. Infrastucture problems tend to get solved a lot faster than that.
I also managed to reproduce the problem at about 2020-01-02 10:30 CET, with /plugins/subversion/2.13.0/subversion.hpi using a 60 sec read timeout setting, long after INFRA-2393 got resolved. I also managed to fix the problem by raising the read timeout setting as described below.
The read timeout only happens within Jenkins. Using the same download link in a browser or in a curl command downloads the affected files. This is because Jenkins has a default read timeout of 60 seconds, while browsers / curl do not.
While diagnosing this earlier, we rearranged our network policy and made Jenkins reach the update sites directly (without using a proxy). In such a context, all the problem downloads worked. This indicates the stalling download is not related to an update site bug, but to our own proxy.
Also, stop asking me to file a report on Jenkins Jira. The very reason I posted this question and my answer is that I got frustrated with trying to find a solution using Jenkins Jira. The material that I found related to my problem there roughly fell into two categories:
Infrastructure issues dating back several years, obviously fixed years ago. In other words: noise.
Genuine Jenkins issues posted with similar symptoms, with dozens / hundreds of comments posted, mostly boiling down to Jenkins staff stating that the problem is on the user's side, and that people should fix their own corporate firewall / proxy / etc. Not a word related to timeout values whatsoever. In other words: a lot more noise.
Again: this is not a problem related to Jenkins update sites, nor is it a Jenkins software bug. If it's related to anything, then it's the lack of documentation. Exactly the thing that SO is meant to cover IMHO.