5

Our team recently had some problems with Jenkins Plugin Manager installing / updating plugins. After we fixed some problems with our firewall and proxy rules (we are behind a corporate firewall and a proxy), we still had issues downloading some of the plugin files.

Some plugin files were downloaded and installed successfully, others failed with the following error:

Failed to install git-client
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
    at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3057)
    at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
    at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
    at java.net.URLConnection.getContentLength(URLConnection.java:485)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)
Caused: java.net.SocketTimeoutException: Read timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)
Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi to /var/lib/jenkins/plugins/git-client.jpi.tmp
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)
Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi (redirected to: http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218)
    at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766)
    at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037)
    at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
    at java.lang.Thread.run(Thread.java:748)

To diagnose potential network problems, I tried downloading git-client.hpi directly from the machine hosting the Jenkins server via curl, which was successful:

[root@servername ~]# curl -vk -x http://proxyserver:7689 http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi -o /tmp/git-client.hpi
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to proxy proxyserver port 6789 (#0)
*   Trying 10.100.50.100...
* Connected to proxyserver (10.100.50.100) port 6789(#0)
> GET http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ftp-nyc.osuosl.org
> Accept: */*
> Proxy-Connection: Keep-Alive
>
  0     0    0     0    0     0      0      0 --:--:--  0:01:01 --:--:--     0< HTTP/1.1 200 OK
< Date: Mon, 11 Nov 2019 08:18:55 GMT
< Server: Apache
< Last-Modified: Sat, 02 Nov 2019 12:03:46 GMT
< ETag: "7dd268-5965bdfed67ce"
< Accept-Ranges: bytes
< Content-Type: application/vnd.hp-hpid
< Age: 6915
< Content-Length: 8245864
< Via: 1.1 proxyserver
<
  0 8052k    0     0    0     0      0      0 --:--:--  0:01:01 --:--:--     0{ [data not shown]
100 8052k  100 8052k    0     0   130k      0  0:01:01  0:01:01 --:--:-- 2132k
* Connection #0 to host proxyserver left intact

Downloading via curl was successful, although I noticed that after successful connection, the download process was stalled at 0% for about a minute. Then it suddenly started, and finished in seconds.

EDIT: this is not an insfrastructure problem, and has nothing to do with INFRA-2393, nor any other problems related to update site infrastructure:

  • This read timeout was encountered first in October 2019, and could be reproduced in the middle of December 2019. Infrastucture problems tend to get solved a lot faster than that.

  • I also managed to reproduce the problem at about 2020-01-02 10:30 CET, with /plugins/subversion/2.13.0/subversion.hpi using a 60 sec read timeout setting, long after INFRA-2393 got resolved. I also managed to fix the problem by raising the read timeout setting as described below.

  • The read timeout only happens within Jenkins. Using the same download link in a browser or in a curl command downloads the affected files. This is because Jenkins has a default read timeout of 60 seconds, while browsers / curl do not.

  • While diagnosing this earlier, we rearranged our network policy and made Jenkins reach the update sites directly (without using a proxy). In such a context, all the problem downloads worked. This indicates the stalling download is not related to an update site bug, but to our own proxy.


Also, stop asking me to file a report on Jenkins Jira. The very reason I posted this question and my answer is that I got frustrated with trying to find a solution using Jenkins Jira. The material that I found related to my problem there roughly fell into two categories:

  • Infrastructure issues dating back several years, obviously fixed years ago. In other words: noise.

  • Genuine Jenkins issues posted with similar symptoms, with dozens / hundreds of comments posted, mostly boiling down to Jenkins staff stating that the problem is on the user's side, and that people should fix their own corporate firewall / proxy / etc. Not a word related to timeout values whatsoever. In other words: a lot more noise.

Again: this is not a problem related to Jenkins update sites, nor is it a Jenkins software bug. If it's related to anything, then it's the lack of documentation. Exactly the thing that SO is meant to cover IMHO.

2
  • While @attila-csipak may have useful guidance, this should be raised as an INFRA issue on the Jenkins JIRA. SO is not the right forum for an issue like this.
    – Ian W
    Dec 20, 2019 at 9:05
  • A similar issue now appears to have been reported there as INFRA-2393
    – Ian W
    Dec 20, 2019 at 10:21

2 Answers 2

11

The important thing to notice from the above is that this is not a connection timeout but a read timeout. The connection to the download mirror is estabilished, but the client does not receive a single byte for about a minute. (We hypothesised that this happens because our proxy analyses the contents of git-client.hpi before forwarding the bytes to the client that requested the download.)

The curl download is successful because there's no read timeout set in the command. Jenkins obviously uses a 60 second default socket read timeout, and so after there's no traffic for 60 seconds, it considers the connection dead and fails with a SocketException.

I did not find any official documentation on this, but looking at Jenkins source code confirms the theory: the UpdateCenter.PLUGIN_DOWNLOAD_READ_TIMEOUT constant is initialized from a property value, and defaults to 60 in its absence.

The other thing I did not find is a comprehensive documentation of Jenkins' properties (key names, possible values, description, etc.).

The exact property key to be set is: hudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds

AFAIK Jenkins does not have any *.properties file, but accepts property settings on startup as JVM parameters.

I solved this problem by finding the jenkins options file (if you use RHEL, and installed Jenkins from package repository, the exact path is: /etc/sysconfig/jenkins), and appending -Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=120 to the value of the JENKINS_JAVA_OPTIONS setting.

UPDATE: I found the official Jenkins documentation about Jenkins system properties, which also has a few words about the hudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds property.

UPDATE (2022-07-08):

Beginning with Jenkins 2.332.1 and Jenkins 2.335, the Linux package installers use systemd to manage services. The RPM and deb package installers migrate configuration settings from System V init to systemd overrides.

This means that configuration files formerly used (/etc/sysconfig/jenkins or /etc/default/jenkins, depending on distribution) are now outdated and systemd unit files should be used to set system properties. See Jenkins documentation.

1
  • 3
    if you install jenkins in windows ,you can add "-Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=120" to <arguments> in jenkins/jenkins.xml, Feb 9, 2020 at 13:48
0

I think it's a global problem with site https://updates.jenkins.io/. The packages are currently updated (the last modified time is still changing) so maybe they have a realease what cause a time out issue.

We noticed the same problem but the time out is shown for different packages (not with a specific package) every time.

2
  • I see everything is back to normal. All plugin files were downloaded and installed successfully.
    – Iwona
    Dec 20, 2019 at 13:06
  • We encountered this problem on various different days this fall and last week. The other thing to note is, that it always failed with the same files (i.e. git-client.hpi), and was always successful with others. Also, we were always able to download via another network (cell phone connection) and from workstation browsers on the same network. Neither had a read timeout set. Dec 21, 2019 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.