0

I have a Java Spark application that retrieves data from a Website as follows:

while(true)
        {
            try{    
                connection = (HttpURLConnection) uRL.openConnection();
                /* optional default is GET */
                connection.setRequestMethod("GET");

                /* add request header */
                connection.setRequestProperty("User-Agent", USER_AGENT);
                connection.getResponseCode();
                connection.setReadTimeout(0);
                /* Read the response code */
                bufferedReader = new BufferedReader(new InputStreamReader(connection.getInputStream(), StandardCharsets.UTF_8));
                break;
            }
            catch(Exception e){
                LOGGER.error("Error in querying Wikipedia: "+e.getMessage());
                continue;
            }
        }
        response = new StringBuffer();
        while ((inputLine = bufferedReader.readLine()) != null) {
            response.append(inputLine);
            response.append("\n");
        }
        bufferedReader.close();

This code works well on Windows.

However, on a Centos machine which has an HTTP and HTTPs proxy server, it fails with Connection Timeout. I set the system Properties for the HTTPs Proxy for the application and make sure it works for some links. However, it doesn't work for some others. For those it doesn't work, I also tried the same URL using wget on the linux server and worked.
Link that doesn't work: https://ar.wikipedia.org/w/api.php?action=query&format=xml&titles=%D9%82%D8%B1%D9%89&redirects&prop=pageprops|categories&cllimit=500
link that works: https://ar.wikipedia.org/w/api.php?action=query&format=xml&list=allpages&apnamespace=14&apfilterredir=nonredirects&aplimit=500

2 Answers 2

1

Java doesn't necessarily respect your system's default proxy settings. Since you are able to "curl" the URL on the Linux machine, the most likely explanation is that Java is not using the proxy that you have configured. The following links explains various ways to configure the proxies for Java:

4
  • Thanks for your comment. I already mentioned that it works for some links as I already set the Proxy inside the application. Commented Apr 29, 2017 at 15:05
  • I'm afraid that what you wrote in your question is unclear. In particular, you DO NOT say that the proxy is working in your Java application for some URLs and not others. What is the difference between the URLs?
    – Stephen C
    Commented Apr 29, 2017 at 15:23
  • Actually I mentioned that. Anyway, I just updated the question. Commented Apr 29, 2017 at 15:41
  • I will repeat what I said. Your question was unclear. The title says that the code works on Windows and fails on Linux. Then you said "I set the system Properties for the HTTPs Proxy for the application and make sure it works". Obviously, since your title says that "it doesn't work on Linux", that is a contradiction. Unclear. Your update now makes it clearer, but I don't have time to help you right now.
    – Stephen C
    Commented Apr 29, 2017 at 16:15
-1

I'm using Ubuntu and It worked with me

    try {
        URL obj = new URL(url);
        HttpURLConnection con = (HttpURLConnection) obj.openConnection();
        con.setRequestMethod("GET");

        //add request header
        int responseCode = con.getResponseCode();

        BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()));
        String inputLine;
        StringBuffer response = new StringBuffer();
        while ((inputLine = in.readLine()) != null) {
            response.append(inputLine);
        }
        in.close();
        System.out.println(response.toString());
    } catch (MalformedURLException e) {
        e.printStackTrace();
    } catch (ProtocolException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
3
  • what User-Agent did you use? Commented Apr 29, 2017 at 11:59
  • you don't need to set User-Agent in http get request Commented Apr 29, 2017 at 12:22
  • Actually ... it can matter. It depends on what site you are trying to talk to. However, if you need to set a User-Agent header, it is usually sign that you shouldn't be fetching the content from a Java application!
    – Stephen C
    Commented Apr 29, 2017 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.