23

After I modified Consuming a RESTful Web Service example to call get users by id from api.stackexchange.com I get JsonParseException:

com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 31)): only regular white space (\r, \n, \t) is allowed between tokens

Response from api.stackexchange.com is gzip compressed.

How to add support for gzip compressed response into Spring-Web RestTemplate?

I am using Spring boot parent ver. 1.3.1.RELEASE hence Spring-Web 4.2.4-RELEASE

Here’s my adjusted example:

User.java

package stackexchange.dto;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;

@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(LowerCaseWithUnderscoresStrategy.class)
public class User {

    // Properties made public in order to shorten the example
    public int userId;
    public String displayName;
    public int reputation;

    @Override
    public String toString() {
        return "user{"
                + "display_name='" + displayName + '\''
                + "reputation='" + reputation + '\''
                + "user_id='" + userId + '\''
                + '}';
    }
}

CommonWrapper.java

package stackexchange.dto;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;

@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(LowerCaseWithUnderscoresStrategy.class)
public class CommonWrapper {

    // Properties made public in order to shorten the example
    public boolean hasMore;
    // an array of the type found in type
    public User[] items;
    public int page;
    public int pageSize;
    public int quotaMax;
    public int quotaRemaining;

    @Override
    public String toString() {
        StringBuilder sb = new StringBuilder();
        for (User user : items) {
            sb.append("{" + user.toString() + "}\n");
        }

        return "common_wrapper{"
        + "\"items\"=[\n"
        + sb
        + "]"
        + "has_more='" + hasMore + '\''
        + "page='" + page + '\''
        + "page_size='" + pageSize + '\''
        + "quota_max='" + quotaMax + '\''
        + "quota_remaining='" + quotaRemaining + '\''
        + '}';
    }
}

StackExchange.java

package stackexchange;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.web.client.RestTemplate;

import stackexchange.dto.CommonWrapper;

import com.fasterxml.jackson.databind.PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;

@JsonNaming(LowerCaseWithUnderscoresStrategy.class)
public class StackExchange implements CommandLineRunner{

    private static final Logger log = LoggerFactory.getLogger(StackExchange.class);

    public static void main(String args[]) {
        SpringApplication.run(StackExchange.class);
    }

    @Override
    public void run(String... strings) throws Exception {

        RestTemplate restTemplate = new RestTemplate();
        CommonWrapper response = restTemplate
                .getForObject(
                        "https://api.stackexchange.com/2.2/users/4607349?site=stackoverflow",
                        CommonWrapper.class);

        log.info(response.toString());
    }

}

pom.xml - same as in example

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>stackexchangetest</groupId>
  <artifactId>stackexchangetest</artifactId>
  <version>0.0.1</version>
  <name>stackexchangetest</name>
  <description>api.stackexchange.com Test</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.1.RELEASE</version>
    </parent>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>   
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>
3
  • 1
    did you read the doc : docs.spring.io/autorepo/docs/spring-android/1.0.x/reference/… ???
    – ben75
    Dec 22, 2015 at 12:44
  • 1
    Yes, I read it. It’s solution for Spring Android, while I use Spring Web. HttpHeaders does not contain setAcceptEncoding method in Spring Web. I will state it in the question to avoid confusion. Thx Dec 22, 2015 at 14:06
  • I love how meta this question is.
    – mR_fr0g
    Jul 20, 2017 at 8:48

3 Answers 3

63

Replace the default requestFactory with one from Apache HttpClient (which decodes GZIP on the fly):

HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory(
            HttpClientBuilder.create().build());
RestTemplate restTemplate = new RestTemplate(clientHttpRequestFactory);

Add Apache Http Client into pom.xml

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <!--Version is not needed when used with Spring Boot parent pom file -->
    <version>4.5.1</version>
</dependency>
6
  • Thanks Michal it worked. Usual documentation describe android not spring web as you said. Why don't you also add reference if there is any blog or link for it? Apr 18, 2017 at 18:35
  • Great, I am happy I could help. I did not find any article or a blog post at the time. Feel free to document it and let Google to properly index it, so that nobody will struggle again :) Apr 18, 2017 at 19:46
  • Hi Michal are you aware of any disadvantage of using HttpComponentsClientHttpRequestFactory in multithreaded environment. I mean when hits from users are very high. Apr 19, 2017 at 10:04
  • No I am not aware of any issues. I find HttpComponentsClientHttpRequestFactory much better than the default factory. It better handles cookies, redirects, proxy servers, etc. Apr 19, 2017 at 10:12
  • 1
    Linking this article because it actually explains how/where httpclient decompresses the response if necessary. See @Garry answer on this post
    – Norbert
    Feb 9, 2019 at 17:30
1
private String callViaRest(String requestString, Steps step) {
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.TEXT_XML);
    headers.add("Accept-Encoding", "application/gzip");
    HttpEntity<String> entity = new HttpEntity<String>(requestString, headers);

    byte[] responseBytes = jsonRestTemplate
            .exchange("yourUrl", HttpMethod.POST, entity, byte[].class).getBody();
    String decompressed = null;
    try {
        decompressed= new String(CompressionUtil.decompressGzipByteArray(responseBytes),Charsets.UTF_8);
    } catch (IOException e) {
        LOGGER.error("network call failed.", e);
    }
    return decompressed;
}
3
  • Add dependency for org.apache.commons.jcs to pom. import org.apache.commons.jcs.utils.zip.CompressionUtil; May 5, 2017 at 10:42
  • by adding headers.add("Accept-Encoding", "application/gzip"); working for me thanks for solution.
    – Kishor K
    Jun 25, 2017 at 21:42
  • I know it's been a long time since this answer was posted but I have a question. What about responseBody when a HttpStatusCodeException is thrown? I mean, with the factory posted in the accepted answer you can get the response body if an HttpStatusCodeException is thrown by exchange method, but with this answer, I wasn't able to recover response in that error case.
    – Anne
    Jan 29, 2019 at 16:45
1

I wanted to solve the same issue without using additional libraries. What helped me was to

  1. use the responseType of byte[].class
  2. unzipping response myself
  3. mapping the entity myself by ObjectMapper mapper.

It is not the most elegant solution but it works.

    ResponseEntity<byte[]> response = restTemplate.exchange(url, HttpMethod.GET, new HttpEntity<byte[]>(createHeaders()), byte[].class);
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    try (GZIPInputStream gzipInputStream = new GZIPInputStream(new ByteArrayInputStream(response.getBody()))) {
        gzipInputStream.transferTo(byteArrayOutputStream);
    }
    byte[] content = byteArrayOutputStream.toByteArray();
    GenericResponseDto question = mapper.readValue(content, GenericResponseDto .class);  
    log.info("Response :" + question.toString());

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.