Spring 假装不压缩响应

Question

我正在使用spring伪装来压缩请求和响应

在服务器端：

server:
  servlet:
    context-path: /api/v1/
  compression:
    enabled: true
    min-response-size: 1024

当我从 chrome 中点击 api 时，我看到它添加了 'Accept-Encoding': "gzip, deflate, br"

在客户端：

    server:
      port: 8192
      servlet:
        context-path: /api/demo



feign.compression.response.enabled: true

feign.client.config.default.loggerLevel: HEADERS

logging.level.com.example.feigndemo.ManagementApiService: DEBUG

eureka:
  client:
    enabled: false

management-api:
  ribbon:
    listOfServers: localhost:8080

当我看到请求headers通过时，feign正在传递两个headers。

Accept-Encoding: deflate
Accept-Encoding: gzip

gradle 文件

plugins {
        id 'org.springframework.boot' version '2.1.8.RELEASE'
        id 'io.spring.dependency-management' version '1.0.8.RELEASE'
        id 'java'
    }

    group = 'com.example'
    version = '0.0.1-SNAPSHOT'
    sourceCompatibility = '1.8'

    configurations {
        compileOnly {
            extendsFrom annotationProcessor
        }
    }

    repositories {
        mavenCentral()
    }

    ext {
        set('springCloudVersion', "Greenwich.SR2")
    }

    dependencies {
        implementation 'org.springframework.boot:spring-boot-starter-web'
        compile ('org.springframework.cloud:spring-cloud-starter-netflix-ribbon')
        compile('org.springframework.cloud:spring-cloud-starter-openfeign')
    // https://mvnrepository.com/artifact/io.github.openfeign/feign-httpclient
    // https://mvnrepository.com/artifact/io.github.openfeign/feign-httpclient
        //compile group: 'io.github.openfeign', name: 'feign-httpclient', version: '9.5.0'

        compileOnly 'org.projectlombok:lombok'
        annotationProcessor 'org.projectlombok:lombok'
        testImplementation 'org.springframework.boot:spring-boot-starter-test'
    }

    dependencyManagement {
        imports {
            mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
        }
    }

响应未压缩。我看到的是 Spring feign 将 "Accept-Encoding" 作为两个不同的值发送

如果这里有问题请告诉我

Answer 1

这实际上是 Tomcat 和 Jetty 中的一个例外 - 上面给出的多重编码 header 是合法的并且应该有效，但是 Tomcat 和 Jetty 有一个错误阻止他们都被阅读了。

已在 spring 引导 github here 中报告了错误。并在tomcathere供参考。

在 Tomcat 中问题已在 9.0.25 中修复，所以如果您可以更新到那个，就可以解决它。如果做不到这一点，您可以使用以下解决方法来修复它：

您需要创建自己的请求拦截器来协调您的 gzip，将 header 压缩为单个 header。

需要将此拦截器添加到 FeignClient 配置中，并将该配置添加到您的 feign 客户端中。

import feign.RequestInterceptor;
import feign.RequestTemplate;
import feign.template.HeaderTemplate;
import java.lang.reflect.Field;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;

/**
 * This is a workaround interceptor based on a known bug in Tomcat and Jetty where
 * the requests are unable to perform gzip compression if the headers are in collection format.
 * This is fixed in tomcat 9.0.25 - once we reach this version we can remove this class
 */
@Slf4j
public class GzipRequestInterceptor implements RequestInterceptor {
    @Override
    public void apply(RequestTemplate template) {
        // don't add encoding to all requests - only to the ones with the incorrect header format
        if (requestHasDualEncodingHeaders(template)) {
            replaceTemplateHeader(template, "Accept-Encoding", Collections.singletonList("gzip,deflate"));
        }
    }

    private boolean requestHasDualEncodingHeaders(RequestTemplate template) {
        return template.headers().get("Accept-Encoding").contains("deflate")
                && template.headers().get("Accept-Encoding").contains("gzip");
    }

    /** Because request template is immutable, we have to do some workarounds to get to the headers */
    private void replaceTemplateHeader(RequestTemplate template, String key, Collection<String> value) {
        try {
            Field headerField = RequestTemplate.class.getDeclaredField("headers");
            headerField.setAccessible(true);
            ((Map)headerField.get(template)).remove(key);
            HeaderTemplate newEncodingHeaderTemplate = HeaderTemplate.create(key, value);
            ((Map)headerField.get(template)).put(key, newEncodingHeaderTemplate);
        } catch (NoSuchFieldException e) {
            LOGGER.error("exception when trying to access the field [headers] via reflection");
        } catch (IllegalAccessException e) {
            LOGGER.error("exception when trying to get properties from the template headers");
        }
    }
}

我知道上面看起来有点矫枉过正，但是因为模板 header 是 unmodifiable，我们只需使用一点反射来将它们修改为我们想要的方式。

将上述拦截器添加到您的配置 bean

import feign.RequestInterceptor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class FeignGzipEncodingConfiguration {

    @Bean
    public RequestInterceptor gzipRequestInterceptor() {

        return new GzipRequestInterceptor();
    }
}

您终于可以使用配置注释参数将其添加到您的 feign 客户端中

@FeignClient(name = "feign-client", configuration = FeignGzipEncodingConfiguration.class)
public interface FeignClient {
    ...
}

当您发送 feign-client 压缩信息请求时，现在应该会命中请求拦截器。这将擦除对偶 header，并以 gzip,deflate

的形式写入一个可接受的字符串连接

Answer 2

几周前我遇到了同样的问题，我开始知道没有 fruitful/straight 解决方法。我还知道当@patan 向 spring 社区 @patan reported issue1 and @patan reported issue2 there was a ticket created for the tomcat side to attempt to fix the issue (issue link). There has been also a ticket (ticket link) present in the Jetty side related to the same. Initially, I planned to use the approach suggested in github 报告问题时，但后来才知道该库已经合并到 [=16= 下的 spring-cloud-openfeign-core jar 中] 包裹。然而，我们并没有达到预期的压缩效果，面临着以下两个挑战：

当我们通过设置 org.springframework.cloud.openfeign.encoding.FeignAcceptGzipEncodingInterceptor (code-link) class adds the Accept-Encoding header with values as gzip and deflate but due to the issue (ticket) 启用假压缩时，tomcat 服务器无法将其解释为压缩信号的标志。作为解决方案，我们必须添加手动 Feign 解释器来覆盖
FeignAcceptGzipEncodingInterceptor 功能并连接 headers.
Feign 的默认压缩设置在最简单的情况下完美工作，但是当出现 Client calling microservice and that microservice calling another microservice through feign 的情况时，feign 无法处理压缩响应，因为 Spring cloud open feign decoder 不默认解压缩响应 (default spring open feign decoder) which eventually ends with the issue (issue link)。所以我们要自己写解码器来实现解压。

我终于找到了一个基于各种可用资源的解决方案，所以只需按照spring假压缩的步骤操作即可：

application.yml

spring:
  http:
    encoding:
      enabled: true

#to enable server side compression
server:
  compression:
    enabled: true
    mime-types:
      - application/json
    min-response-size: 2048

#to enable feign side request/response compression
feign:
  httpclient:
    enabled: true
  compression:
    request:
      enabled: true
      mime-types:
        - application/json
      min-request-size: 2048
    response:
      enabled: true

注意：上面的feign配置我默认对所有feign客户端启用压缩。

CustomFeignDecoder


import feign.Response;
import feign.Util;
import feign.codec.Decoder;
import org.springframework.cloud.openfeign.encoding.HttpEncoding;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.reflect.Type;
import java.nio.charset.StandardCharsets;
import java.util.Collection;
import java.util.Objects;
import java.util.zip.GZIPInputStream;

public class CustomGZIPResponseDecoder implements Decoder {

    final Decoder delegate;

    public CustomGZIPResponseDecoder(Decoder delegate) {
        Objects.requireNonNull(delegate, "Decoder must not be null. ");
        this.delegate = delegate;
    }

    @Override
    public Object decode(Response response, Type type) throws IOException {
        Collection<String> values = response.headers().get(HttpEncoding.CONTENT_ENCODING_HEADER);
        if(Objects.nonNull(values) && !values.isEmpty() && values.contains(HttpEncoding.GZIP_ENCODING)){
            byte[] compressed = Util.toByteArray(response.body().asInputStream());
            if ((compressed == null) || (compressed.length == 0)) {
               return delegate.decode(response, type);
            }
            //decompression part
            //after decompress we are delegating the decompressed response to default 
            //decoder
            if (isCompressed(compressed)) {
                final StringBuilder output = new StringBuilder();
                final GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
                final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, StandardCharsets.UTF_8));
                String line;
                while ((line = bufferedReader.readLine()) != null) {
                    output.append(line);
                }
                Response uncompressedResponse = response.toBuilder().body(output.toString().getBytes()).build();
                return delegate.decode(uncompressedResponse, type);
            }else{
                return delegate.decode(response, type);
            }
        }else{
            return delegate.decode(response, type);
        }
    }

    private static boolean isCompressed(final byte[] compressed) {
        return (compressed[0] == (byte) (GZIPInputStream.GZIP_MAGIC)) && (compressed[1] == (byte) (GZIPInputStream.GZIP_MAGIC >> 8));
    }
}

FeignCustomConfiguration

import feign.RequestInterceptor;
import feign.RequestTemplate;
import feign.optionals.OptionalDecoder;
import org.springframework.beans.factory.ObjectFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.http.HttpMessageConverters;
import org.springframework.cloud.openfeign.support.ResponseEntityDecoder;
import org.springframework.cloud.openfeign.support.SpringDecoder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class CustomFeignConfiguration {


    @Autowired
    private ObjectFactory<HttpMessageConverters> messageConverters;

    //concatenating headers because of https://github.com/spring-projects/spring-boot/issues/18176
    @Bean
    public RequestInterceptor gzipInterceptor() {
        return new RequestInterceptor() {
            @Override
            public void apply(RequestTemplate template) {
                template.header("Accept-Encoding", "gzip, deflate");
            }
        };
    }

    @Bean
    public CustomGZIPResponseDecoder customGZIPResponseDecoder() {
        OptionalDecoder feignDecoder = new OptionalDecoder(new ResponseEntityDecoder(new SpringDecoder(this.messageConverters)));
        return new CustomGZIPResponseDecoder(feignDecoder);
    }
}

其他提示

如果您打算仅使用 feign-core 个库构建 CustomDecoder


import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.type.TypeFactory;
import feign.Response;
import feign.Util;
import feign.codec.DecodeException;
import feign.codec.Decoder;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.util.StringUtils;
import org.springframework.web.client.HttpMessageConverterExtractor;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
import java.lang.reflect.WildcardType;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.LinkedList;
import java.util.Map;
import java.util.Objects;
import java.util.zip.GZIPInputStream;

import static java.util.zip.GZIPInputStream.GZIP_MAGIC;

public class CustomGZIPResponseDecoder implements Decoder {

    private final Decoder delegate;

    public CustomGZIPResponseDecoder(Decoder delegate) {
        Objects.requireNonNull(delegate, "Decoder must not be null. ");
        this.delegate = delegate;
    }

    @Override
    public Object decode(Response response, Type type) throws IOException {
        Collection<String> values = response.headers().get("Content-Encoding");
        if (Objects.nonNull(values) && !values.isEmpty() && values.contains("gzip")) {
            byte[] compressed = Util.toByteArray(response.body().asInputStream());
            if ((compressed == null) || (compressed.length == 0)) {
                return delegate.decode(response, type);
            }
            if (isCompressed(compressed)) {
                Response uncompressedResponse = getDecompressedResponse(response, compressed);
                return getObject(type, uncompressedResponse);
            } else {
                return getObject(type, response);
            }
        } else {
            return getObject(type, response);
        }
    }

    private Object getObject(Type type, Response response) throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        if (response.status() == 404 || response.status() == 204)
            return Util.emptyValueOf(type);
        if (Objects.isNull(response.body()))
            return null;
        if (byte[].class.equals(type))
            return Util.toByteArray(response.body().asInputStream());
        if (isParameterizeHttpEntity(type)) {
            type = ((ParameterizedType) type).getActualTypeArguments()[0];
            if (type instanceof Class || type instanceof ParameterizedType
                    || type instanceof WildcardType) {
                @SuppressWarnings({"unchecked", "rawtypes"})
                HttpMessageConverterExtractor<?> extractor = new HttpMessageConverterExtractor(
                        type, Collections.singletonList(new MappingJackson2HttpMessageConverter(mapper)));
                Object decodedObject = extractor.extractData(new FeignResponseAdapter(response));
                return createResponse(decodedObject, response);
            }
            throw new DecodeException(HttpStatus.INTERNAL_SERVER_ERROR.value(),
                    "type is not an instance of Class or ParameterizedType: " + type);
        } else if (isHttpEntity(type)) {
            return delegate.decode(response, type);
        } else if (String.class.equals(type)) {
            String responseValue = Util.toString(response.body().asReader());
            return StringUtils.isEmpty(responseValue) ? Util.emptyValueOf(type) : responseValue;
        } else {
            String s = Util.toString(response.body().asReader());
            JavaType javaType = TypeFactory.defaultInstance().constructType(type);
            return !StringUtils.isEmpty(s) ? mapper.readValue(s, javaType) : Util.emptyValueOf(type);
        }
    }

    public static boolean isCompressed(final byte[] compressed) {
        return (compressed[0] == (byte) (GZIP_MAGIC)) && (compressed[1] == (byte) (GZIP_MAGIC >> 8));
    }

    public static Response getDecompressedResponse(Response response, byte[] compressed) throws IOException {
        final StringBuilder output = new StringBuilder();
        final GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
        final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, StandardCharsets.UTF_8));
        String line;
        while ((line = bufferedReader.readLine()) != null) {
            output.append(line);
        }
        return response.toBuilder().body(output.toString().getBytes()).build();
    }

    public static String getDecompressedResponseAsString(byte[] compressed) throws IOException {
        final StringBuilder output = new StringBuilder();
        final GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
        final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gis, StandardCharsets.UTF_8));
        String line;
        while ((line = bufferedReader.readLine()) != null) {
            output.append(line);
        }
        return output.toString();
    }

    private boolean isParameterizeHttpEntity(Type type) {
        if (type instanceof ParameterizedType) {
            return isHttpEntity(((ParameterizedType) type).getRawType());
        }
        return false;
    }

    private boolean isHttpEntity(Type type) {
        if (type instanceof Class) {
            Class c = (Class) type;
            return HttpEntity.class.isAssignableFrom(c);
        }
        return false;
    }

    private <T> ResponseEntity<T> createResponse(Object instance, Response response) {

        MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
        for (String key : response.headers().keySet()) {
            headers.put(key, new LinkedList<>(response.headers().get(key)));
        }

        return new ResponseEntity<>((T) instance, headers, HttpStatus.valueOf(response
                .status()));
    }

    private class FeignResponseAdapter implements ClientHttpResponse {

        private final Response response;

        private FeignResponseAdapter(Response response) {
            this.response = response;
        }

        @Override
        public HttpStatus getStatusCode() throws IOException {
            return HttpStatus.valueOf(this.response.status());
        }

        @Override
        public int getRawStatusCode() throws IOException {
            return this.response.status();
        }

        @Override
        public String getStatusText() throws IOException {
            return this.response.reason();
        }

        @Override
        public void close() {
            try {
                this.response.body().close();
            } catch (IOException ex) {
                // Ignore exception on close...
            }
        }

        @Override
        public InputStream getBody() throws IOException {
            return this.response.body().asInputStream();
        }

        @Override
        public HttpHeaders getHeaders() {
            return getHttpHeaders(this.response.headers());
        }

        private HttpHeaders getHttpHeaders(Map<String, Collection<String>> headers) {
            HttpHeaders httpHeaders = new HttpHeaders();
            for (Map.Entry<String, Collection<String>> entry : headers.entrySet()) {
                httpHeaders.put(entry.getKey(), new ArrayList<>(entry.getValue()));
            }
            return httpHeaders;
        }
    }

}

如果你打算构建自己的 Feign 构建器，那么你可以像下面这样配置

 Feign.builder().decoder(new CustomGZIPResponseDecoder(new feign.optionals.OptionalDecoder(new feign.codec.StringDecoder())))
                 .target(SomeFeignClient.class, "someurl");

更新以上答案： 如果您计划将 spring-cloud-openfeign-core 的依赖版本更新为 'org.springframework.cloud:spring-cloud-openfeign-core:2.2.5.RELEASE'，请注意 FeignContentGzipEncodingAutoConfiguration class 中的以下更改。在 FeignContentGzipEncodingAutoConfiguration class 中，ConditionalOnProperty 注释的签名从 @ConditionalOnProperty("feign.compression.request.enabled", matchIfMissing = false) 到 @ConditionalOnProperty(value = "feign.compression.request.enabled")，因此默认情况下 FeignContentGzipEncodingInterceptor bean 将被注入到 spring 容器中，如果您的环境中有应用程序属性 feign.request.compression=true 并压缩如果 default/configured 超出大小限制，则请求 body。如果您的服务器没有处理压缩请求的机制，这会导致问题，在这种情况下 add/modify the 属性 as feign.request.compression=false

Answer 3

如果您使用的是最新的 spring 引导版本，那么它会提供默认的 Gzip 解码器，因此无需编写您的自定义解码器。请改用下面的属性：-

feign:
  compression:
    response:
      enabled: true
      useGzipDecoder: true

Spring 假装不压缩响应

Spring Feign Not Compressing Response

spring

spring-boot

spring-cloud-feign