使用 REST 模板从服务器下载大文件 Java Spring MVC
Download Large file from server using REST template Java Spring MVC
我有一个 REST 服务向我发送了一个大的 ISO 文件，REST 服务没有问题。
现在我已经编写了一个调用其余服务来获取文件的 Web 应用程序，在客户端（Web 应用程序）端我收到内存不足 Exception.Below 是我的代码
HttpHeaders headers = new HttpHeaders();//1 Line
headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
headers.set("Content-Type","application/json");//3 Line
headers.set("Cookie", "session=abc");//4 Line
HttpEntity statusEntity=new HttpEntity(headers);//5 Line
String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line
ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line
我在第 7 行收到内存不足异常，我想我必须缓冲并获取部分内容，但不知道如何从服务器获取此文件，文件大小约为 500 到 700 MB。
任何人都可以帮忙.
异常堆栈：
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause
java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:3236)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)
我工作正常的服务器端 REST 服务代码是
@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{
byte[] reportBytes = null;
File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
if(result.exists()){
InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
response.setHeader("Content-Type",type);
reportBytes=new byte[100];//New change
OutputStream os=response.getOutputStream();//New change
int read=0;
while((read=inputStream.read(reportBytes))!=-1){
os.write(reportBytes,0,read);
}
os.flush();
os.close();
}
这会阻止将整个请求加载到内存中。
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate rest = new RestTemplate(requestFactory);
对于java.lang.OutOfMemoryError：Java heap space 可以解决添加更多内存到JVM：
-Xmxn Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB. Append the
letter k or K to indicate kilobytes, or m or M to indicate megabytes.
The default value is chosen at runtime based on system configuration.
For server deployments, -Xms and -Xmx are often set to the same value.
See Garbage Collector Ergonomics at
http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m
您遇到的问题可能与您尝试执行的请求（下载大文件）没有严格关系，但为进程分配的内存不足。
您应该使用多部分文件附件，这样文件流就不会加载到内存中。
在此示例中，我使用了通过 Apache CXF 实现的休息服务。
...
import org.apache.cxf.jaxrs.ext.multipart.Attachment;
...
@Override
@Path("/put")
@Consumes("multipart/form-data")
@Produces({ "application/json" })
@POST
public SyncResponseDTO put( List<Attachment> attachments) {
SyncResponseDTO response = new SyncResponseDTO();
try {
for (Attachment attr : attachments) {
log.debug("get input filestream: " + new Date());
InputStream is = attr.getDataHandler().getInputStream();
这是我的做法。基于此 Spring Jira issue.
的提示
RestTemplate restTemplate // = ...;
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);
来自上述 Jira 问题：
Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.
更新 Spring 5
Spring 5 介绍了 WebClient class 允许异步（例如非阻塞）http 请求。来自文档：
By comparison to the RestTemplate, the WebClient is:
- non-blocking, reactive, and supports higher concurrency with less hardware resources.
- provides a functional API that takes advantage of Java 8 lambdas.
- supports both synchronous and asynchronous scenarios.
- supports streaming up or down from a server.
要在 Spring 引导中获得 WebClient，您需要此依赖项：
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
目前，我坚持使用 RestTemplate，因为我不想为了访问 WebClient.
而引入另一个依赖项
上述正确答案的更好版本可能是以下代码。此方法会将下载请求发送到另一个应用程序或服务，作为下载信息的实际来源。
public void download(HttpServletRequest req, HttpServletResponse res, String url)
throws ResourceAccessException, GenericException {
try {
logger.info("url::" + url);
if (restTemplate == null)
logger.info("******* rest template is null***********************");
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<ResponseEntity<InputStream>> responseExtractor = response -> {
String contentDisposition = response.getHeaders().getFirst("Content-Disposition");
if (contentDisposition != null) {
// Temporary location for files that will be downloaded from micro service and
// act as final source of download to user
String filePath = "/home/devuser/fileupload/download_temp/" + contentDisposition.split("=")[1];
Path path = Paths.get(filePath);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
// Create a new input stream from temporary location and use it for downloading
InputStream inputStream = new FileInputStream(filePath);
String type = req.getServletContext().getMimeType(filePath);
res.setHeader("Content-Disposition", "attachment; filename=" + contentDisposition.split("=")[1]);
res.setHeader("Content-Type", type);
byte[] outputBytes = new byte[100];
OutputStream os = res.getOutputStream();
int read = 0;
while ((read = inputStream.read(outputBytes)) != -1) {
os.write(outputBytes, 0, read);
}
os.flush();
os.close();
inputStream.close();
}
return null;
};
restTemplate.execute(url, HttpMethod.GET, requestCallback, responseExtractor);
} catch (Exception e) {
logger.info(e.toString());
throw e;
}
}
如@bernie所述，您可以使用 WebClient 来实现此目的：
public Flux<DataBuffer> downloadFileUrl( ) throws IOException {
WebClient webClient = WebClient.create();
// Request service to get file data
return Flux<DataBuffer> fileDataStream = webClient.get()
.uri( this.fileUrl )
.accept( MediaType.APPLICATION_OCTET_STREAM )
.retrieve()
.bodyToFlux( DataBuffer.class );
}
@GetMapping( produces = MediaType.APPLICATION_OCTET_STREAM_VALUE )
public void downloadFile( HttpServletResponse response ) throws IOException
{
Flux<DataBuffer> dataStream = this.downloadFileUrl( );
// Streams the stream from response instead of loading it all in memory
DataBufferUtils.write( dataStream, response.getOutputStream() )
.map( DataBufferUtils::release )
.blockLast();
}
即使您没有 Reactive Server 堆栈，您仍然可以使用 WebClient - Rossen Stoyanchev（Spring Framework 团队的成员）在 Guide to "Reactive" for Spring MVC Developers 演示文稿中对其进行了很好的解释。在本次演讲中，Rossen Stoyanchev 提到他们考虑过弃用 RestTemplate，但最终还是决定推迟，但未来可能还会发生[=21] =]!
到目前为止，使用 WebClient 的主要缺点是学习曲线相当陡峭（响应式编程），但我认为将来无法避免，所以最好早点看一下。
我有一个 REST 服务向我发送了一个大的 ISO 文件，REST 服务没有问题。 现在我已经编写了一个调用其余服务来获取文件的 Web 应用程序，在客户端（Web 应用程序）端我收到内存不足 Exception.Below 是我的代码
HttpHeaders headers = new HttpHeaders();//1 Line
headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
headers.set("Content-Type","application/json");//3 Line
headers.set("Cookie", "session=abc");//4 Line
HttpEntity statusEntity=new HttpEntity(headers);//5 Line
String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line
ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line
我在第 7 行收到内存不足异常，我想我必须缓冲并获取部分内容，但不知道如何从服务器获取此文件，文件大小约为 500 到 700 MB。 任何人都可以帮忙.
异常堆栈：
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause
java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:3236)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)
我工作正常的服务器端 REST 服务代码是
@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{
byte[] reportBytes = null;
File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
if(result.exists()){
InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
response.setHeader("Content-Type",type);
reportBytes=new byte[100];//New change
OutputStream os=response.getOutputStream();//New change
int read=0;
while((read=inputStream.read(reportBytes))!=-1){
os.write(reportBytes,0,read);
}
os.flush();
os.close();
}
这会阻止将整个请求加载到内存中。
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate rest = new RestTemplate(requestFactory);
对于java.lang.OutOfMemoryError：Java heap space 可以解决添加更多内存到JVM：
-Xmxn Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration.
For server deployments, -Xms and -Xmx are often set to the same value. See Garbage Collector Ergonomics at http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m
您遇到的问题可能与您尝试执行的请求（下载大文件）没有严格关系，但为进程分配的内存不足。
您应该使用多部分文件附件，这样文件流就不会加载到内存中。 在此示例中，我使用了通过 Apache CXF 实现的休息服务。
...
import org.apache.cxf.jaxrs.ext.multipart.Attachment;
...
@Override
@Path("/put")
@Consumes("multipart/form-data")
@Produces({ "application/json" })
@POST
public SyncResponseDTO put( List<Attachment> attachments) {
SyncResponseDTO response = new SyncResponseDTO();
try {
for (Attachment attr : attachments) {
log.debug("get input filestream: " + new Date());
InputStream is = attr.getDataHandler().getInputStream();
这是我的做法。基于此 Spring Jira issue.
的提示RestTemplate restTemplate // = ...;
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);
来自上述 Jira 问题：
Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.
更新 Spring 5
Spring 5 介绍了 WebClient class 允许异步（例如非阻塞）http 请求。来自文档：
By comparison to the RestTemplate, the WebClient is:
- non-blocking, reactive, and supports higher concurrency with less hardware resources.
- provides a functional API that takes advantage of Java 8 lambdas.
- supports both synchronous and asynchronous scenarios.
- supports streaming up or down from a server.
要在 Spring 引导中获得 WebClient，您需要此依赖项：
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
目前，我坚持使用 RestTemplate，因为我不想为了访问 WebClient.
上述正确答案的更好版本可能是以下代码。此方法会将下载请求发送到另一个应用程序或服务，作为下载信息的实际来源。
public void download(HttpServletRequest req, HttpServletResponse res, String url)
throws ResourceAccessException, GenericException {
try {
logger.info("url::" + url);
if (restTemplate == null)
logger.info("******* rest template is null***********************");
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<ResponseEntity<InputStream>> responseExtractor = response -> {
String contentDisposition = response.getHeaders().getFirst("Content-Disposition");
if (contentDisposition != null) {
// Temporary location for files that will be downloaded from micro service and
// act as final source of download to user
String filePath = "/home/devuser/fileupload/download_temp/" + contentDisposition.split("=")[1];
Path path = Paths.get(filePath);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
// Create a new input stream from temporary location and use it for downloading
InputStream inputStream = new FileInputStream(filePath);
String type = req.getServletContext().getMimeType(filePath);
res.setHeader("Content-Disposition", "attachment; filename=" + contentDisposition.split("=")[1]);
res.setHeader("Content-Type", type);
byte[] outputBytes = new byte[100];
OutputStream os = res.getOutputStream();
int read = 0;
while ((read = inputStream.read(outputBytes)) != -1) {
os.write(outputBytes, 0, read);
}
os.flush();
os.close();
inputStream.close();
}
return null;
};
restTemplate.execute(url, HttpMethod.GET, requestCallback, responseExtractor);
} catch (Exception e) {
logger.info(e.toString());
throw e;
}
}
如@bernie所述，您可以使用 WebClient 来实现此目的：
public Flux<DataBuffer> downloadFileUrl( ) throws IOException {
WebClient webClient = WebClient.create();
// Request service to get file data
return Flux<DataBuffer> fileDataStream = webClient.get()
.uri( this.fileUrl )
.accept( MediaType.APPLICATION_OCTET_STREAM )
.retrieve()
.bodyToFlux( DataBuffer.class );
}
@GetMapping( produces = MediaType.APPLICATION_OCTET_STREAM_VALUE )
public void downloadFile( HttpServletResponse response ) throws IOException
{
Flux<DataBuffer> dataStream = this.downloadFileUrl( );
// Streams the stream from response instead of loading it all in memory
DataBufferUtils.write( dataStream, response.getOutputStream() )
.map( DataBufferUtils::release )
.blockLast();
}
即使您没有 Reactive Server 堆栈，您仍然可以使用 WebClient - Rossen Stoyanchev（Spring Framework 团队的成员）在 Guide to "Reactive" for Spring MVC Developers 演示文稿中对其进行了很好的解释。在本次演讲中，Rossen Stoyanchev 提到他们考虑过弃用 RestTemplate，但最终还是决定推迟，但未来可能还会发生[=21] =]!
到目前为止，使用 WebClient 的主要缺点是学习曲线相当陡峭（响应式编程），但我认为将来无法避免，所以最好早点看一下。