最近公司项目在短期内频繁的报java.lang.OutOfMemoryError: GC overhead limit exceeded内存溢出,造成系统不可用的情况,经过两天的排查,终于找到了原因,以下为本次排查的全部过程,做个记录~
在刚开始排查的时候,观察项目内的日志文件,会出现一堆的fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。然后java.lang.OutOfMemoryError: GC overhead limit exceeded,造成系统宕机,初步怀疑是网络环境问题,但由于项目是局域网环境,就没往这上边想,先是使用jmap查看了下堆内存中的对象:
发现byte [B占用内存过高,占用达到了百分之九十之多,进行分析过后,发现byte里边大多都是和HTTP请求相关的参数,中间没有截图,这里就不发截图了,大致就是基本全部都是HTTP相关的buffer,就是说Springboot tomcat为每个请求分发大概了10M以上的内存,导致内存过高,经过检查代码,发现配置文件有如下配置
max-http-header-size: 40485763
至此，基本已经确定了大概就是这个不合理的最大http请求头参数导致的内存过多的问题。将该配置进行删除,使用默认的配置,并限制tomcat最大连接数及线程数,然后重新启动并使用VisualVM进行程序观察,发现byte的占用已经从百分之九十多将到了百分之十几
但是问题来了,虽然程序不会出现内存溢出的情况了,但是那个报错依然还在,这个时候通过监控程序发现了以下的异常点
发现程序在一瞬间线程数飙升,达到限制值,而在这些飙升的点对应日志文件也会出现大量的报错信息,如下:
2022-02-15 10:36:49.123 ERROR 13692 --- [http-nio-8070-exec-171] c.t.w.controller.ExceptionController : Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.handleParseFailure(StandardMultipartHttpServletRequest.java:122) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.parseRequest(StandardMultipartHttpServletRequest.java:113) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.<init>(StandardMultipartHttpServletRequest.java:86) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.multipart.support.StandardServletMultipartResolver.resolveMultipart(StandardServletMultipartResolver.java:93) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1128) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:960) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.4.0.jar!/:1.4.0]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.4.0.jar!/:1.4.0]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) [shiro-core-1.4.0.jar!/:1.4.0]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at com.tongyu.workstation.filter.SystemFilter.doFilter(SystemFilter.java:21) [classes!/:0.1.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) [druid-1.1.9.jar!/:1.1.9]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_211]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_211]
Caused by: java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
at org.apache.catalina.connector.Request.parseParts(Request.java:2932) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.Request.parseParameters(Request.java:3232) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.Request.getParameter(Request.java:1137) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:75) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
... 23 common frames omitted
Caused by: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:308) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.Request.parseParts(Request.java:2884) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
... 27 common frames omitted
Caused by: java.io.IOException: 远程主机强迫关闭了一个现有的连接。
at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_211]
at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[na:1.8.0_211]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[na:1.8.0_211]
at sun.nio.ch.IOUtil.read(Unknown Source) ~[na:1.8.0_211]
at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.8.0_211]
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:147) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:173) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:235) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:216) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1250) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:725) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer.access$300(Http11InputBuffer.java:40) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1080) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:140) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:263) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.Request.doRead(Request.java:581) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:349) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:183) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:881) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at java.io.InputStream.read(Unknown Source) ~[na:1.8.0_211]
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:98) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:68) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream.readBodyData(MultipartStream.java:572) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream.discardBodyData(MultipartStream.java:596) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream.skipPreamble(MultipartStream.java:614) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:865) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:845) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:256) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:280) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
... 28 common frames omitted
这个时候忽然想到不会真的是因为网络问题吧?在短时间内多个客户端正在上传文件时网络忽然波动断开,然后重连,造成这个错误,为了验证这个猜想,我们在测试环境进行测试,在使用大量的文件上传过程中拔掉网线,然后再插上,观察服务器日志输出:
2022-02-17 13:49:06.159 ERROR 31024 --- [http-nio-7087-exec-4] c.t.w.controller.ExceptionController : Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.handleParseFailure(StandardMultipartHttpServletRequest.java:122) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.parseRequest(StandardMultipartHttpServletRequest.java:113) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.<init>(StandardMultipartHttpServletRequest.java:86) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.multipart.support.StandardServletMultipartResolver.resolveMultipart(StandardServletMultipartResolver.java:93) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1128) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:960) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) [spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) [spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) [spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.4.0.jar!/:1.4.0]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.4.0.jar!/:1.4.0]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) [shiro-core-1.4.0.jar!/:1.4.0]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.4.0.jar!/:1.4.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at com.tongyu.workstation.filter.SystemFilter.doFilter(SystemFilter.java:21) [classes!/:0.1.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) [druid-1.1.9.jar!/:1.1.9]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_261]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_261]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_261]
Caused by: java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
at org.apache.catalina.connector.Request.parseParts(Request.java:2932) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.Request.parseParameters(Request.java:3232) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.Request.getParameter(Request.java:1137) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:75) [spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
... 23 common frames omitted
Caused by: org.apache.tomcat.util.http.fileupload.FileUploadException: 远程主机强迫关闭了一个现有的连接。
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:308) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.Request.parseParts(Request.java:2884) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
... 27 common frames omitted
Caused by: java.io.IOException: 远程主机强迫关闭了一个现有的连接。
at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_261]
at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[na:1.8.0_261]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[na:1.8.0_261]
at sun.nio.ch.IOUtil.read(Unknown Source) ~[na:1.8.0_261]
at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.8.0_261]
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:147) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:173) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:235) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:216) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1250) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:725) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer.access$300(Http11InputBuffer.java:40) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1080) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:140) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:263) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.coyote.Request.doRead(Request.java:581) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:349) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:183) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:881) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at java.io.InputStream.read(Unknown Source) ~[na:1.8.0_261]
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:98) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:68) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream.readBodyData(MultipartStream.java:572) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream.discardBodyData(MultipartStream.java:596) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.MultipartStream.skipPreamble(MultipartStream.java:614) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:865) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:845) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:256) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:280) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
... 28 common frames omitted
果然.......就是因为网络问题造成该报错,在报错时线程并未关掉,而当时我们的服务器tomcat并未限制最大连接数及线程数,在短时间内恢复网络,程序连接造成线程堆积,内存飙升=_=
在程序部署上线时应合理考虑服务器配置并配置限制连接数及线程数,其中max-http-header-size配置要合理,无特殊情况不需要特殊配置,当然,同时要确保网络环境的畅通=_=
