How I fixed java.io.IOException: Broken Pipe in Java (Wildfly 10.1)
Command Palette
Search for a command to run...
Comments (1)
Join the discussionB
Bishwa Poudel5y ago
Thank you 😍
Search for a command to run...
Thank you 😍
My first path crossed with Broken Pipe issue was when I was gazing at the logs of a software I was working on.
The stack for the software was:
I could see following traces flooding the log:
java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
I have ignored it a couple of times previously, thinking it hasn't created any havoc in the production server and wasn't quite sure how it was generated and how to handle, or completely prevent it.
In simple term, Broken Pipe means that a machine is attempting to read or write data from/to a pipe, while the machine on other end of the pipe has died or been terminated. Now, since the connection is closed, new connection should be established for further transfer of data, or else the data transfer ceases.
One of the most common cause for this issue is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment). It also happens when forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download) or when the connection is slow.
A simple scenario: a browser requesting resource from server, and in response server returns response to browser. What if user closes the browser when server was sending response to browser? The connection between server to browser gets closed unexpectedly. This then gives rise to Broken Pipe and the exception is termed as java.io.IOException: Broken Pipe in Java.
This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency.
Not every Broken Pipe exception is developer's fault
Possible factors giving rise to this exception:
Number of end users
As one of the major cause of Broken Pipe is the behavior of user (unexpectedly closing active browser session before server could complete sending response), increase in number of end user increases chance of having Broken Pipe.
Heavy response load
Heavy response from server takes significant amount of time to get transferred to client side, and this huge time span can be the case for Broken Pipe.
Timeout of server
If the web server couldn't get the response from service in certain time which is equal to timeout value set in server, it closes connection to the client end returning 503: Gateway Timeout and consequently giving rise to Broken Pipe.
Well to be fair, it isn't a red signal, because largely it is caused by normal behavior of user, and there are always a chance of services being shutdown due to some failures. But, if a server is running on relatively large number of user requests at a time, then not only Broken Pipe, but any exception seems to pose a problem.
In my case, due to the high network traffic, logs were flooded with broken pipe exception. As, writing to a file (I/O operation) is one of the expensive operation performed by server, imagine server being flooded with Broken Pipe related exception, and the amount of resource server has to put in order to write that exception stacktrace to log file. This had caused server to response slowly and had made it sluggish.
At this point, I realized, YES BROKEN PIPE EXCEPTION IS A RED SIGNAL when scaled to large traffic.
The system was using Wildfly 10.1 as application server, and was written on JavaEE 7. Handling this scenario wasn't a cakewalk for me as:
Replicating this exception in local or QA environment requires all planets to be aligned properly ( just kidding ), but yes it's toooo tough.
Handling exception in Java is easy, as long as the exception is caught inside catch block. The nature of Unhandled exception: java.io.IOException: Broken pipe is such that, it is raised from Wildfly container and gets logged out in stacktrace instead of being trapped in catch block. Now, imagine having to deal with exception that cannot be caught from code. Gosh!!!!
You will never know, which request raised the issue, as, the server was getting high amount of requests, any of them could have been the cause of exception. Adding logs to every rest endpoints along with socket endpoints is not feasible.
Finally, after so much of out of context talk, here comes the main section (hope its worth the wait).
Two ways to remove exception from the system are:
Asking user not to close connection unexpectedly
This is impossible to do, for god's sake
Reducing api response load
This is somehow achievable, but in a legacy system, operating on large amount of data, rewriting all the logic so that api responses aren't heavy is also not feasible in all case.
Increasing Timeout of Server
Nginx has a variable named proxy_read_timeout which has default value of 60s, increasing this value will also minimize the chances of Broken Pipe.
Even after eliminating exact root cause, which itself is hard to detect in this case, we can't completely rule out the existence of Broken Pipe. Can we?
Hence, next solution is to handle Broken Pipe gracefully.
log4j.logger.org.apache.catalina.connector.ClientAbortException = ERROR, console, cloudAppender
log4j.additivity.org.apache.catalina.connector.ClientAbortException = false
Steps to upgrade Resteasy within Wildfly
Find location of Resteasy distribution jars:
The location of Resteasy distribution jars lies inside:
WILDFLY_HOME/modules/system/layers/base/org/jboss/resteasy/resteasy-jaxrs/main
and the file name is: resteasy-jaxrs-3.0.19.Final.jar
Download the zip file of target version of resteasy (3.1.1 and above) from Resteasy downloads
Unpack the file in temporary directory. It has two other zip files within it:
resteasy-jboss-modules-3.1.1.Final-mavenized.zip &
resteasy-jboss-modules-3.1.1.Final.zip
Unpack both .zip files at:
WILDFLY_HOME/modules/system/layers/base
Update resteasy dependencies' version in pom.xml
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<version>3.1.1.Final</version>
<scope>provided</scope>
</dependency>
Though I had gathered multiple techniques to solve the issue, following are the things I actually implemented:
and with this the issue of Broken Pipe now only exists in old logs archives.
It's my first blog, yeah it's a bit long, but I was giving insights into how I actually approached the issue and fixed it. Feel free to provide feedback in the comment section & watch this space for more development blogs in future.
Also, a bit of self promotion at the end. Reach out to me at Twitter & LinkedIn