Preview

My first path crossed with Broken Pipe issue was when I was gazing at the logs of a software I was working on.

The stack for the software was:

• JavaEE 7 with JAX-RS
• Java 8
• Wildfly 10.1 as application server
• Nginx was used as reverse proxy and load balancer.

I could see following traces flooding the log:

java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

I have ignored it a couple of times previously, thinking it hasn't created any havoc in the production server and wasn't quite sure how it was generated and how to handle, or completely prevent it.

What exactly is Broken Pipe?

In simple term, Broken Pipe means that a machine is attempting to read or write data from/to a pipe, while the machine on other end of the pipe has died or been terminated. Now, since the connection is closed, new connection should be established for further transfer of data, or else the data transfer ceases.

How does it occur?

One of the most common cause for this issue is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment). It also happens when forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download) or when the connection is slow.

A simple scenario: a browser requesting resource from server, and in response server returns response to browser. What if user closes the browser when server was sending response to browser? The connection between server to browser gets closed unexpectedly. This then gives rise to Broken Pipe and the exception is termed as java.io.IOException: Broken Pipe in Java.

This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency.

Not every Broken Pipe exception is developer's fault

Possible factors giving rise to this exception:

• Number of end users
  As one of the major cause of Broken Pipe is the behavior of user (unexpectedly closing active browser session before server could complete sending response), increase in number of end user increases chance of having Broken Pipe.

• Heavy response load
  Heavy response from server takes significant amount of time to get transferred to client side, and this huge time span can be the case for Broken Pipe.

• Timeout of server
  If the web server couldn't get the response from service in certain time which is equal to timeout value set in server, it closes connection to the client end returning 503: Gateway Timeout and consequently giving rise to Broken Pipe.

Is Broken Pipe Exception a Red Signal?

Well to be fair, it isn't a red signal, because largely it is caused by normal behavior of user, and there are always a chance of services being shutdown due to some failures. But, if a server is running on relatively large number of user requests at a time, then not only Broken Pipe, but any exception seems to pose a problem.

In my case, due to the high network traffic, logs were flooded with broken pipe exception. As, writing to a file (I/O operation) is one of the expensive operation performed by server, imagine server being flooded with Broken Pipe related exception, and the amount of resource server has to put in order to write that exception stacktrace to log file. This had caused server to response slowly and had made it sluggish.
At this point, I realized, YES BROKEN PIPE EXCEPTION IS A RED SIGNAL when scaled to large traffic.

Challenges on handling or removing Broken Pipe

The system was using Wildfly 10.1 as application server, and was written on JavaEE 7. Handling this scenario wasn't a cakewalk for me as:

• Replicating this exception in local or QA environment requires all planets to be aligned properly ( just kidding ), but yes it's toooo tough.

• Handling exception in Java is easy, as long as the exception is caught inside catch block. The nature of Unhandled exception: java.io.IOException: Broken pipe is such that, it is raised from Wildfly container and gets logged out in stacktrace instead of being trapped in catch block. Now, imagine having to deal with exception that cannot be caught from code. Gosh!!!!

• You will never know, which request raised the issue, as, the server was getting high amount of requests, any of them could have been the cause of exception. Adding logs to every rest endpoints along with socket endpoints is not feasible.

Fixing java.io.IOException : Broken Pipe

Finally, after so much of out of context talk, here comes the main section (hope its worth the wait).

Two ways to remove exception from the system are:

• Investigate the root cause of the exception, and eliminate it.
• Handle the exception gracefully, with proper logging or some action.

Eliminating root causes

• Asking user not to close connection unexpectedly
  This is impossible to do, for god's sake

• Reducing api response load
  This is somehow achievable, but in a legacy system, operating on large amount of data, rewriting all the logic so that api responses aren't heavy is also not feasible in all case.

• Increasing Timeout of Server
  Nginx has a variable named proxy_read_timeout which has default value of 60s, increasing this value will also minimize the chances of Broken Pipe.

Even after eliminating exact root cause, which itself is hard to detect in this case, we can't completely rule out the existence of Broken Pipe. Can we?

Hence, next solution is to handle Broken Pipe gracefully.

Handling java.io.IOException: Broken Pipe

• Suppress the log from logger itself
  If you use log4j as log manager, adding following configuration to log4j.properties will help to get rid of exception flooding the logs due to Broken Pipe.

log4j.logger.org.apache.catalina.connector.ClientAbortException = ERROR, console, cloudAppender
log4j.additivity.org.apache.catalina.connector.ClientAbortException = false

• Upgrading Resteasy within Wildfly
  We were using Resteasy implementation of JAX-RS for implementing REST in Java, and Resteasy v3.0.19 is bundled with Wildfly 10.1. With hope of finding some fixes in the resteasy itself, I started to dig into release notes of Resteasy and found out that after Resteasy-Client v3.1.1, Unhandled Exception: java.io.IOException can be caught from code, unlike container exposing it directly onto logtrace.
  Hence, upgrading version of Resteasy Client allows us to handle exception through Global Exception Handler (described in later point).

Steps to upgrade Resteasy within Wildfly

1. Find location of Resteasy distribution jars:
   The location of Resteasy distribution jars lies inside: WILDFLY_HOME/modules/system/layers/base/org/jboss/resteasy/resteasy-jaxrs/main and the file name is: resteasy-jaxrs-3.0.19.Final.jar

2. Download the zip file of target version of resteasy (3.1.1 and above) from Resteasy downloads

3. Unpack the file in temporary directory. It has two other zip files within it: resteasy-jboss-modules-3.1.1.Final-mavenized.zip &
   resteasy-jboss-modules-3.1.1.Final.zip

4. Unpack both .zip files at:
   WILDFLY_HOME/modules/system/layers/base

5. Update resteasy dependencies' version in pom.xml

<dependency> 
  <groupId>org.jboss.resteasy</groupId> 
  <artifactId>resteasy-jaxrs</artifactId> 
  <version>3.1.1.Final</version>
  <scope>provided</scope>
</dependency>

• Use Global Exception Handler
  I further added Generic Exception Mapper from the java code itself, to catch and handle the Broken Pipe.

My final approach

Though I had gathered multiple techniques to solve the issue, following are the things I actually implemented:

• Upgrade Wildfly to v11.0, as it automatically comes with Resteasy v3.1.1 (the version that I exactly wanted) bundled within it.
• Handle exception through Global Exception Handler

and with this the issue of Broken Pipe now only exists in old logs archives.

Parting words

It's my first blog, yeah it's a bit long, but I was giving insights into how I actually approached the issue and fixed it. Feel free to provide feedback in the comment section & watch this space for more development blogs in future.

Also, a bit of self promotion at the end. Reach out to me at Twitter & LinkedIn

References

• RESTEASY-1238

• Github Issue - 756

• Github - 1169

• Github - 772

• Blog post

• Jira - Atlassian