Error java io ioexception broken pipe

Preview

My first path crossed with Broken Pipe issue was when I was gazing at the logs of a software I was working on.

The stack for the software was:

• JavaEE 7 with JAX-RS
• Java 8
• Wildfly 10.1 as application server
• Nginx was used as reverse proxy and load balancer.

I could see following traces flooding the log:

java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

I have ignored it a couple of times previously, thinking it hasn’t created any havoc in the production server and wasn’t quite sure how it was generated and how to handle, or completely prevent it.

What exactly is Broken Pipe?

In simple term, Broken Pipe means that a machine is attempting to read or write data from/to a pipe, while the machine on other end of the pipe has died or been terminated. Now, since the connection is closed, new connection should be established for further transfer of data, or else the data transfer ceases.

How does it occur?

One of the most common cause for this issue is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment). It also happens when forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download) or when the connection is slow.

A simple scenario: a browser requesting resource from server, and in response server returns response to browser. What if user closes the browser when server was sending response to browser? The connection between server to browser gets closed unexpectedly. This then gives rise to Broken Pipe and the exception is termed as java.io.IOException: Broken Pipe in Java.

This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency.

Not every Broken Pipe exception is developer’s fault

Possible factors giving rise to this exception:

• Number of end users
  As one of the major cause of Broken Pipe is the behavior of user (unexpectedly closing active browser session before server could complete sending response), increase in number of end user increases chance of having Broken Pipe.

• Heavy response load
  Heavy response from server takes significant amount of time to get transferred to client side, and this huge time span can be the case for Broken Pipe.

• Timeout of server
  If the web server couldn’t get the response from service in certain time which is equal to timeout value set in server, it closes connection to the client end returning 503: Gateway Timeout and consequently giving rise to Broken Pipe.

Is Broken Pipe Exception a Red Signal?

Well to be fair, it isn’t a red signal, because largely it is caused by normal behavior of user, and there are always a chance of services being shutdown due to some failures. But, if a server is running on relatively large number of user requests at a time, then not only Broken Pipe, but any exception seems to pose a problem.

In my case, due to the high network traffic, logs were flooded with broken pipe exception. As, writing to a file (I/O operation) is one of the expensive operation performed by server, imagine server being flooded with Broken Pipe related exception, and the amount of resource server has to put in order to write that exception stacktrace to log file. This had caused server to response slowly and had made it sluggish.

At this point, I realized, YES BROKEN PIPE EXCEPTION IS A RED SIGNAL when scaled to large traffic.

Challenges on handling or removing Broken Pipe

The system was using Wildfly 10.1 as application server, and was written on JavaEE 7. Handling this scenario wasn’t a cakewalk for me as:

• Replicating this exception in local or QA environment requires all planets to be aligned properly ( just kidding ), but yes it’s toooo tough.

• Handling exception in Java is easy, as long as the exception is caught inside catch block. The nature of Unhandled exception: java.io.IOException: Broken pipe is such that, it is raised from Wildfly container and gets logged out in stacktrace instead of being trapped in catch block. Now, imagine having to deal with exception that cannot be caught from code. Gosh!!!!

• You will never know, which request raised the issue, as, the server was getting high amount of requests, any of them could have been the cause of exception. Adding logs to every rest endpoints along with socket endpoints is not feasible.

Fixing java.io.IOException : Broken Pipe

Finally, after so much of out of context talk, here comes the main section (hope its worth the wait).

Two ways to remove exception from the system are:

• Investigate the root cause of the exception, and eliminate it.
• Handle the exception gracefully, with proper logging or some action.

Eliminating root causes

• Asking user not to close connection unexpectedly
  This is impossible to do, for god’s sake

• Reducing api response load
  This is somehow achievable, but in a legacy system, operating on large amount of data, rewriting all the logic so that api responses aren’t heavy is also not feasible in all case.

• Increasing Timeout of Server
  Nginx has a variable named proxy_read_timeout which has default value of 60s, increasing this value will also minimize the chances of Broken Pipe.

Even after eliminating exact root cause, which itself is hard to detect in this case, we can’t completely rule out the existence of Broken Pipe. Can we?

Hence, next solution is to handle Broken Pipe gracefully.

Handling java.io.IOException: Broken Pipe

• Suppress the log from logger itself
  If you use log4j as log manager, adding following configuration to log4j.properties will help to get rid of exception flooding the logs due to Broken Pipe.

log4j.logger.org.apache.catalina.connector.ClientAbortException = ERROR, console, cloudAppender
log4j.additivity.org.apache.catalina.connector.ClientAbortException = false

• Upgrading Resteasy within Wildfly
  We were using Resteasy implementation of JAX-RS for implementing REST in Java, and Resteasy v3.0.19 is bundled with Wildfly 10.1. With hope of finding some fixes in the resteasy itself, I started to dig into release notes of Resteasy and found out that after Resteasy-Client v3.1.1, Unhandled Exception: java.io.IOException can be caught from code, unlike container exposing it directly onto logtrace.
  Hence, upgrading version of Resteasy Client allows us to handle exception through Global Exception Handler (described in later point).

Steps to upgrade Resteasy within Wildfly

1. Find location of Resteasy distribution jars:
   The location of Resteasy distribution jars lies inside:
   WILDFLY_HOME/modules/system/layers/base/org/jboss/resteasy/resteasy-jaxrs/main
   and the file name is: resteasy-jaxrs-3.0.19.Final.jar

2. Download the zip file of target version of resteasy (3.1.1 and above) from Resteasy downloads

3. Unpack the file in temporary directory. It has two other zip files within it:
   resteasy-jboss-modules-3.1.1.Final-mavenized.zip &
   resteasy-jboss-modules-3.1.1.Final.zip 

4. Unpack both .zip files at:  
   WILDFLY_HOME/modules/system/layers/base

5. Update resteasy dependencies’ version in pom.xml

<dependency> 
  <groupId>org.jboss.resteasy</groupId> 
  <artifactId>resteasy-jaxrs</artifactId> 
  <version>3.1.1.Final</version>
  <scope>provided</scope>
</dependency>

• Use Global Exception Handler
  I further added Generic Exception Mapper from the java code itself, to catch and handle the Broken Pipe.

My final approach

Though I had gathered multiple techniques to solve the issue, following are the things I actually implemented:

• Upgrade Wildfly to v11.0, as it automatically comes with Resteasy v3.1.1 (the version that I exactly wanted) bundled within it.
• Handle exception through Global Exception Handler

and with this, the issue of Broken Pipe now only exists in old logs archives 😎😎.

Parting words

It’s my first blog, yeah it’s a bit long, but I was giving insights into how I actually approached the issue and fixed it. Feel free to provide feedback in the comment section &  watch this space for more development blogs in future.

Also, a bit of self promotion at the end 😉. Reach out to me at  Twitter & LinkedIn

References

• RESTEASY-1238

• Github Issue — 756

• Github — 1169

• Github — 772

• Blog post

• Jira — Atlassian

java.io.IOException: Broken pipe

Related Article — Java Error

Java.Lang.OutOfMemoryError: Unable to Create New Native Thread

Today, the company’s technical support reported that a customer’s service did not work, and they urgently asked for help, so I remotely logged on to the server to check the problem.

Check the tomcat log of the collected data, and habitually turn to the end of the log to see if there is any exception printing. several kinds of exception information are found, but this is the most one

24-Nov-2016 09:54:21.116 SEVERE [http-nio-8081-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
 java.io.IOException: Too many open files
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
	at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:688)
	at java.lang.Thread.run(Thread.java:745)

The problem of «manage open files» is very common. If the file descriptor exceeds the limit, the file cannot be opened or the network connection cannot be created. This problem will lead to some other problems. It must be that ulimit is not optimized, so check the ulimit setting;

[root@sdfassd logs]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62819
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62819
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

open files is 65535. It has been optimized. Is it tomcat and other services start first, and then optimize ulimit? It’s possible, if werestart the service, it’s ok. Then I restart all the services and run normally. After a while, the report will display the data. Then I tell the technical support that the problem has been solved, and then I go to deal with other cases;

As a result, within 20 minutes, the technical support said that there was no data in the report again, so I checked the tomcat log of the data collection application and found a bunch of exceptions, all of which were this error

24-Nov-2016 09:54:24.574 WARNING [http-nio-18088-exec-699] org.apache.catalina.core.StandardHostValve.throwable Exception Processing ErrorPage[exceptionType=java.lang.Throwable, location=/views/error/500.jsp]
 org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
	at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:295)
	at org.apache.catalina.connector.Response.finishResponse(Response.java:453)
	at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:378)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1085)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1556)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1513)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)

There are a lot of exceptions. Read the error message. It is the Broken pipe exception of tomcat’s connector when it performs the write operation. The connector is used by tomcat to process the network request. Is there something wrong with the network? But why is it that the exception is all written and the read is OK? In order to determine whether it is a network problem, I use wget command to access an interface of the server locally. The result shows that we have waited for a long time but have not responded. Normally, we should have responded immediately. This indicates that it is not the network problem, but the server problem. Then I use the command to check the current tcpip connection status

[root@sdfassd logs]# netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"t",state[key]}'
CLOSE_WAIT        3853
TIME_WAIT         40
ESTABLISHED       285
LAST_ACT          6

    There are 3853 CLOSE_WAIT state connections. This is abnormal. It means that the client has closed the connection first, but the server has not closed the connection. As a result, the server is always CLOSE_WAIT state, If we do not optimize the keepalive of the operating system, this state will last for two hours by default. Check the settings of the following system:

[root@sdfassd logs]# sysctl -a |grep keepalive
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75

Sure enough, it’s 7200 seconds, which explains why the last error of checking tomcat log for the first time is «Too manay open files». It must be within two hours, CLOSE_WAIT state increases sharply, the file descriptor exceeds the maximum limit of 65535;

This state should be caused by the broken pipe exception. What causes the broken pipe exception? Why does the probe close the connection, but the data acquisition server does not? The exception is the connector of tomcat. tomcat can’t forget to call the close method to close the connection, which eliminates the problem of the program and can’t figure out what caused it;

So I went to check the log of collection server, and there was a large number of exceptions:

2016-11-24 16:27:36,217 [TingYun Harvest Service 1] 166 WARN  - Error occurred sending metric data to TingYun. There can be intermittent connection failures. Please wait for a short period of time: java.net.SocketTimeoutException: Read timed out
java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_60]
	at java.net.SocketInputStream.read(SocketInputStream.java:152) ~[na:1.7.0_60]
	at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[na:1.7.0_60]
	at com.tingyun.agent.libs.org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SourceFile:136) ~[tingyun-agent-java.jar:2.1.3]
        .................

If all of them are read time out exceptions, then the problem is clear. It is that the probe side reads overtime and disconnects. At this time, the data acquisition server is still processing the request. It does not know that the probe side has disconnected. After processing the request, it sends the processing result to the probe, and then it breaks the pipe;

The original exception is that the client has closed the connection due to reading timeout. At this time, when the server writes data to the disconnected connection of the client, the broken pipe exception occurs!

The probe read timeout time is 2 minutes. Why did the server not respond for such a long time? So we use jstack command to export the thread stack information of tomcat for analysis, and finally find that there are time-consuming operations in the code are locked, leading to thread blocking (for confidentiality reasons, we don’t paste the code here);

To sum up, some of my friends who sent me private messages didn’t get the key point of the broken pile problem. It’s not only the timeout that causes this problem. As long as the connection is disconnected and the write operation is performed on the disconnected connection, this exception will appear. The client’s overtime disconnection is just one of the cases

In addition, when you see the «Too manay open files» exception, in addition to checking the ulimit of system you should also check the number of file handles opened by the process. The cat /proc/sys/fs/file-nr command is used to check the total number of system handles, The number of file handles opened by the current application uses the ls -l /proc/<pid>/fd | wc -l command. Fortunately, this step is ignored here, otherwise it may take some time to find the real problems of the system;

From this case, we can see that in some cases, the exception information you see at the first sight may not be the root of the problem, but a chain reaction in the follow-up. Especially when a large number of the same exception occurs, instead of looking at the last exception log, you should first go to the log to find the location where the first exception occurs and see the reason why the exception occurs;

java tcp/ip exception

• 1 java.net.SocketTimeoutException . 

This exception is quite common, socket timeout. Generally, there are two places to throw this. One is when connecting, the timeout parameter is determined by second parameter in connect(SocketAddress endpoint,int timeout), and the other is setSoTimeout(int timeout), which is to set the reading timeout. They are set to 0 to indicate infinity.

• 2 java.net.BindException:Address already in use: JVM_Bind 

The exception occurs when a new ServerSocket(port) or socket.bind(SocketAddress bindpoint) operation.

Reason: the port has been started and monitored. At this point, with netstat –an command, you can see the port in the listening state. Just find a port that is not occupied to solve this problem.

• 3 java.net.ConnectException: Connection refused: connect

The exception occurs when the client performs new Socket(ip, port) or socket.connect(address,timeout) operation, reason: the machine with the specified ip address cannot be found (that is, the ip route from the current machine does not exist to the specified ip route), or the ip exists, but the specified port cannot be found for listening. You should first check whether the ip and port of the client are wrongly written. If it is correct, ping the server from the client to see if it can be pinged. If it can be pinged (another method is needed if the server disables ping), check whether the program monitoring the specified port on the server is started.

• 4 java.net.SocketException: Socket is closed 

This exception can occur on both the client and the server. The reason for the exception is that the network connection is read and written after the connection is closed (the close method of Socket is called).

• 5 java.net.SocketException : connection reset or Connect reset by peer:Socket write error

Connection reset by peer will appear when calling write or read. According to glibc, such as by the remote machine rebooting or an unrecoverable protocol violation. Literally, it means that the remote machine restarts or an unrecoverable error occurs. According to my tests, only the opposite end directly kills the process. What’s the difference between the two? Compared with tcpdump’s packet interception diagram, if the remote process is directly killed, the remote end does not send the FIN serial number to tell the other party that I have closed the pipeline, but directly sends the RST serial number. If the remote end calls close or shutdown, it will send the FIN serial number. According to the four waves of TCP, FIN is needed. I guess that if I receive the RST sequence number directly instead of receiving the FIN sequence number of the other party at the local end, it indicates that machine rebooting or an unrecoverable protocol violation at the opposite end. At this time, the IO operation of this pipeline will result in a connection reset by peer error.

This exception may occur on both the client and the server. There are two reasons for this exception. The first is that if the Socket at one end is closed (or actively closed or closed due to an exception exit), the other end still sends data, and the first packet sent raises this exception (Connect reset by peer). The other is that one end exits, but the connection is not closed when exiting. If the other end is reading data from the connection, the exception (Connection reset) will be thrown. In short, it is caused by the read and write operations after the connection is disconnected.

In another case, if one end sends an RST packet to interrupt the TCP connection, the other end will also have this exception. If it is tomcat, the exception is as follows:

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer

In order to improve the performance of Alibaba’s tcp mode health check, the server will send an RST to terminate the connection, which will cause this exception;

For the server, the general reason can be considered as follows:

a) If the number of concurrent connections exceeds its capacity, the server will actively Down some of them

b) In the process of data transmission, the browser or the receiving client is closed, while the server is still sending data to the client.

• 6 java.net.SocketException: Broken pipe

This exception can occur on both the client and the server. after SocketExcepton:Connect reset by peer:Socket write error,  If the data continues to be written, the exception will be thrown. The solution to the first two exceptions is to make sure that all network connections are closed before the program exits. The second is to detect the closing operation of the other party. After the other party closes the connection, it should also close the connection itself.

Broken pipe only appears when write is called. Broken pipe means that the opposite end of the pipe has been disconnected, which often occurs If the read / write pipe is closed, you cannot read and write to the pipe. In terms of the four waves of tcp, the far end has sent the FIN serial number to tell you that the pipeline has been closed. At this time, if you continue to write data into the pipe, the first time you will receive an RST signal sent by the far end. If you continue to write data into the pipeline, the operating system will send you a SIGPIPE signal and set errno to broken Pipe (32). If your program does not process SIGPIPE by default, the program will interrupt and exit. In general, signal(SIGPIPE,SIG_IGN) can be used to ignores this signal, so the program will not exit, but write will return — 1 and set errno to broken pipe (32). Broken pipe only occurs when data is written to the closed pipeline of the opposite end (after receiving the rst serial number of the opposite end, the first write will not appear broken pipe, but write returns — 1. At this time, the correct way is to close the pipe of the local end. If you continue to write, this error will appear).

java.net.SocketException: Broken pipe (Write failed)        at java.net.SocketOutputStream.socketWrite0(Native Method)        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)        at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)        at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)        at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)        at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:120)        at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:167)        at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)        at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)

For the exception of 4 and 5, special attention should be paid to the maintenance of the connection. In the case of short connection, it’s OK. In the case of long connection, if the maintenance of the connection state is improper, it’s very easy to get exceptions. Basically, what we need to do for long connection is:

a) Detect the active disconnection of the other party (the other party calls the close method of Socket). Because the other party actively disconnects, if the other party is reading, the return value at this time is -1. Therefore, once the disconnection of the other party is detected, cloase connection in self end (call the close method of Socket).

b) To detect the other party’s downtime, abnormal exit and network failure, the general method is heartbeat detection. Both sides send data to the other side periodically, and receive «heartbeat data» from the other side at the same time. If the other side’s heartbeat is not received for several consecutive cycles, it can be judged that the other side is down, or exited abnormally, or the network is out of order. At this time, it also needs to actively close its own connection. If it is a client, it can restart the connection after a certain delay. Although Socket has a keep alive option to maintain the connection, it usually takes two hours to find the other party’s downtime, abnormal exit and network failure.

• 7 java.net.SocketException: Too many open files

Reason: the maximum number of open file handles in the operating system is limited, which often occurs when many concurrent users access the server. Because in order to execute each user’s application, the server has to load many files (a new socket needs a file handle), which leads to the lack of open file handles.

Solution:

a) Try to make the class a jar package, because a jar package consumes only one file handle. If it is not packaged, a class consumes one file handle.

b) java GC cannot close the file handle opened by network connection. If close() is not executed, the file handle will always exist and cannot be closed.

You can also consider setting the maximum number of open socket s to control this problem. Set the operating system to increase the maximum number of file handles.

ulimit -a can view the current resource limit of the system, ulimit -n 10240 can be modified, which is only valid for the current window.

• 8 Cannot assign requested address                                    

1. The address cannot be bound because the port number is occupied:

java.net.BindException: Cannot assign requested address: bind: it is caused by the change of IP address;

2. server network configuration exception:

The address configured in /etc/hosts is wrong;

3. Another case is that there is no loop address when ipconfig is executed, because the loop address configuration file is lost;

Posted by fakedec
at Apr 02, 2021 — 11:59 AM
Tag:
Tomcat

What are the possible causes for this exception when running code on a linux machine that has previosuly worked perfectly on a windows machine in a tomcat environment?

I’m using the following api:

http://eclipse.crosscoreop.com/doc/javadoc/JavaEclipseInterface/index.html

and I get the following errors:

java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.BufferedWriter.flush(BufferedWriter.java:236)
at com.parctechnologies.eclipse.OutOfProcessEclipse.startLocalEclipse(OutOfProcessEclipse.java:290)
at com.parctechnologies.eclipse.OutOfProcessEclipse.<init>(OutOfProcessEclipse.java:73)
at com.jae.jaeQuestion.QuestionSet.<init>(QuestionSet.java:79)
at com.jae.jaeQuestion.QuestionServlet.getOutput(QuestionServlet.java:96)
at com.jae.jaeQuestion.QuestionServlet.doGet(QuestionServlet.java:63)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

then subsequently:

java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:415)
at java.lang.Integer.parseInt(Integer.java:497)
at com.parctechnologies.eclipse.OutOfProcessEclipse.<init>(OutOfProcessEclipse.java:82)
at com.jae.jaeQuestion.QuestionSet.<init>(QuestionSet.java:79)
at com.jae.jaeQuestion.QuestionServlet.getOutput(QuestionServlet.java:96)
at com.jae.jaeQuestion.QuestionServlet.doGet(QuestionServlet.java:63)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:25)

I get similar errors when I run the code as a standalone program (with main method).

The code works perfectly in tomcat running on windows.

Thanks,

Lilalfyalien

Message was edited by:
lil_alfy_alien

Message was edited by:
lil_alfy_alien

Minecraft Forums

More Info: How to Fix Internal Exception: java.io.ioexception

1.8.9 Java Edition Ubuntu Broken Pipe Error

li» data-page-inline=»False» data-scroll-inline=»False»>

• Tree Puncher
• Join Date: 2/20/2015
• Posts: 37
• Location: West Kelowna, BC
• Minecraft: FluffyySnow
• Member Details

I did get this to work, I downloaded version 8 of java and made the profile use it. But when I am on a server network like hypixel, it takes 10-20 seconds to switch between servers. Then after 5-10 minutes I get this error:

Internal Exception: java.io.IOException: Error while write(. ): Broken pipe

Are there any users out there with the knowledge on how to fix this? I am fairly new to Ubuntu.

This error only seems to appear when I have a render distance above 8

• Tree Puncher
• Join Date: 2/20/2015
• Posts: 37
• Location: West Kelowna, BC
• Minecraft: FluffyySnow
• Member Details

• Tree Puncher
• Join Date: 2/20/2015
• Posts: 37
• Location: West Kelowna, BC
• Minecraft: FluffyySnow
• Member Details

• Farlander
• Join Date: 9/20/2016
• Posts: 15,670
• Member Details

Maybe it is an issue with your internet connection

• Tree Puncher
• Join Date: 2/20/2015
• Posts: 37
• Location: West Kelowna, BC
• Minecraft: FluffyySnow
• Member Details

Maybe it is an issue with your internet connection

This only happens on my Ubuntu os.

• Farlander
• Join Date: 9/20/2016
• Posts: 15,670
• Member Details

Check your drivers or for software on your system they may cause this issue

Maybe VPN tools or similar

• Out of the Water
• Join Date: 6/23/2021
• Posts: 2
• Member Details

I just tried disabling server texture packs for Hypixel and clearing all the packs in my server resource packs folder, and it seems to have solved the problem.

• Tree Puncher
• Join Date: 2/20/2015
• Posts: 37
• Location: West Kelowna, BC
• Minecraft: FluffyySnow
• Member Details

I just tried disabling server texture packs for Hypixel and clearing all the packs in my server resource packs folder, and it seems to have solved the problem.

Hey cool! I’ve stopped using ubuntu in wait overall better software support like Logitech G Hub and such but when I start using it again I’ll give your solution a go.

• Newbie
• Join Date: 10/16/2021
• Posts: 0
• Member Details

As you mentioned. It happens only on Linux OS and then the render distance too. Since your computer needs to receive chunks and put them in your game, it takes a lot of time and for reason Linux can’t do that job quickly. I dual Windows 11 and POP OS, I have never seen this issue in Windows till now even though I play Minecraft with 12 Chunks render distance. Changing JAVA versions or the publisher doesn’t fix it too. So the best we can do to avoid this is to lower the render distance so that your computer doesn’t need to take a lot of time to put your chunks in the game. The more time spent in loading chunks, the more are the chances of you getting disconnected from the server with a BROKEN PIPE or relatively similar ioexpections. And changing resource pack did not help with me though.

• Out of the Water
• Join Date: 6/23/2021
• Posts: 2
• Member Details

I thought that disabling texture packs had fixed the problem, but it didn’t. And when playing Hypixel Skyblock, I keep getting this error even when playing with render distance of 8.

Источник

Resolve java.io.IOException: Broken Pipe

This article covers the causes of the java.io.IOException: Broken pipe exception and provides solutions to fix it in Java. But before that, let’s look at the java.io.IOException: Broken pipe exception.

Understand the java.io.IOException: Broken pipe

java.io.IOException: Broken pipe refers to the situation in which one device is trying to read/write data from/to a pipe while the machine previously connected to the other end of the pipe has either died or been terminated.

We must establish a new connection to continue data transmission because that connection has been severed. If we don’t do it, it will terminate the data transfer, and we will receive the following exception:

Causes of java.io.IOException: Broken pipe

Following are some potential causes that trigger the java.io.IOException: Broken pipe :

• When a client terminates an open connection while doing any operation, such as loading a page or downloading a file. It is one of the most common things that can cause this exception.
• Another common reason for a java.io.IOException: Broken pipe is when one of two machines interacting via a socket has shut down the socket on its end before the exchange of information has been finished.
• Performance difficulties or low network efficiency can disrupt a connection between a client and a server and lead to exceptions.
• When the web server cannot get a response from the service within a specified amount of time, which is equal to the timeout value defined in the server, it will shut the connection on the client end, resulting in a Broken Pipe .

Solution for the java.io.IOException: Broken pipe

• We can try inserting some delays throughout the application to see if this makes a difference. However, because of the delays, the users should have sufficient time to complete the transfer.
• Handle the exception dignifiedly by performing the necessary logging or taking action.
• To be prepared for any situation, we should improve your internet connection as soon as possible.
• Increasing the Timeout of the Server, which has a default value of 60 seconds. Increasing this amount will not only reduce the likelihood of a Broken Pipe , but it will also improve performance.

I have been working as a Flutter app developer for a year now. Firebase and SQLite have been crucial in the development of my android apps. I have experience with C#, Windows Form Based C#, C, Java, PHP on WampServer, and HTML/CSS on MYSQL, and I have authored articles on their theory and issue solving. I’m a senior in an undergraduate program for a bachelor’s degree in Information Technology.

Источник

How I fixed java.io.IOException: Broken Pipe in Java (Wildfly 10.1)

Preview

My first path crossed with Broken Pipe issue was when I was gazing at the logs of a software I was working on.

The stack for the software was:

• JavaEE 7 with JAX-RS
• Java 8
• Wildfly 10.1 as application server
• Nginx was used as reverse proxy and load balancer.

I could see following traces flooding the log:

Exit fullscreen mode

I have ignored it a couple of times previously, thinking it hasn’t created any havoc in the production server and wasn’t quite sure how it was generated and how to handle, or completely prevent it.

What exactly is Broken Pipe?

In simple term, Broken Pipe means that a machine is attempting to read or write data from/to a pipe, while the machine on other end of the pipe has died or been terminated. Now, since the connection is closed, new connection should be established for further transfer of data, or else the data transfer ceases.

How does it occur?

One of the most common cause for this issue is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment). It also happens when forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download) or when the connection is slow.

A simple scenario: a browser requesting resource from server, and in response server returns response to browser. What if user closes the browser when server was sending response to browser? The connection between server to browser gets closed unexpectedly. This then gives rise to Broken Pipe and the exception is termed as java.io.IOException: Broken Pipe in Java.

This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency.

Not every Broken Pipe exception is developer’s fault

Possible factors giving rise to this exception:

Number of end users
As one of the major cause of Broken Pipe is the behavior of user (unexpectedly closing active browser session before server could complete sending response), increase in number of end user increases chance of having Broken Pipe.

Heavy response load
Heavy response from server takes significant amount of time to get transferred to client side, and this huge time span can be the case for Broken Pipe.

Timeout of server
If the web server couldn’t get the response from service in certain time which is equal to timeout value set in server, it closes connection to the client end returning 503: Gateway Timeout and consequently giving rise to Broken Pipe.

Is Broken Pipe Exception a Red Signal?

Well to be fair, it isn’t a red signal, because largely it is caused by normal behavior of user, and there are always a chance of services being shutdown due to some failures. But, if a server is running on relatively large number of user requests at a time, then not only Broken Pipe, but any exception seems to pose a problem.

In my case, due to the high network traffic, logs were flooded with broken pipe exception. As, writing to a file (I/O operation) is one of the expensive operation performed by server, imagine server being flooded with Broken Pipe related exception, and the amount of resource server has to put in order to write that exception stacktrace to log file. This had caused server to response slowly and had made it sluggish.

At this point, I realized, YES BROKEN PIPE EXCEPTION IS A RED SIGNAL when scaled to large traffic.

Challenges on handling or removing Broken Pipe

The system was using Wildfly 10.1 as application server, and was written on JavaEE 7. Handling this scenario wasn’t a cakewalk for me as:

Replicating this exception in local or QA environment requires all planets to be aligned properly ( just kidding ), but yes it’s toooo tough.

Handling exception in Java is easy, as long as the exception is caught inside catch block. The nature of Unhandled exception: java.io.IOException: Broken pipe is such that, it is raised from Wildfly container and gets logged out in stacktrace instead of being trapped in catch block. Now, imagine having to deal with exception that cannot be caught from code. Gosh.

You will never know, which request raised the issue, as, the server was getting high amount of requests, any of them could have been the cause of exception. Adding logs to every rest endpoints along with socket endpoints is not feasible.

Fixing java.io.IOException : Broken Pipe

Finally, after so much of out of context talk, here comes the main section (hope its worth the wait).

Two ways to remove exception from the system are:

• Investigate the root cause of the exception, and eliminate it.
• Handle the exception gracefully, with proper logging or some action.

Eliminating root causes

Asking user not to close connection unexpectedly
This is impossible to do, for god’s sake

Reducing api response load
This is somehow achievable, but in a legacy system, operating on large amount of data, rewriting all the logic so that api responses aren’t heavy is also not feasible in all case.

Increasing Timeout of Server
Nginx has a variable named proxy_read_timeout which has default value of 60s, increasing this value will also minimize the chances of Broken Pipe.

Even after eliminating exact root cause, which itself is hard to detect in this case, we can’t completely rule out the existence of Broken Pipe. Can we?

Hence, next solution is to handle Broken Pipe gracefully.

Handling java.io.IOException: Broken Pipe

• Suppress the log from logger itself If you use log4j as log manager, adding following configuration to log4j.properties will help to get rid of exception flooding the logs due to Broken Pipe.

Exit fullscreen mode

• Upgrading Resteasy within Wildfly We were using Resteasy implementation of JAX-RS for implementing REST in Java, and Resteasy v3.0.19 is bundled with Wildfly 10.1. With hope of finding some fixes in the resteasy itself, I started to dig into release notes of Resteasy and found out that after Resteasy-Client v3.1.1, Unhandled Exception: java.io.IOException can be caught from code, unlike container exposing it directly onto logtrace. Hence, upgrading version of Resteasy Client allows us to handle exception through Global Exception Handler (described in later point).

Steps to upgrade Resteasy within Wildfly

Find location of Resteasy distribution jars:
The location of Resteasy distribution jars lies inside:
WILDFLY_HOME/modules/system/layers/base/org/jboss/resteasy/resteasy-jaxrs/main
and the file name is: resteasy-jaxrs-3.0.19.Final.jar

Download the zip file of target version of resteasy (3.1.1 and above) from Resteasy downloads

Unpack the file in temporary directory. It has two other zip files within it:
resteasy-jboss-modules-3.1.1.Final-mavenized.zip &
resteasy-jboss-modules-3.1.1.Final.zip

Unpack both .zip files at:
WILDFLY_HOME/modules/system/layers/base

Update resteasy dependencies’ version in pom.xml

Exit fullscreen mode

• Use Global Exception Handler I further added Generic Exception Mapper from the java code itself, to catch and handle the Broken Pipe.

My final approach

Though I had gathered multiple techniques to solve the issue, following are the things I actually implemented:

• Upgrade Wildfly to v11.0, as it automatically comes with Resteasy v3.1.1 (the version that I exactly wanted) bundled within it.
• Handle exception through Global Exception Handler

and with this, the issue of Broken Pipe now only exists in old logs archives 😎😎.

Parting words

It’s my first blog, yeah it’s a bit long, but I was giving insights into how I actually approached the issue and fixed it. Feel free to provide feedback in the comment section & watch this space for more development blogs in future.

Also, a bit of self promotion at the end 😉. Reach out to me at Twitter & LinkedIn

Источник

I’ve just experienced many/constant of the following ‘Broken pipe’ errors on Vaadin 14.0.15 and 14.1.16, using either a Chrome and/or a Brave browser on localhost where I was developing.

05:58:37.867 ERROR[http-nio-80-exec-8] org.apache.juli.logging.DirectJDKLog: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [com.vaadin.flow.server.ServiceException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe] with root cause
java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62)
	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:484)
	at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:138)
    ....
    ...

Apparently related, after one of the views had rendered, in the browser I saw a hovering box with red warning letters stating that ‘Cookies have been deactivated’ and that ‘the application needed cookies to work’ (or similar). The application was completely unusable. I note that cookies had not been deactivated for these browsers.

When I removed the all cookies on Brave, the ‘Broken pipe’ errors disappeared and the application became usable again. When reloading the same page on Chrome, the ‘Broken pipe’ errors were still present. Intrigued by this, I’ve printed the cookies on Chrome and I note that there are several cookies with a set name and value, but with path, domain and comment which are null.

************ Printing all cookies ************
1 - Name: Idea-f742bc50
1 - Value: 8ae30062-40d2-4cf0-b7ec-9362c14046af
1 - Path: null
1 - Domain: null
1 - Comment: null

2 - Name: ApplicationA
2 - Value: Somevalue
2 - Path: null
2 - Domain: null
2 - Comment: null

3 - Name: ApplicationB
3 - Value: 75:85:87:94:122:143:80:65:59:76:88:93:126:138:84:62
3 - Path: null
3 - Domain: null
3 - Comment: null

I note that this application I was working on did not set its own cookie (as ApplicationA or ApplicationB); I have set these for other unrelated projects.

Interested in your thoughts on what might be the issue here. I hope this helps to replay/isolate this issue. Let me know if I might assist.

OpenJDK 13.0.1
Spring Boot 2.2.0.RELEASE
Vaadin 14.0.15 / 14.1.16