Failed network connector can not be re-established

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Failed network connector can not be re-established

alt_alt
Configuration for network connector:

<networkConnector name="external01_01" conduitSubscriptions="false" uri="static://(tcp://10.102.44.181:61616)" userName="MQ-PRD-ADMIN" password="$
{connector.password}
">
<excludedDestinations>
<topic physicalName="VirtualTopic.>"/>
</excludedDestinations>
</networkConnector>
log:
2016-03-03 14:41:13,070 | WARN | JDBC Failure: Borrow prepareStatement from pool failed | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | ActiveMQ Transport: tcp://10.102.44.181:61616@38805
org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed
at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)[commons-dbcp-1.4.jar:1.4]
at org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRemoveMessage(DefaultJDBCAdapter.java:381)[activemq-jdbc-store-5.11.1.jar:5.11.1]
at org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:260)[activemq-jdbc-store-5.11.1.jar:5.11.1]
at org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:373)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:170)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:923)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1718)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:55)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:277)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:441)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:484)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:97)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:550)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.network.DemandForwardingBridgeSupport$11.onCompletion(DemandForwardingBridgeSupport.java:1012)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:65)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)[activemq-client-5.11.1.jar:5.11.1]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.11.1.jar:5.11.1]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_80]
2016-03-03 14:41:13,073 | WARN | Transport Connection to: vm://PRD_ACTIVEMQ_C1#0 failed: java.io.IOException: Failed to broker message: ID:*-36573-1456931506213-1:3:1:1:7874 in container: org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp://10.102.44.181:61616@38805
2016-03-03 14:41:13,074 | INFO | PRD_ACTIVEMQ_C1 Shutting down | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[PRD_ACTIVEMQ_C1] Task-25434
2016-03-03 14:41:23,080 | INFO | Network Could not shutdown in a timely manner | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[PRD_ACTIVEMQ_C1] Task-25434
2016-03-03 14:41:24,083 | INFO | PRD_ACTIVEMQ_C1 bridge to localhost stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[PRD_ACTIVEMQ_C1] Task-25434

It is very URGENT since we found this issue on our production environment! Please help !!!!!!
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

Tim Bain
"JDBC Failure: Borrow prepareStatement from pool failed"

Your problem is in the JDBC store (specifically the connection pool), whose
configuration you didn't post or tell us anything about.

One possible workaround would be to simply tell the DataSource to not pool
prepared statements via
https://commons.apache.org/proper/commons-dbcp/api-1.4/org/apache/commons/dbcp/BasicDataSource.html#poolPreparedStatements
.
On Mar 8, 2016 9:24 PM, "alt_alt" <[hidden email]> wrote:

> *Configuration for network connector:
> *
> <networkConnector name="external01_01" conduitSubscriptions="false"
> uri="static://(tcp://10.102.44.181:61616)" userName="MQ-PRD-ADMIN"
> password="$
> {connector.password}
> ">
> <excludedDestinations>
> <topic physicalName="VirtualTopic.>"/>
> </excludedDestinations>
> </networkConnector>
> *log:
> *2016-03-03 14:41:13,070 | WARN | JDBC Failure: Borrow prepareStatement
> from
> pool failed | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter |
> ActiveMQ Transport: tcp://10.102.44.181:61616@38805
> org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from
> pool failed
> at
>
> org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)[commons-dbcp-1.4.jar:1.4]
> at
>
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)[commons-dbcp-1.4.jar:1.4]
> at
>
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)[commons-dbcp-1.4.jar:1.4]
> at
>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRemoveMessage(DefaultJDBCAdapter.java:381)[activemq-jdbc-store-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:260)[activemq-jdbc-store-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:373)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:170)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:923)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1718)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:55)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:277)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:441)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:484)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:87)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:97)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:550)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:130)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.network.DemandForwardingBridgeSupport$11.onCompletion(DemandForwardingBridgeSupport.java:1012)[activemq-broker-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:65)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)[activemq-client-5.11.1.jar:5.11.1]
> at
>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.11.1.jar:5.11.1]
> at java.lang.Thread.run(Thread.java:745)[:1.7.0_80]
> 2016-03-03 14:41:13,073 | WARN | Transport Connection to:
> vm://PRD_ACTIVEMQ_C1#0 failed: java.io.IOException: Failed to broker
> message: ID:*-36573-1456931506213-1:3:1:1:7874 in container:
> org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from
> pool failed | org.apache.activemq.broker.TransportConnection.Transport |
> ActiveMQ Transport: tcp://10.102.44.181:61616@38805
> 2016-03-03 14:41:13,074 | INFO | PRD_ACTIVEMQ_C1 Shutting down |
> org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[PRD_ACTIVEMQ_C1] Task-25434
> 2016-03-03 14:41:23,080 | INFO | Network Could not shutdown in a timely
> manner | org.apache.activemq.network.DemandForwardingBridgeSupport |
> ActiveMQ BrokerService[PRD_ACTIVEMQ_C1] Task-25434
> 2016-03-03 14:41:24,083 | INFO | PRD_ACTIVEMQ_C1 bridge to localhost
> stopped
> | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[PRD_ACTIVEMQ_C1] Task-25434
>
> *It is very URGENT since we found this issue on our production environment!
> Please help !!!!!!*
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

alt_alt
This post was updated on .
Hi Tim,
Really appreciated for your reply, here is the datasource configuration, just the same as the one shown in ActiveMQ webpage.

    <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
       <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
       <property name="url" value="jdbc:mysql://*.*.*.*:3306/activemq?relaxAutoCommit=true"/>
       <property name="username" value="ACTIVEMQSERVER"/>
       <property name="password" value="${jdbc.password}"/>
       <property name="poolPreparedStatements" value="true"/>
    </bean>

What's the performance impact if I set poolPreparedStatements to false?

Best Regards
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

Tim Bain
In reply to this post by Tim Bain
I've never run ActiveMQ with the JDBC persistence store, so I can't answer
that question directly, and I don't get the sense that most of the people
on this mailing list have either, so you may not be able to answer that
question other than by trying it.  With that being said, I wouldn't expect
it to be that large of a hit.

Tim

On Wed, Mar 9, 2016 at 5:10 PM, alt_alt <[hidden email]> wrote:

> Hi Tim,
> Really appreciated for your reply, here is the datasource configuration,
> just the same as the one shown in ActiveMQ webpage.
>
>     <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
>        <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
>        <property name="url"
> value="jdbc:mysql://
> eanprdactivemq01writer01.eao.abn-iad.ea.com:3306/activemq?relaxAutoCommit=true
> "/>
>        <property name="username" value="ACTIVEMQSERVER"/>
>        <property name="password" value="${jdbc.password}"/>
>        <property name="poolPreparedStatements" value="true"/>
>     </bean>
>
> What's the performance impact if I set poolPreparedStatements to false?
>
> Best Regards
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709136.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

alt_alt
Hi Tim,
Thanks for your useful tips. I agree that there may be some configuration problem in JDBS store, but I don't think the network connector should just be stopped and never be restarted again without restarting ActiveMQ service due to exeption in JDBC store. I don't think it is an acceptable behavior for ActiveMQ network relying on the message forwarding. Maybe you could add some feature to let the network still alive even such exception happens. For example, adding a peoperty of ignoreLocalIOException for networkConnector?

Best Regards
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

Tim Bain
I think the current behavior of stopping the networkConnector when it will
be impossible for it to do any work (because we can't write to the
persistent store) is the right behavior; having a zombie networkConnector
hanging around just hoping for the store to become available sounds like an
awful option for a number of reasons, one of which is that it will "steal"
as many messages as its prefetch buffer size so your consumers will be
missing messages.  And no, simply ignoring all IOExceptions is not the
right approach; what does it mean to ignore an EOFException or a
SocketTimeoutException?

But it seems reasonable to have a periodic retry if the networkConnector
fails.  Submit an enhancement request in JIRA and maybe someone will
implement it.  (Or implement it yourself and submit a patch or a pull
request, if you want it sooner.)

In the meantime, disabling statement pooling should let you work around
this.  Or if you think statement pooling is an absolute must-have, you
could move to Oracle, since their driver can do its own pooling of both
connections and statements, if I'm remembering correctly.

Tim
On Mar 10, 2016 4:43 PM, "alt_alt" <[hidden email]> wrote:

> Hi Tim,
> Thanks for your useful tips. I agree that there may be some configuration
> problem in JDBS store, but I don't think the network connector should just
> be stopped and never be restarted again without restarting ActiveMQ service
> due to exeption in JDBC store. I don't think it is an acceptable behavior
> for ActiveMQ network relying on the message forwarding. Maybe you could add
> some feature to let the network still alive even such exception happens.
> For
> example, adding a peoperty of ignoreLocalIOException for networkConnector?
>
> Best Regards
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709183.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

alt_alt
Hi Tim,
Thanks for your advice, I've already filed a jira ticket for it. It is AMQ-6200
Add a retry mechanism would be a decent solution to my problem, and I agree with you that just ignoring the exception is not a good one. Please add this feature to future ActiveMQ release. We can disable pooling preparestatement. But I am looking for another way as a workaround, I am enabling testOnBorrow or testwhileIdle to see whether that could help since I doubt it is because we used invalid connection to create preparestatement and throw that exception.

Anyway, adding retry for network connector is definitely a good improvement. :)

Best Regards
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

artnaseef
Any idea what is causing "Borrow prepareStatement from pool failed"?

The only possible cause coming to mind right now is a pool that is exhausted - meaning there are long-running DB operations.  If that's the case, then tuning the DB, moving to another store, or possible tuning the pooling settings for JDBC may help.

It is also possible here that such a scenario will occur when there are too many messages backing up in the broker due to slow consumption.  If so, then it is critical to eliminate the slow consumption.
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

artnaseef
Also - network connectors typically automatically reconnect.

Is the network connector configured to use the failover transport?  If not, try adding it to see if that helps to eliminate the problem.
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

artnaseef
Ahh, here it is from the original post:

uri="static://(tcp://10.102.44.181:61616)"

try this instead:

uri="static://(failover://(tcp://10.102.44.181:61616))"
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

Tim Bain
In reply to this post by artnaseef
Other possibilities are a bug in ActiveMQ code where we fail to close() our
PreparedStatements in certain cases, and a bug in the connection pool.
On Mar 12, 2016 11:13 PM, "artnaseef" <[hidden email]> wrote:

> Any idea what is causing "Borrow prepareStatement from pool failed"?
>
> The only possible cause coming to mind right now is a pool that is
> exhausted
> - meaning there are long-running DB operations.  If that's the case, then
> tuning the DB, moving to another store, or possible tuning the pooling
> settings for JDBC may help.
>
> It is also possible here that such a scenario will occur when there are too
> many messages backing up in the broker due to slow consumption.  If so,
> then
> it is critical to eliminate the slow consumption.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709240.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

alt_alt
In reply to this post by artnaseef
Thanks for you advice for tuning JDBC and configuration of network connector, and we'll definitely try that. But it looks like the network connector is closed from the source side due to local JDBC exception(the source side means the source ActiveMQ server of forwarded message via network connector), I am not sure whether failover makes difference, but we'll try it first.
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

Tim Bain
In reply to this post by artnaseef
Art, it's always been my understanding that static TCP transports would
reconnect when used as a networkConnector, and that the only reason to use
failover is if you have multiple brokers.  Is that not accurate?
On Mar 12, 2016 11:17 PM, "artnaseef" <[hidden email]> wrote:

> Ahh, here it is from the original post:
>
> uri="static://(tcp://10.102.44.181:61616)"
>
> try this instead:
>
> uri="static://(failover://(tcp://10.102.44.181:61616))"
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-network-connector-can-not-be-re-established-tp4709054p4709242.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed network connector can not be re-established

artnaseef
Hey Tim, the demand forwarding network connector should automatically attempt to reconnect.  At one point, years ago, I tried removing the failover transport and sticking to the reties there because of other issues and found that without the failover transport, the reconnects are not as reliable.

In this instance, with the exception involved, I suspect the bridge's reconnect logic may be failing.

So, I personally always recommend using the failover transport with network connections.

If you find otherwise, I would be interested to learn what you find.