Active MQ loses connection with HA database

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Active MQ loses connection with HA database

BharathMohan
Active MQ 5.14.0 loses connection to Database that is persisted on MS SQL
database.

Mulesoft reads the file and sends them as messages to topic

This messages are taken for transformation and written as file to target

HA cluster when failover occurs from primary to secondary

The messages that are enqueued are not dequeued as the DB connection is
closed.

Could you please check the attached mq xml file and let us know why activemq
not able to reach the secondary node when failover occurs.

activemq.xml
<http://activemq.2283324.n4.nabble.com/file/t378761/activemq.xml>  



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

Tim Bain
The place we'd want to look is the broker logs. Can you post the relevant
section around the time of the failover?

Tim

On Nov 6, 2017 9:04 AM, "BharathMohan" <[hidden email]> wrote:

Active MQ 5.14.0 loses connection to Database that is persisted on MS SQL
database.

Mulesoft reads the file and sends them as messages to topic

This messages are taken for transformation and written as file to target

HA cluster when failover occurs from primary to secondary

The messages that are enqueued are not dequeued as the DB connection is
closed.

Could you please check the attached mq xml file and let us know why activemq
not able to reach the secondary node when failover occurs.

activemq.xml
<http://activemq.2283324.n4.nabble.com/file/t378761/activemq.xml>



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

BharathMohan
activemq.log
<http://activemq.2283324.n4.nabble.com/file/t378761/activemq.log>  

017-11-07 10:57:26,808 | INFO  | Suppressing reply to: MessageAck {commandId
= 20591, responseRequired = false, ackType = 2, consumerId =
ID:QAUSNTYMUL02-52268-1510050833098-1:27:1:1, firstMessageId =
ID:QAUSNTYMUL02-52268-1510050833098-1:3:41174:1:1, lastMessageId =
ID:QAUSNTYMUL02-52268-1510050833098-1:3:41174:1:1, destination =
topic://topicTgtLearningDocumentInformationFromDRMS, transactionId = null,
messageCount = 1, poisonCause = null} on:
org.apache.activemq.broker.SuppressReplyException:
Stop/RestartTransportsInitiated, cause: java.io.IOException: The target
database, 'EDGE_MULE_APP', is participating in an availability group and is
currently not accessible for queries. Either data movement is suspended or
the availability replica is not enabled for read access. To allow read-only
access to this and other databases in the availability group, enable read
access to one or more secondary availability replicas in the group.  For
more information, see the ALTER AVAILABILITY GROUP statement in SQL Server
Books Online. ClientConnectionId:b43e8a21-2060-4dd0-82c5-656d65327955 |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.112.116.222:52599@8501
2017-11-07 10:57:27,808 | INFO  | The connection to
'tcp://10.112.116.222:52599' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | IOExceptionHandler: stop
transports
2017-11-07 10:57:27,839 | INFO  | Connector openwire stopped |
org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stop
transports
2017-11-07 10:57:27,839 | INFO  | Connector amqp stopped |
org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stop
transports
2017-11-07 10:57:27,839 | INFO  | Connector stomp stopped |
org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stop
transports
2017-11-07 10:57:27,855 | INFO  | Connector mqtt stopped |
org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stop
transports
2017-11-07 10:57:27,855 | INFO  | socketQueue interrupted - stopping |
org.apache.activemq.transport.tcp.TcpTransportServer | ActiveMQ Transport
Server Thread Handler:
amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2017-11-07 10:57:27,871 | ERROR | Could not accept connection  :
java.lang.InterruptedException |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2017-11-07 10:57:27,871 | INFO  | socketQueue interrupted - stopping |
org.apache.activemq.transport.tcp.TcpTransportServer | ActiveMQ Transport
Server Thread Handler:
stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2017-11-07 10:57:27,871 | ERROR | Could not accept connection  :
java.lang.InterruptedException |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2017-11-07 10:57:27,871 | INFO  | socketQueue interrupted - stopping |
org.apache.activemq.transport.tcp.TcpTransportServer | ActiveMQ Transport
Server Thread Handler:
mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2017-11-07 10:57:27,871 | ERROR | Could not accept connection  :
java.lang.InterruptedException |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2017-11-07 10:57:27,886 | INFO  | Connector ws stopped |
org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stop
transports
2017-11-07 10:57:27,886 | INFO  | Successfully stopped transports on
BrokerService[localhost] |
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler:
stop transports
2017-11-07 10:57:27,902 | INFO  | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler:
restart transports
2017-11-07 10:57:32,918 | INFO  | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler:
restart transports
2017-11-07 10:57:37,933 | INFO  | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler:
restart transports



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

BharathMohan
Tim

Did the logs helped you related to the issue

Thanks,
Bharath



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

Tim Bain
Seeing those logs sparked a half-memory of a thread on this mailing list
where the conclusion was that MSSQL HA wasn't supported yet but the OP of
the thread was encouraged to submit an enhancement request in JIRA. But I
spent half an hour this morning looking for that thread and couldn't find
it, so I think I may have been thinking of something else. I'll need to dig
into the MSSQL code to see how (if) it handles this exception. I'll get
back to you with what I find.

On Nov 9, 2017 9:50 AM, "BharathMohan" <[hidden email]> wrote:

> Tim
>
> Did the logs helped you related to the issue
>
> Thanks,
> Bharath
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

BharathMohan
Hi Tim,

Thanks a lot for the response.

To add more detail to the issue
1.) ESB Server is single instance that connects to Single instance of Active
MQ.
2.) This single Broker connects to MSSQL HA.

I will submit the JIRA for this issue.

Just wanted to confirm one thing from your reply "*MSSQL HA wasn't
supported*"

Are you saying that ActiveMQ 5.14.0 *does not support MSSQL HA Cluster at
all*.


Thanks,
Bharath




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

Tim Bain
Bharath,

I looked at the code that handles the connections to various different
database types (
https://github.com/apache/activemq/tree/master/activemq-jdbc-store/src/main/java/org/apache/activemq/store/jdbc/adapter).
I see specialized subclasses (which handle certain non-standard behaviors)
for a number of database types, including Oracle, MySQL, DB2, Sybase, etc.
I don't see one for MSSQL, which means that MSSQL uses DefaultJDBCAdapter
and there's no attempt to capture the MSSQL-specific logic of needing to
reconnect when you receive that specific exception.

So in summary, it appears that although MSSQL is supported, the HA features
of MSSQL are not, and you should submit an enhancement request in JIRA to
ask for that feature to be added.

Tim

On Nov 13, 2017 3:49 AM, "BharathMohan" <[hidden email]> wrote:

> Hi Tim,
>
> Thanks a lot for the response.
>
> To add more detail to the issue
> 1.) ESB Server is single instance that connects to Single instance of
> Active
> MQ.
> 2.) This single Broker connects to MSSQL HA.
>
> I will submit the JIRA for this issue.
>
> Just wanted to confirm one thing from your reply "*MSSQL HA wasn't
> supported*"
>
> Are you saying that ActiveMQ 5.14.0 *does not support MSSQL HA Cluster at
> all*.
>
>
> Thanks,
> Bharath
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

BharathMohan
Hi Tim,

Raised JIRA for the same

https://issues.apache.org/jira/browse/AMQ-6862

what will be the resolution time if this needs an enhancement.

Thanks,
Bharath



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ loses connection with HA database

Tim Bain
That's entirely unknown. Maybe someone involved in the development process
will feel it's important and will work on it for 5.16, or maybe not and
it'll never be implemented, it just depends. Since there hasn't been much
clamor on this mailing list for MSSQL HA, the enhancement will probably
only affect a small number of people (possibly only you), so the odds of it
not being worked are higher than if this was something that affected all
users, but anything is possible. The way to guarantee that it gets
implemented is to implement it yourself and contribute the code back, if
that was something you would be able to do.

Tim

On Nov 14, 2017 4:26 AM, "BharathMohan" <[hidden email]> wrote:

> Hi Tim,
>
> Raised JIRA for the same
>
> https://issues.apache.org/jira/browse/AMQ-6862
>
> what will be the resolution time if this needs an enhancement.
>
> Thanks,
> Bharath
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>