Admin console causing activemq service crash, on cluster environment when DB persistence is being used

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Admin console causing activemq service crash, on cluster environment when DB persistence is being used

All in
This post was updated on .
Hi,
In a activemq clustered environment (Mysql DB persisted) whenever there is DB connection glitch (DB is not available for a couple of seconds) at least 1  activemq cluster node gets into a weird state, which eventually will cause the termination of the activemq process. This his how i reproduce this behavior:

1. I have an activemq cluster composed of 2 nodes: Node 1 and Node 2. The messages persistence is done using a MYSQL instance all set and good here. If there are no issue on the network the nodes are properly being activated one at a time with the second node taking over when the first one fails.

    <broker brokerName="node1" xmlns="http://activemq.apache.org/schema/core">
        <persistenceAdapter>
            <jdbcPersistenceAdapter lockKeepAlivePeriod="1000" lockAcquireSleepInterval="2000" dataSource="#mysql-ds">
                                <locker>
                    <lease-database-locker lockAcquireSleepInterval="2000"/>
                </locker>
                                <statements>
                    <statements binaryDataType ="LONGBLOB"/>
                </statements>
                </jdbcPersistenceAdapter>
        </persistenceAdapter>
        <transportConnectors>
            <transportConnector name="default" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>
    </broker>

    <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <property name="driverClassName" value="com.mysql.jdbc.Driver"/>

2. Let's say that currently Node1 is  the owner of the lock. At this moment if i try to access the Node2 admin console i will not be able to get anything in the browser window. But i'm able to get the console for Node1

3. If i mess with the DB network (to simulate a network glitch) #> service network restart. If i'm lucky enough Node1 will loose the key and Node2 will become master. At this moment if i check again the admin consoles for the 2 nodes I can see that Node2 admin console is available, but Node1 console will be available too. Is jsut that it will present an error on the page (this behavior is different than the usual slave behavior.)

4. If at this point I stop the Node2 activemq service, I would have expected Node1 to take over. unfortunately Node1 will crash at this point with some java.net.BindException: Address already in use errors. You can find the logs for the failing Node1 below.

Any idea what is going on here?

Thanks


2017-02-03 13:22:58,193 | DEBUG | Scratch dir for the JSP engine is: /tmp/jetty-0.0.0.0-8161-api-_api-any-3039768726853340463.dir/jsp | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,193 | DEBUG | IMPORTANT: Do not modify the generated servlets | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,194 | DEBUG | Defaulting to use topics: true | org.apache.activemq.web.MessageServletSupport | WrapperSimpleAppMain
2017-02-03 13:22:58,197 | INFO  | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml | /api | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | InitialContextFactory.getInitialContext() | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@d0793bc | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Looking up name="comp/env/jmx/runtime" | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Trying thread context classloader | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | supportDeepBinding=false | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Made context comp for classloader: WebAppClassLoader=Apache ActiveMQ REST API@2c29a88d | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Looking up name="env/jmx/runtime" | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,207 | INFO  | jolokia-agent: jolokia:type=Config is already registered. Adding it with jolokia:type=Config,uuid=a5975fe0-a299-4be1-85c0-7431ef20c86c, but you should revise your setup in order to either use a qualifier or ensure, that only a single agent gets registered (otherwise history functionality might not work) | /api | WrapperSimpleAppMain
2017-02-03 13:22:58,207 | INFO  | jolokia-agent: Cannot register (legacy) MBean handler for config store with name jmx4perl:type=Config since it already exists. This is the case if another agent has been already started within the same JVM. The registration is skipped. | /api | WrapperSimpleAppMain
2017-02-03 13:22:58,207 | INFO  | jolokia-agent: Jolokia Discovery MBean registration is skipped because there is already one registered. | /api | WrapperSimpleAppMain
2017-02-03 13:22:58,208 | WARN  | FAILED ServerConnector@1175f42f{HTTP/1.1}{0.0.0.0:8161}: java.net.BindException: Address already in use | org.eclipse.jetty.util.component.AbstractLifeCycle | WrapperSimpleAppMain
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102]
        at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102]
        at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)[:1.8.0_102]
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:1.8.0_102]
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.config.MethodInvokingBean.invokeWithTargetException(MethodInvokingBean.java:119)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.config.MethodInvokingFactoryBean.afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1]
        at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
2017-02-03 13:22:58,210 | WARN  | FAILED org.eclipse.jetty.server.Server@5f70f898: java.net.BindException: Address already in use | org.eclipse.jetty.util.component.AbstractLifeCycle | WrapperSimpleAppMain
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102]
        at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102]
        at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)[:1.8.0_102]
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:1.8.0_102]
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.config.MethodInvokingBean.invokeWithTargetException(MethodInvokingBean.java:119)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.config.MethodInvokingFactoryBean.afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1]
        at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
2017-02-03 13:22:58,211 | WARN  | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.BindException: Address already in use | org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2017-02-03 13:22:58,214 | DEBUG | JspServlet.destroy() | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,217 | INFO  | Destroying Spring FrameworkServlet 'dispatcher' | /admin | WrapperSimpleAppMain
2017-02-03 13:22:58,218 | DEBUG | JspServlet.destroy() | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,222 | INFO  | Apache ActiveMQ 5.14.1 (jdbcBroker2, ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutting down | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2017-02-03 13:22:58,223 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | TRACE | Execute[ActiveMQ BrokerService[jdbcBroker2] Task] runnable: org.apache.activemq.broker.TransportConnection$4@7af5520a | org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | TRACE | Created and running thread[ActiveMQ BrokerService[jdbcBroker2] Task-2]: Thread[ActiveMQ BrokerService[jdbcBroker2] Task-2,5,main] | org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default,connectionViewType=clientId,connectionName=mtl-nes-dev-cn3 | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default,connectionViewType=remoteAddress,connectionName=tcp_//10.3.22.198_41192 | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | DEBUG | Stopping connection: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | DEBUG | Stopping transport tcp:///10.3.22.198:41192@61616 | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task] using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | TRACE | Execute[ActiveMQ Task] runnable: org.apache.activemq.transport.tcp.TcpTransport$1@7a152e97 | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | TRACE | Created and running thread[ActiveMQ Task-1]: Thread[ActiveMQ Task-1,5,main] | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | TRACE | Closing socket Socket[addr=/10.3.22.198,port=41192,localport=61616] | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2017-02-03 13:22:58,228 | DEBUG | Closed socket Socket[addr=/10.3.22.198,port=41192,localport=61616] | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2017-02-03 13:22:58,228 | DEBUG | Stopped transport: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,228 | TRACE | Shutdown timeout: 1 task: Transport Connection to: tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,228 | TRACE | Run task done: Transport Connection to: tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ Connection Dispatcher: tcp://10.3.22.198:41192
2017-02-03 13:22:58,230 | DEBUG | Cleaning up connection resources: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,230 | DEBUG | remove connection id: ID:mtl-nes-dev-cn3-38468-1486139024804-1:1 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,230 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Queue,destinationName=ComputeNodeQueue-Alin,endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-cn3-38468-1486139024804-1_1_1_1 | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,230 | DEBUG | jdbcBroker2 removing consumer: ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1 for destination: queue://ComputeNodeQueue-Alin | org.apache.activemq.broker.region.AbstractRegion | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,234 | DEBUG | queue://ComputeNodeQueue-Alin remove sub: QueueSubscription: consumer=ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1, destinations=1, dispatched=0, delivered=0, pending=0, prefetch=1, prefetchExtension=0, lastDeliveredSeqId: -2, dequeues: 0, dispatched: 0, inflight: 0, groups: 0 | org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,236 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.TempQueue_ActiveMQ.Advisory.TempTopic,endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-cn3-38468-1486139024804-1_1_-1_1 | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,236 | DEBUG | jdbcBroker2 removing consumer: ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:-1:1 for destination: ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic | org.apache.activemq.broker.region.AbstractRegion | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,237 | TRACE | Running task queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 | org.apache.activemq.thread.DedicatedTaskRunner | Queue:ComputeNodeQueue-Alin
2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,237 | DEBUG | queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 toPageIn: 0, force:false, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0, enqueueCount: 0, dequeueCount: 0, memUsage:0, maxPageSize:200 | org.apache.activemq.broker.region.Queue | Queue:ComputeNodeQueue-Alin
2017-02-03 13:22:58,238 | DEBUG | Connection Stopped: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,238 | INFO  | Connector default stopped | org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain
2017-02-03 13:22:58,239 | TRACE | Shutdown timeout: 0 task: queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,240 | TRACE | Run task done: queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 | org.apache.activemq.thread.DedicatedTaskRunner | Queue:ComputeNodeQueue-Alin
2017-02-03 13:22:58,241 | TRACE | ComputeNodeQueue-Alin,pendingSize:0 resetBatching. last recovered: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1] | org.apache.activemq.store.jdbc.JDBCMessageStore | WrapperSimpleAppMain
2017-02-03 13:22:58,241 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.Connection, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,242 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic  ActiveMQ.Advisory.MasterBroker
2017-02-03 13:22:58,242 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.Connection, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic  ActiveMQ.Advisory.Connection
2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.Queue, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.Queue, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic  ActiveMQ.Advisory.Queue
2017-02-03 13:22:58,243 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic  ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Connection | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Queue,destinationName=ComputeNodeQueue-Alin | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Queue | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | INFO  | PListStore:[/usr/local/Nuance/activemq/bin/linux-x86-64/activemq-data/jdbcBroker2/tmp_storage] stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | WrapperSimpleAppMain
2017-02-03 13:22:58,287 | INFO  | jdbcBroker2, released lease | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | WrapperSimpleAppMain
2017-02-03 13:22:58,287 | TRACE | Shutdown of ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Running, pool size = 4, active threads = 0, queued tasks = 2, completed tasks = 2] with await termination: 0 millis | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Shutdown of ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Shutting down, pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 2] is shutdown: true and terminated: false took: 0.001 seconds. | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Health | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Log4JConfiguration | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2 | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Stopping jmx connector | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | InitialContextFactory.getInitialContext() | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@574009c8 | jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | DEBUG | Unexported JMX RMI Registry | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | DEBUG | Forcing shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@68f23545[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | TRACE | Shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@68f23545[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown: true and terminated: true. | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | INFO  | Apache ActiveMQ 5.14.1 (jdbcBroker2, ID:mtl-nes-dev-web-48232-1486146033598-0:2) uptime 1 minute | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | INFO  | Apache ActiveMQ 5.14.1 (jdbcBroker2, ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutdown | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2017-02-03 13:22:58,295 | ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.BindException: Address already in use | org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.BindException: Address already in use
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1572)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1]
        at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
        at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
Reply | Threaded
Open this post in threaded view
|

Re: Failover problems when DB replication is being used

Tim Bain
Please submit a bug in JIRA. Please attach config files for both ActiveMQ
and MySQL, to make it easier for someone to reproduce; the database
persistence store code doesn't get a lot of love, so it's more likely
someone will investigate if you can provide everything needed to reproduce
the problem.  Does data need to be running for the problem to occur, or can
you make it happen without ever connecting a client?

Also, please take thread dumps of broker 1 at each step, and attach them to
the JIRA. Hopefully they will shed some light on what is going on.

Tim

On Feb 3, 2017 12:12 PM, "All in" <[hidden email]> wrote:

> Hi,
> In a activemq clustered environment (Mysql DB persisted) whenever there is
> DB connection glitch (DB is not available for a couple of seconds) at least
> 1  activemq cluster node gets into a weird state, which eventually will
> cause the termination of the activemq process. This his how i reproduce
> this
> behavior:
>
> 1. I have an activemq cluster composed of 2 nodes: Node 1 and Node 2. The
> messages persistence is done using a MYSQL instance all set and good here.
> If there are no issue on the network the nodes are properly being activated
> one at a time with the second node taking over when the first one fails.
>
>     <broker brokerName="node1"
> xmlns="http://activemq.apache.org/schema/core">
>         <persistenceAdapter>
>             <jdbcPersistenceAdapter lockKeepAlivePeriod="1000"
> lockAcquireSleepInterval="2000" dataSource="#mysql-ds">
>                                 <locker>
>                     <lease-database-locker lockAcquireSleepInterval="
> 2000"/>
>                 </locker>
>                                 <statements>
>                     <statements binaryDataType ="LONGBLOB"/>
>                 </statements>
>                 </jdbcPersistenceAdapter>
>         </persistenceAdapter>
>         <transportConnectors>
>             <transportConnector name="default" uri="tcp://0.0.0.0:61616"/>
>         </transportConnectors>
>     </broker>
>
>     <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
>         <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
>
> 2. Let's say that currently Node1 is  the owner of the lock. At this moment
> if i try to access the Node2 admin console i will not be able to get
> anything in the browser window. But i'm able to get the console for Node1
>
> 3. If i mess with the DB network (to simulate a network glitch) #> service
> network restart. If i'm lucky enough Node1 will loose the key and Node2
> will
> become master. At this moment if i check again the admin consoles for the 2
> nodes I can see that Node2 admin console is available, but Node1 console
> will be available too. Is jsut that it will present an error on the page
> (this behavior is different than the usual slave behavior.)
>
> 4. If at this point I stop the Node2 activemq service, I would have
> expected
> Node1 to take over. unfortunately Node1 will crash at this point with some
> java.net.BindException: Address already in use errors. You can find the
> logs
> for the failing Node1 below.
>
> Any idea what is going on here?
>
> Thanks
>
>
> 2017-02-03 13:22:58,193 | DEBUG | Scratch dir for the JSP engine is:
> /tmp/jetty-0.0.0.0-8161-api-_api-any-3039768726853340463.dir/jsp |
> org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
> 2017-02-03 13:22:58,193 | DEBUG | IMPORTANT: Do not modify the generated
> servlets | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
> 2017-02-03 13:22:58,194 | DEBUG | Defaulting to use topics: true |
> org.apache.activemq.web.MessageServletSupport | WrapperSimpleAppMain
> 2017-02-03 13:22:58,197 | INFO  | jolokia-agent: Using policy access
> restrictor classpath:/jolokia-access.xml | /api | WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | InitialContextFactory.
> getInitialContext()
> | jndi | WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | Created initial context delegate for
> local
> namespace:org.eclipse.jetty.jndi.local.localContextRoot@d0793bc | jndi |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | Looking up name="comp/env/jmx/runtime" |
> jndi | WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | Trying thread context classloader | jndi
> |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | supportDeepBinding=false | jndi |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | Made context comp for classloader:
> WebAppClassLoader=Apache ActiveMQ REST API@2c29a88d | jndi |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,203 | DEBUG | Looking up name="env/jmx/runtime" | jndi
> |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,207 | INFO  | jolokia-agent: jolokia:type=Config is
> already registered. Adding it with
> jolokia:type=Config,uuid=a5975fe0-a299-4be1-85c0-7431ef20c86c, but you
> should revise your setup in order to either use a qualifier or ensure, that
> only a single agent gets registered (otherwise history functionality might
> not work) | /api | WrapperSimpleAppMain
> 2017-02-03 13:22:58,207 | INFO  | jolokia-agent: Cannot register (legacy)
> MBean handler for config store with name jmx4perl:type=Config since it
> already exists. This is the case if another agent has been already started
> within the same JVM. The registration is skipped. | /api |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,207 | INFO  | jolokia-agent: Jolokia Discovery MBean
> registration is skipped because there is already one registered. | /api |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,208 | WARN  | FAILED
> ServerConnector@1175f42f{HTTP/1.1}{0.0.0.0:8161}: java.net.BindException:
> Address already in use | org.eclipse.jetty.util.
> component.AbstractLifeCycle
> | WrapperSimpleAppMain
> java.net.BindException: Address already in use
>         at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102]
>         at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102]
>         at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102]
>         at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:
> 223)[:1.8.0_102]
>         at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:
> 1.8.0_102]
>         at
> org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[
> jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
>         at
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(
> AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.
> v20150730]
>         at
> org.eclipse.jetty.server.ServerConnector.doStart(
> ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.
> start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.
> v20150730]
>         at
> org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.
> v20150730.jar:9.2.13.v20150730]
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.
> start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.
> v20150730]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[
> spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.config.MethodInvokingBean.
> invokeWithTargetException(MethodInvokingBean.java:119)[
> spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.config.MethodInvokingFactoryBean.
> afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.invokeInitMethods(AbstractAutowireCapableBeanFac
> tory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.initializeBean(AbstractAutowireCapableBeanFac
> tory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.doCreateBean(AbstractAutowireCapableBeanFac
> tory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.createBean(AbstractAutowireCapableBeanFac
> tory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.
> getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.
> getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(
> AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.
> jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
> AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.
> jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.
> preInstantiateSingletons(DefaultListableBeanFactory.
> java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.context.support.AbstractApplicationContext.
> finishBeanFactoryInitialization(AbstractApplicationContext.
> java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.context.support.AbstractApplicationContext.refresh(
> AbstractApplicationContext.java:480)[spring-context-4.1.
> 9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(
> ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
>         at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(
> ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(
> XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(
> XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(
> XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.broker.BrokerFactory.createBroker(
> BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.broker.BrokerFactory.createBroker(
> BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.StartCommand.runTask(
> StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.AbstractCommand.
> execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.ShellCommand.runTask(
> ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.AbstractCommand.
> execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.ShellCommand.main(
> ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.apache.activemq.console.Main.runTaskClass(Main.java:
> 262)[activemq.jar:5.14.1]
>         at
> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.tanukisoftware.wrapper.WrapperSimpleApp.run(
> WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
>         at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
> 2017-02-03 13:22:58,210 | WARN  | FAILED
> org.eclipse.jetty.server.Server@5f70f898: java.net.BindException: Address
> already in use | org.eclipse.jetty.util.component.AbstractLifeCycle |
> WrapperSimpleAppMain
> java.net.BindException: Address already in use
>         at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102]
>         at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102]
>         at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102]
>         at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:
> 223)[:1.8.0_102]
>         at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:
> 1.8.0_102]
>         at
> org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[
> jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
>         at
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(
> AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.
> v20150730]
>         at
> org.eclipse.jetty.server.ServerConnector.doStart(
> ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.
> start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.
> v20150730]
>         at
> org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.
> v20150730.jar:9.2.13.v20150730]
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.
> start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.
> v20150730]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[
> spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.config.MethodInvokingBean.
> invokeWithTargetException(MethodInvokingBean.java:119)[
> spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.config.MethodInvokingFactoryBean.
> afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.invokeInitMethods(AbstractAutowireCapableBeanFac
> tory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.initializeBean(AbstractAutowireCapableBeanFac
> tory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.doCreateBean(AbstractAutowireCapableBeanFac
> tory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.createBean(AbstractAutowireCapableBeanFac
> tory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.
> getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.
> getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(
> AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.
> jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
> AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.
> jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.
> preInstantiateSingletons(DefaultListableBeanFactory.
> java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.context.support.AbstractApplicationContext.
> finishBeanFactoryInitialization(AbstractApplicationContext.
> java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.context.support.AbstractApplicationContext.refresh(
> AbstractApplicationContext.java:480)[spring-context-4.1.
> 9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(
> ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
>         at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(
> ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(
> XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(
> XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(
> XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.broker.BrokerFactory.createBroker(
> BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.broker.BrokerFactory.createBroker(
> BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.StartCommand.runTask(
> StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.AbstractCommand.
> execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.ShellCommand.runTask(
> ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.AbstractCommand.
> execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.ShellCommand.main(
> ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.apache.activemq.console.Main.runTaskClass(Main.java:
> 262)[activemq.jar:5.14.1]
>         at
> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.tanukisoftware.wrapper.WrapperSimpleApp.run(
> WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
>         at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
> 2017-02-03 13:22:58,211 | WARN  | Exception encountered during context
> initialization - cancelling refresh attempt:
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean
> with name 'invokeStart' defined in class path resource [jetty.xml]:
> Invocation of init method failed; nested exception is
> java.net.BindException: Address already in use |
> org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
> 2017-02-03 13:22:58,214 | DEBUG | JspServlet.destroy() |
> org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
> 2017-02-03 13:22:58,217 | INFO  | Destroying Spring FrameworkServlet
> 'dispatcher' | /admin | WrapperSimpleAppMain
> 2017-02-03 13:22:58,218 | DEBUG | JspServlet.destroy() |
> org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
> 2017-02-03 13:22:58,222 | INFO  | Apache ActiveMQ 5.14.1 (jdbcBroker2,
> ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutting down |
> org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
> 2017-02-03 13:22:58,223 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> connector=clientConnectors,connectorName=default
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,225 | TRACE | Execute[ActiveMQ
> BrokerService[jdbcBroker2] Task] runnable:
> org.apache.activemq.broker.TransportConnection$4@7af5520a |
> org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain
> 2017-02-03 13:22:58,225 | TRACE | Created and running thread[ActiveMQ
> BrokerService[jdbcBroker2] Task-2]: Thread[ActiveMQ
> BrokerService[jdbcBroker2] Task-2,5,main] |
> org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain
> 2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> connector=clientConnectors,connectorName=default,
> connectionViewType=clientId,connectionName=mtl-nes-dev-cn3
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> connector=clientConnectors,connectorName=default,connectionViewType=
> remoteAddress,connectionName=tcp_//10.3.22.198_41192
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,225 | DEBUG | Stopping connection:
> tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection |
> ActiveMQ BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,227 | DEBUG | Stopping transport
> tcp:///10.3.22.198:41192@61616 |
> org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,227 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ
> Task] using ExecutorService: null |
> org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,227 | TRACE | Execute[ActiveMQ Task] runnable:
> org.apache.activemq.transport.tcp.TcpTransport$1@7a152e97 |
> org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,227 | TRACE | Created and running thread[ActiveMQ
> Task-1]: Thread[ActiveMQ Task-1,5,main] |
> org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,227 | TRACE | Closing socket
> Socket[addr=/10.3.22.198,port=41192,localport=61616] |
> org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
> 2017-02-03 13:22:58,228 | DEBUG | Closed socket
> Socket[addr=/10.3.22.198,port=41192,localport=61616] |
> org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
> 2017-02-03 13:22:58,228 | DEBUG | Stopped transport: tcp://
> 10.3.22.198:41192
> | org.apache.activemq.broker.TransportConnection | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,228 | TRACE | Shutdown timeout: 1 task: Transport
> Connection to: tcp://10.3.22.198:41192 |
> org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,228 | TRACE | Run task done: Transport Connection to:
> tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner |
> ActiveMQ Connection Dispatcher: tcp://10.3.22.198:41192
> 2017-02-03 13:22:58,230 | DEBUG | Cleaning up connection resources:
> tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection |
> ActiveMQ BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,230 | DEBUG | remove connection id:
> ID:mtl-nes-dev-cn3-38468-1486139024804-1:1 |
> org.apache.activemq.broker.TransportConnection | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,230 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Queue,destinationName=ComputeNodeQueue-Alin,
> endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_
> mtl-nes-dev-cn3-38468-1486139024804-1_1_1_1
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,230 | DEBUG | jdbcBroker2 removing consumer:
> ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1 for destination:
> queue://ComputeNodeQueue-Alin |
> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,234 | DEBUG | queue://ComputeNodeQueue-Alin remove sub:
> QueueSubscription: consumer=ID:mtl-nes-dev-cn3-
> 38468-1486139024804-1:1:1:1,
> destinations=1, dispatched=0, delivered=0, pending=0, prefetch=1,
> prefetchExtension=0, lastDeliveredSeqId: -2, dequeues: 0, dispatched: 0,
> inflight: 0, groups: 0 | org.apache.activemq.broker.region.Queue |
> ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
> for broker transport URI: tcp://mtl-nes-dev-web:61616 |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
> for broker transport URI: tcp://mtl-nes-dev-web:61616 |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,236 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Topic,destinationName=ActiveMQ.
> Advisory.TempQueue_ActiveMQ.Advisory.TempTopic,endpoint=
> Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-
> cn3-38468-1486139024804-1_1_-1_1
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,236 | DEBUG | jdbcBroker2 removing consumer:
> ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:-1:1 for destination:
> ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic |
> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,237 | TRACE | Running task
> queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0,
> pending=0
> | org.apache.activemq.thread.DedicatedTaskRunner |
> Queue:ComputeNodeQueue-Alin
> 2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
> for broker transport URI: tcp://mtl-nes-dev-web:61616 |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
> for broker transport URI: tcp://mtl-nes-dev-web:61616 |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,237 | DEBUG | queue://ComputeNodeQueue-Alin,
> subscriptions=0, memory=0%, size=0, pending=0 toPageIn: 0, force:false,
> Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
> enqueueCount: 0, dequeueCount: 0, memUsage:0, maxPageSize:200 |
> org.apache.activemq.broker.region.Queue | Queue:ComputeNodeQueue-Alin
> 2017-02-03 13:22:58,238 | DEBUG | Connection Stopped:
> tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection |
> ActiveMQ BrokerService[jdbcBroker2] Task-2
> 2017-02-03 13:22:58,238 | INFO  | Connector default stopped |
> org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain
> 2017-02-03 13:22:58,239 | TRACE | Shutdown timeout: 0 task:
> queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0,
> pending=0
> | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
> 2017-02-03 13:22:58,240 | TRACE | Run task done:
> queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0,
> pending=0
> | org.apache.activemq.thread.DedicatedTaskRunner |
> Queue:ComputeNodeQueue-Alin
> 2017-02-03 13:22:58,241 | TRACE | ComputeNodeQueue-Alin,pendingSize:0
> resetBatching. last recovered: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1] |
> org.apache.activemq.store.jdbc.JDBCMessageStore | WrapperSimpleAppMain
> 2017-02-03 13:22:58,241 | TRACE | Shutdown timeout: 0 task: Topic:
> destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 |
> org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
> 2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic:
> destination=ActiveMQ.Advisory.Connection, subscriptions=0 |
> org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
> 2017-02-03 13:22:58,242 | TRACE | Run task done: Topic:
> destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 |
> org.apache.activemq.thread.DedicatedTaskRunner | Topic
> ActiveMQ.Advisory.MasterBroker
> 2017-02-03 13:22:58,242 | TRACE | Run task done: Topic:
> destination=ActiveMQ.Advisory.Connection, subscriptions=0 |
> org.apache.activemq.thread.DedicatedTaskRunner | Topic
> ActiveMQ.Advisory.Connection
> 2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic:
> destination=ActiveMQ.Advisory.Queue, subscriptions=0 |
> org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | TRACE | Run task done: Topic:
> destination=ActiveMQ.Advisory.Queue, subscriptions=0 |
> org.apache.activemq.thread.DedicatedTaskRunner | Topic
> ActiveMQ.Advisory.Queue
> 2017-02-03 13:22:58,243 | TRACE | Shutdown timeout: 0 task: Topic:
> destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin,
> subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | TRACE | Run task done: Topic:
> destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin,
> subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic
> ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin
> 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Topic,destinationName=ActiveMQ.Advisory.Connection
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Queue,destinationName=ComputeNodeQueue-Alin
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Topic,destinationName=ActiveMQ.Advisory.Queue
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.
> ComputeNodeQueue-Alin
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,243 | INFO  |
> PListStore:[/usr/local/Nuance/activemq/bin/linux-x86-64/
> activemq-data/jdbcBroker2/tmp_storage]
> stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,287 | INFO  | jdbcBroker2, released lease |
> org.apache.activemq.store.jdbc.LeaseDatabaseLocker | WrapperSimpleAppMain
> 2017-02-03 13:22:58,287 | TRACE | Shutdown of ExecutorService:
> java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Running, pool
> size
> = 4, active threads = 0, queued tasks = 2, completed tasks = 2] with await
> termination: 0 millis | org.apache.activemq.util.ThreadPoolUtils |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | Shutdown of ExecutorService:
> java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Shutting down,
> pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 2]
> is
> shutdown: true and terminated: false took: 0.001 seconds. |
> org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Health |
> org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2,
> service=Log4JConfiguration
> | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=jdbcBroker2 |
> org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | Stopping jmx connector |
> org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | InitialContextFactory.
> getInitialContext()
> | jndi | WrapperSimpleAppMain
> 2017-02-03 13:22:58,288 | DEBUG | Created initial context delegate for
> local
> namespace:org.eclipse.jetty.jndi.local.localContextRoot@574009c8 | jndi |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,292 | DEBUG | Unexported JMX RMI Registry |
> org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
> 2017-02-03 13:22:58,292 | DEBUG | Forcing shutdown of ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@68f23545[Running, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 0] |
> org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
> 2017-02-03 13:22:58,292 | TRACE | Shutdown of ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@68f23545[Terminated, pool size =
> 0,
> active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown:
> true
> and terminated: true. | org.apache.activemq.util.ThreadPoolUtils |
> WrapperSimpleAppMain
> 2017-02-03 13:22:58,292 | INFO  | Apache ActiveMQ 5.14.1 (jdbcBroker2,
> ID:mtl-nes-dev-web-48232-1486146033598-0:2) uptime 1 minute |
> org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
> 2017-02-03 13:22:58,292 | INFO  | Apache ActiveMQ 5.14.1 (jdbcBroker2,
> ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutdown |
> org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
> 2017-02-03 13:22:58,295 | ERROR | Failed to load: class path resource
> [activemq.xml], reason: Error creating bean with name 'invokeStart' defined
> in class path resource [jetty.xml]: Invocation of init method failed;
> nested
> exception is java.net.BindException: Address already in use |
> org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean
> with name 'invokeStart' defined in class path resource [jetty.xml]:
> Invocation of init method failed; nested exception is
> java.net.BindException: Address already in use
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.initializeBean(AbstractAutowireCapableBeanFac
> tory.java:1572)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.doCreateBean(AbstractAutowireCapableBeanFac
> tory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.createBean(AbstractAutowireCapableBeanFac
> tory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.
> getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.
> getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.
> RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(
> AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.
> jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
> AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.
> jar:4.1.9.RELEASE]
>         at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.
> preInstantiateSingletons(DefaultListableBeanFactory.
> java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.context.support.AbstractApplicationContext.
> finishBeanFactoryInitialization(AbstractApplicationContext.
> java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.springframework.context.support.AbstractApplicationContext.refresh(
> AbstractApplicationContext.java:480)[spring-context-4.1.
> 9.RELEASE.jar:4.1.9.RELEASE]
>         at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(
> ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
>         at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(
> ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(
> XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(
> XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(
> XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.broker.BrokerFactory.createBroker(
> BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.broker.BrokerFactory.createBroker(
> BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.StartCommand.runTask(
> StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.AbstractCommand.
> execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.ShellCommand.runTask(
> ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.AbstractCommand.
> execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
>         at
> org.apache.activemq.console.command.ShellCommand.main(
> ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.apache.activemq.console.Main.runTaskClass(Main.java:
> 262)[activemq.jar:5.14.1]
>         at
> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_102]
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)[:1.8.0_102]
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
>         at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
>         at
> org.tanukisoftware.wrapper.WrapperSimpleApp.run(
> WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
>         at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Failover-problems-when-DB-replication-is-being-
> used-tp4721586.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failover problems when DB replication is being used

All in
Hi Tim,
I've created the JIRA : https://issues.apache.org/jira/browse/AMQ-6592.

To answer some of your questions, no data has to be running on the queues.
One thing i noticed is that if i disable the admin console by commenting out the line:  <import resource="jetty.xml"/> in the activemq.xml, the problem will not show up.
It looks to me the problem is related to the GUI admin console activation/deactivation mechanism.

All
Reply | Threaded
Open this post in threaded view
|

Re: Failover problems when DB replication is being used

Tim Bain
Two of the three instances of the error saying that the port was in use
were related to Jetty, but the third looked unrelated.  Thanks for
clarifying that all three relate only to Jetty.

Tim

On Feb 6, 2017 7:34 AM, "All in" <[hidden email]> wrote:

Hi Tim,
I've created the JIRA : https://issues.apache.org/jira/browse/AMQ-6592.

To answer some of your questions, no data has to be running on the queues.
One thing i noticed is that if i disable the admin console by commenting out
the line:  <import resource="jetty.xml"/> in the activemq.xml, the problem
will not show up.
It looks to me the problem is related to the GUI admin console
activation/deactivation mechanism.

All




--
View this message in context: http://activemq.2283324.n4.
nabble.com/Admin-console-causing-activemq-service-
crash-on-cluster-environment-when-DB-persistence-is-being-
used-tp4721586p4721619.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Failover problems when DB replication is being used

Venkatesh Laguduva
In reply to this post by All in
I am wondering if you have got the shutdownhook set so that jetty goes down properly:


        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>