Duplex networkConnector error with mKahaDB after inactive destinations deleted

Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Duplex networkConnector error with mKahaDB after inactive destinations deleted

nexus
This post was updated on .

Hello,

Encountering following issue with network of brokers on Windows 10 / 64-bit / ActiveMQ v5.15.12

Problem

Cannot re-establish duplex network connection on remote broker with following error:

INFO   | jvm 1    | 2020/03/21 14:41:32 | ERROR | Failed to create responder end of duplex network bridge Q-bridge@ID:LHC-59471-1584794492297-0:1
INFO   | jvm 1    | 2020/03/21 14:41:32 | java.lang.IllegalStateException: PageFile is not loaded
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.store.kahadb.disk.page.PageFile.assertLoaded(PageFile.java:906)[activemq-kahadb-store-5.15.12.jar:5.15.12]
...

Remote broker is configured to delete inactive destinations and uses mKahaDB persistence adapters for different destinations (Automatic Per Destination Persistence Adapter).

Same setup, but single kahaDB persistence adapter on remote broker is not causing the issue.

See attached files for detailed configuration and logs:

Use case

Simulate network connection loss and then re-astablish duplex communication after remote broker destinations were purged due to inactivity:

  1. start remote broker
  2. start local broker
  3. destination queue created automatically on remote (active consumer from local broker is also shown correctly in web-console)
  4. stop local broker
  5. wait for a while until destination is deleted on remote due to inactivity
  6. start local broker again

Results

GOOD: single kahaDB persistence adapter on remote broker

INFO   | jvm 1    | 2020/03/21 14:28:34 |  INFO | ActiveMQ WebConsole available at http://0.0.0.0:8161/
INFO   | jvm 1    | 2020/03/21 14:28:34 |  INFO | ActiveMQ Jolokia REST API available at http://0.0.0.0:8161/api/jolokia/
INFO   | jvm 1    | 2020/03/21 14:28:43 |  INFO | Connector vm://remote started
INFO   | jvm 1    | 2020/03/21 14:28:43 |  INFO | Started responder end of duplex bridge Q-bridge@ID:LHC-59179-1584793723328-0:1
INFO   | jvm 1    | 2020/03/21 14:28:43 |  INFO | Network connection between vm://remote#0 and tcp:///192.168.8.109:59180@61616 (local) has been established.
INFO   | jvm 1    | 2020/03/21 14:28:55 |  WARN | Network connection between vm://remote#0 and tcp:///192.168.8.109:59180@61616 shutdown due to a remote error: java.io.EOFException
INFO   | jvm 1    | 2020/03/21 14:28:55 |  INFO | Connector vm://remote stopped
INFO   | jvm 1    | 2020/03/21 14:28:55 |  INFO | remote bridge to local stopped
INFO   | jvm 1    | 2020/03/21 14:29:33 |  INFO | App.Data Inactive for longer than 10000 ms - removing ...
INFO   | jvm 1    | 2020/03/21 14:30:18 |  INFO | Connector vm://remote started
INFO   | jvm 1    | 2020/03/21 14:30:18 |  INFO | Started responder end of duplex bridge Q-bridge@ID:LHC-59210-1584793818508-0:1
INFO   | jvm 1    | 2020/03/21 14:30:18 |  INFO | Network connection between vm://remote#4 and tcp:///192.168.8.109:59211@61616 (local) has been established.

BAD: multiple kahaDb persistence adapters on remote broker with perDestination="true"

INFO   | jvm 1    | 2020/03/21 14:39:46 |  INFO | ActiveMQ WebConsole available at http://0.0.0.0:8161/
INFO   | jvm 1    | 2020/03/21 14:39:46 |  INFO | ActiveMQ Jolokia REST API available at http://0.0.0.0:8161/api/jolokia/
INFO   | jvm 1    | 2020/03/21 14:40:12 |  INFO | Connector vm://remote started
INFO   | jvm 1    | 2020/03/21 14:40:12 |  INFO | Started responder end of duplex bridge Q-bridge@ID:LHC-59435-1584794412439-0:1
INFO   | jvm 1    | 2020/03/21 14:40:13 |  INFO | Network connection between vm://remote#0 and tcp:///192.168.8.109:59436@61616 (local) has been established.
INFO   | jvm 1    | 2020/03/21 14:40:21 |  WARN | Network connection between vm://remote#0 and tcp:///192.168.8.109:59436@61616 shutdown due to a remote error: java.io.EOFException
INFO   | jvm 1    | 2020/03/21 14:40:21 |  INFO | Connector vm://remote stopped
INFO   | jvm 1    | 2020/03/21 14:40:21 |  INFO | remote bridge to local stopped
INFO   | jvm 1    | 2020/03/21 14:41:14 |  INFO | App.Data Inactive for longer than 10000 ms - removing ...
INFO   | jvm 1    | 2020/03/21 14:41:14 |  INFO | Stopping async queue tasks
INFO   | jvm 1    | 2020/03/21 14:41:14 |  INFO | Stopping async topic tasks
INFO   | jvm 1    | 2020/03/21 14:41:14 |  INFO | Stopped KahaDB
INFO   | jvm 1    | 2020/03/21 14:41:32 |  INFO | Connector vm://remote started
INFO   | jvm 1    | 2020/03/21 14:41:32 | ERROR | Failed to create responder end of duplex network bridge Q-bridge@ID:LHC-59471-1584794492297-0:1
INFO   | jvm 1    | 2020/03/21 14:41:32 | java.lang.IllegalStateException: PageFile is not loaded
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.store.kahadb.disk.page.PageFile.assertLoaded(PageFile.java:906)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.store.kahadb.disk.page.PageFile.tx(PageFile.java:315)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:1211)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:148)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.getDestinations(MultiKahaDBPersistenceAdapter.java:267)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.region.RegionBroker.getDurableDestinations(RegionBroker.java:671)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.TransportConnection.processBrokerInfo(TransportConnection.java:1442)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.command.BrokerInfo.visit(BrokerInfo.java:128)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/21 14:41:32 | 	at java.lang.Thread.run(Thread.java:745)[:1.8.0_45]
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

Tim Bain
Can you please provide a listing (ls -l) of the KahaDB data directory?

Also, can you please configure mKahaDB to point to a different (empty)
directory rather than the same one that the normal KahaDB points to, and
try again? I've never used mKahaDB so I don't have direct experience, but I
wouldn't expect the broker to be happy if you switch between KahaDB and
mKahaDB using the same data directory (and the same files within it, more
importantly). Unless your process for switching the KahaDB type already
involved deleting the files in the data directory?

Tim

Tim

On Sat, Mar 21, 2020, 8:03 AM nexus <[hidden email]> wrote:

> Hello,
>
> Encountering following issue with network of brokers on Windows 10 / 64-bit
> / ActiveMQ v5.15.12
>
> Problem
>
>
> Cannot re-establish duplex network connection on remote broker with
> following error:
>
>
>
> Remote broker is configured to  delete inactive destinations
> <https://activemq.apache.org/delete-inactive-destinations.html>   and uses
> mKahaDB persistence adapters for different destinations ( Automatic Per
> Destination Persistence Adapter <https://activemq.apache.org/kahadb>  ).
>
> Same setup, but single kahaDB persistence adapter on remote broker is not
> causing the issue.
>
> See attached files for detailed configuration and logs:
>
> *  local-activemq.xml
> <http://activemq.2283324.n4.nabble.com/file/t379838/local-activemq.xml>
> *  good-remote-activemq.xml
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-activemq.xml>
>
> *  good-remote-wrapper.log
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/good-remote-wrapper.log>
>
> *  bad-remote-activemq.xml
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-activemq.xml>
>
> *  bad-remote-wrapper.log
> <http://activemq.2283324.n4.nabble.com/file/t379838/bad-remote-wrapper.log>
>
>
> Use case
>
>
> Simulate network connection loss and then re-astablish duplex communication
> after remote broker destinations were purged due to inactivity:
>
> 1. start remote broker
> 2. start local broker
> 3. destination queue created automatically on remote (active consumer from
> local broker is also shown correctly in web-console)
> 4. stop local broker
> 5. wait for a while until destination is deleted on remote due to
> inactivity
> 6. start local broker again
>
> Results
>
>
> GOOD: single kahaDB persistence adapter on remote broker
>
>
>
>
> BAD: multiple kahaDb persistence adapters on remote broker with
> perDestination="true"
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

nexus
This post was updated on .
> Can you please provide a listing (ls -l) of the KahaDB data directory?

See details below aligned with use case.

> Also, can you please configure mKahaDB to point to a different (empty) directory rather than the same one that the normal KahaDB points to, and try again? I've never used mKahaDB so I don't have direct experience, but I wouldn't expect the broker to be happy if you switch between KahaDB and mKahaDB using the same data directory (and the same files within it, more importantly). Unless your process for switching the KahaDB type already involved deleting the files in the data directory?

There was no switch from single to multi kahadb. Problem is reproduced with clean installation of apache-activemq-5.15.12-bin.zip on 2 PCs and using configuration for local and remote brokers I a have provided in the first post.

Some observations

The main difference I see in logs, that in bad situtation (mKahaDB + perDestination="true") following messages appear after inactive destinations deleted:

INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | Stopping async queue tasks
INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | Stopping async topic tasks
INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | Stopped KahaDB

Since this moment local broker cannot establish duplex connection any more, and it doesn't matter which destinations has been purged -- with the same name (App.Data) or any other. Also it doesn't matter whether local broker already had any successful communication with the remote. As soon as these messages appear, broker cannot "create responder end of duplex network bridge" because of "PageFile is not loaded".

When I try to do the same with single kahadb instance, these messages do not appear and no such problem.

Use case with logs and kahadb listing

Attached logs

1. Remote broker started

kahadb

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb
total 1
-rw-r--r-- 1 Nexus 197121 8 Mar 22 11:46 lock
drwxr-xr-x 1 Nexus 197121 0 Mar 22 11:46 txStore/

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb/txStore
total 32768
-rw-r--r-- 1 Nexus 197121 33554432 Mar 22 11:46 db-1.log

2. Local broker started

Duplex network connection established.
Statically included queue "App.Data" created on remote.
Active consumer is shown in web-console.

wrapper.log

INFO   | jvm 1    | 2020/03/22 11:48:36 |  INFO | Connector vm://remote started
INFO   | jvm 1    | 2020/03/22 11:48:36 |  INFO | Started responder end of duplex bridge Q-bridge@ID:LHC-53664-1584870516210-0:1
INFO   | jvm 1    | 2020/03/22 11:48:36 |  INFO | Network connection between vm://remote#0 and tcp:///192.168.8.109:53665@61616 (local) has been established.

kahadb

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb
total 1
-rw-r--r-- 1 Nexus 197121 8 Mar 22 11:46 lock
drwxr-xr-x 1 Nexus 197121 0 Mar 22 11:48 queue#3a#2f#2fApp.Data/
drwxr-xr-x 1 Nexus 197121 0 Mar 22 11:46 txStore/

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb/queue#3a#2f#2fApp.Data
total 32864
-rw-r--r-- 1 Nexus 197121    32768 Mar 22 11:49 db.data
-rw-r--r-- 1 Nexus 197121    32824 Mar 22 11:49 db.redo
-rw-r--r-- 1 Nexus 197121 33554432 Mar 22 11:48 db-1.log

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb/txStore
total 32768
-rw-r--r-- 1 Nexus 197121 33554432 Mar 22 11:46 db-1.log

3. Local broker stopped and remote deletes inactive destinations after a while

wrapper.log

INFO   | jvm 1    | 2020/03/22 11:50:27 |  WARN | Network connection between vm://remote#0 and tcp:///192.168.8.109:53665@61616 shutdown due to a remote error: java.io.EOFException
INFO   | jvm 1    | 2020/03/22 11:50:27 |  INFO | Connector vm://remote stopped
INFO   | jvm 1    | 2020/03/22 11:50:27 |  INFO | remote bridge to local stopped
INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | App.Data Inactive for longer than 10000 ms - removing ...
INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | Stopping async queue tasks
INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | Stopping async topic tasks
INFO   | jvm 1    | 2020/03/22 11:51:20 |  INFO | Stopped KahaDB

kahadb

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb
total 1
-rw-r--r-- 1 Nexus 197121 8 Mar 22 11:46 lock
drwxr-xr-x 1 Nexus 197121 0 Mar 22 11:46 txStore/

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb/txStore/
total 32768
-rw-r--r-- 1 Nexus 197121 33554432 Mar 22 11:46 db-1.log

4. Local broker started again

wrapper.log

INFO   | jvm 1    | 2020/03/22 11:53:46 |  INFO | Connector vm://remote started
INFO   | jvm 1    | 2020/03/22 11:53:46 | ERROR | Failed to create responder end of duplex network bridge Q-bridge@ID:LHC-53699-1584870825922-0:1
INFO   | jvm 1    | 2020/03/22 11:53:46 | java.lang.IllegalStateException: PageFile is not loaded
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.store.kahadb.disk.page.PageFile.assertLoaded(PageFile.java:906)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.store.kahadb.disk.page.PageFile.tx(PageFile.java:315)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:1211)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:148)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.getDestinations(MultiKahaDBPersistenceAdapter.java:267)[activemq-kahadb-store-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.region.RegionBroker.getDurableDestinations(RegionBroker.java:671)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.BrokerFilter.getDurableDestinations(BrokerFilter.java:259)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.TransportConnection.processBrokerInfo(TransportConnection.java:1442)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.command.BrokerInfo.visit(BrokerInfo.java:128)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)[activemq-broker-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.12.jar:5.15.12]
INFO   | jvm 1    | 2020/03/22 11:53:46 | 	at java.lang.Thread.run(Thread.java:745)[:1.8.0_45]

kahadb

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb
total 1
-rw-r--r-- 1 Nexus 197121 8 Mar 22 11:46 lock
drwxr-xr-x 1 Nexus 197121 0 Mar 22 11:46 txStore/

Nexus@LHC MINGW64 /d/etc/activemq-5.15.12/data
$ ls -l kahadb/txStore/
total 32768
-rw-r--r-- 1 Nexus 197121 33554432 Mar 22 11:46 db-1.log
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

Tim Bain
This definitely sounds like a bug in the mKahaDB code, so can you please
submit a bug report for it in JIRA?

I'd be curious to know whether the problem occurs if you cycle the remote
broker between steps 5 and 6. If not, it's likely something in-memory that
represents an inconsistency between the filesystem and what's in memory for
the mKahaDB database within the remote broker.

Tim

On Sun, Mar 22, 2020, 4:03 AM nexus <[hidden email]> wrote:

> > Can you please provide a listing (ls -l) of the KahaDB data directory?
> See details below aligned with use case.
> > Also, can you please configure mKahaDB to point to a different (empty)
> > directory rather than the same one that the normal KahaDB points to, and
> > try again? I've never used mKahaDB so I don't have direct experience, but
> > I wouldn't expect the broker to be happy if you switch between KahaDB and
> > mKahaDB using the same data directory (and the same files within it, more
> > importantly). Unless your process for switching the KahaDB type already
> > involved deleting the files in the data directory?
> There was no switch from single to multi kahadb. Problem is reproduced with
> clean installation of  apache-activemq-5.15.12-bin.zip
> <
> http://www.apache.org/dyn/closer.cgi?filename=/activemq/5.15.12/apache-activemq-5.15.12-bin.zip&action=download>
>
> on 2 PCs and using configuration for local and remote brokers I a have
> provided in the first post.
> Some observations
> The main difference I see in logs, that in bad situtation (mKahaDB +
> perDestination="true") following messages appear after inactive
> destinations
> deleted:
> Since this moment local broker cannot establish duplex connection any more,
> and it doesn't matter which destinations has been purged -- with the same
> name (App.Data) or any other. Also it doesn't matter whether local broker
> already had any successful communication with the remote. As soon as these
> messages appear, broker cannot "create responder end of duplex network
> bridge" because of "PageFile is not loaded".
> When I try to do the same with single kahadb instance, these messages do
> not
> appear and no such problem.
> Use case with logs and kahadb listing
> Attached logs
> activemq.log
> <http://activemq.2283324.n4.nabble.com/file/t379838/activemq.log>
> wrapper.log <
> http://activemq.2283324.n4.nabble.com/file/t379838/wrapper.log>
> 1. Remote broker started
> kahadb
> 2. Local broker started
> Duplex network connection established.
> Statically included queue "App.Data" created on remote.
> Active consumer is shown in web-console.
> wrapper.log
> kahadb
> 3. Local broker stopped and remote deletes inactive destinations after a
> while
> wrapper.log
> kahadb
> 4. Local broker started again
> wrapper.log
> kahadb
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

nexus
> can you please submit a bug report for it in JIRA?

Done:
https://issues.apache.org/jira/projects/AMQ/issues/AMQ-7453

> whether the problem occurs if you cycle the remote broker between steps 5
> and 6

Steps 5. - 6. can be repeated multiple times -- problem persists.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

Tim Bain
What I meant was, if you cycle the remote broker while the local one is
down, do you get the same behavior? I'm trying to characterize whether the
problem is purely something in-memory on the remote broker, or whether the
problem is also in the content written to disk (and therefore the problem
persists even if you cycle the remote broker).

Tim

On Thu, Mar 26, 2020, 1:38 AM nexus <[hidden email]> wrote:

> > can you please submit a bug report for it in JIRA?
>
> Done:
> https://issues.apache.org/jira/projects/AMQ/issues/AMQ-7453
>
> > whether the problem occurs if you cycle the remote broker between steps 5
> > and 6
>
> Steps 5. - 6. can be repeated multiple times -- problem persists.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

nexus
> if you cycle the remote broker while the local one is down, do you get the
same behavior?

I've performed the test, there's no error after clean restart of the remote
broker, but the problem repeats whenever inactive destinations are purged
once again. So, I suppose, your assumption is right "it is purely something
in-memory on the remote broker".

bad-activemq-restart.log
<http://activemq.2283324.n4.nabble.com/file/t379838/bad-activemq-restart.log>  



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

Tim Bain
Thanks for testing that; it will help make sure that whoever works on a fix
focuses quickly on the right part of the code. Would you be able to submit
a bug in JIRA to capture the problem as you observed it? If you're able to
attach any config files that would make it easier to quickly reproduce the
problem, that would be helpful as well.

Thanks for working through this preliminary testing and helping better
characterize the problem.

Tim

On Fri, Mar 27, 2020, 12:55 AM nexus <[hidden email]> wrote:

> > if you cycle the remote broker while the local one is down, do you get
> the
> same behavior?
>
> I've performed the test, there's no error after clean restart of the remote
> broker, but the problem repeats whenever inactive destinations are purged
> once again. So, I suppose, your assumption is right "it is purely something
> in-memory on the remote broker".
>
> bad-activemq-restart.log
> <
> http://activemq.2283324.n4.nabble.com/file/t379838/bad-activemq-restart.log>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

nexus
I have already opened an issue AMQ-7453 with detailed description, logs and
examples, please, take a look:
https://issues.apache.org/jira/browse/AMQ-7453



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Duplex networkConnector error with mKahaDB after inactive destinations deleted

Tim Bain
I had forgotten you'd already submitted one. Thank you for doing that.

Tim

On Sat, Mar 28, 2020, 12:13 AM nexus <[hidden email]> wrote:

> I have already opened an issue AMQ-7453 with detailed description, logs and
> examples, please, take a look:
> https://issues.apache.org/jira/browse/AMQ-7453
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>