offline Durable Topic Subscribers issue in activemq 5.15.8

[Ashish Vaishno]

Hello,

We are using activemq version : 5.15.8

We are creating topics via config and also persisting the data via config :
    <persistenceAdapter>
      <mKahaDB directory="${activemq.data}/mkahadb">
        <filteredPersistenceAdapters>
          <filteredKahaDB perDestination="true">
            <persistenceAdapter>

              <kahaDB ignoreMissingJournalfiles="false"
checkForCorruptJournalFiles="true" checksumJournalFiles="true"/>
            </persistenceAdapter>
          </filteredKahaDB>
        </filteredPersistenceAdapters>
      </mKahaDB>
    </persistenceAdapter>

While doing so, we have noticed the below
1. Creation of topics via config is fast
2. But this leads to creation of "Offline Durable Topic Subscribers" with a
wrong state.
3. From my understanding 1 client id is mapped to one subscription id which
is mapped to one topic. But this mapping always ends up in a different state
after a restart of the broker
Example :
<http://activemq.2283324.n4.nabble.com/file/t379708/Capture.png>
From the above image you will see that after the restart of the broker,
correct client id goes in offline state and a wrong client id gets created
as active subscriber.

This causes incoming event to pile up, even though the config is correct.
The broker treats the active subscriber as non-active.

Has anyone faced the same issue ?

Regards
Ashish




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[jbonofre]

Hi Ashish,

I didn't notice such behavior but I don't use lot of durable subscripters.

Let me try to reproduce on a simple test case.

Regards
JB

On 08/11/2019 14:47, Ashish Vaishno wrote:

> Hello,
>
> We are using activemq version : 5.15.8
>
> We are creating topics via config and also persisting the data via config :
>     <persistenceAdapter>
>       <mKahaDB directory="${activemq.data}/mkahadb">
>         <filteredPersistenceAdapters>
>           <filteredKahaDB perDestination="true">
>             <persistenceAdapter>
>
>               <kahaDB ignoreMissingJournalfiles="false"
> checkForCorruptJournalFiles="true" checksumJournalFiles="true"/>
>             </persistenceAdapter>
>           </filteredKahaDB>
>         </filteredPersistenceAdapters>
>       </mKahaDB>
>     </persistenceAdapter>
>
> While doing so, we have noticed the below
> 1. Creation of topics via config is fast
> 2. But this leads to creation of "Offline Durable Topic Subscribers" with a
> wrong state.
> 3. From my understanding 1 client id is mapped to one subscription id which
> is mapped to one topic. But this mapping always ends up in a different state
> after a restart of the broker
> Example :
> <http://activemq.2283324.n4.nabble.com/file/t379708/Capture.png>
> From the above image you will see that after the restart of the broker,
> correct client id goes in offline state and a wrong client id gets created
> as active subscriber.
>
> This causes incoming event to pile up, even though the config is correct.
> The broker treats the active subscriber as non-active.
>
> Has anyone faced the same issue ?
>
> Regards
> Ashish
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

[Ashish Vaishno]

Hello Jean,

I also noticed that even a simple restart of the broker, causes an
inconsistent stage of offline durable subscriber.

Did you managed to find anything?

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[jbonofre]

Thanks for the update, let me try the restart (normal use case works
fine in my tests).

Regards
JB

On 13/11/2019 09:47, Ashish Vaishno wrote:

> Hello Jean,
>
> I also noticed that even a simple restart of the broker, causes an
> inconsistent stage of offline durable subscriber.
>
> Did you managed to find anything?
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

[Ashish Vaishno]

Hello JB,

Were you able to reproduce the issue, while creating topic and its persistence via config?

I also wiped out my persistence today. Started the broker and then restarted the broker.
I again ended up in the same issue. Below are the actual debug logs wrt to the diagram attached in the original ticket

2019-11-15 13:55:39,664 | DEBUG | Restoring durable subscription: SubscriptionInfo {subscribedDestination = topic://Hello.V1, destination = topic://Hello.V1, clientId = broker-name-2-world_world, subscriptionName = hello, selector = null, noLocal = false} | org.apache.activemq.broker.region.TopicRegion | main
2019-11-15 13:55:39,668 | DEBUG | Restoring durable subscription: SubscriptionInfo {subscribedDestination = topic://World.V1, destination = topic://World.V1, clientId = broker-name-2-world_world, subscriptionName = world, selector = null, noLocal = false} | org.apache.activemq.broker.region.TopicRegion | main
2019-11-15 13:55:40,194 | DEBUG | Setting up new connection id: ID:MY_SERVER-40009-1573807342637-4:169, address: tcp://MY_IP:59706, info: ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:MY_SERVER-40009-1573807342637-4:169, clientId = broker-name-2-world_world, clientIp = null, userName = MY_USERNAME, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false, failoverReconnect = false} | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///MY_IP:PORT@61616
2019-11-15 13:55:40,261 | DEBUG | Deactivating keepActive=true, DurableTopicSubscription-broker-name-2-world:world, id=ID:MY_SERVER-40009-1573807342637-4:169:1:1, active=false, destinations=1, total=0, pending=0, dispatched=0, inflight=0, prefetchExtension=0 | org.apache.activemq.broker.region.DurableTopicSubscription | ActiveMQ Transport: tcp:///MY_IP:PORT@61616
2019-11-15 13:55:40,261 | DEBUG | Activating DurableTopicSubscription-broker-name-2-world:world, id=ID:MY_SERVER-40009-1573807342637-4:169:1:1, active=false, destinations=1, total=0, pending=0, dispatched=0, inflight=0, prefetchExtension=0 | org.apache.activemq.broker.region.DurableTopicSubscription | ActiveMQ Transport: tcp:///MY_IP:PORT@61616
2019-11-15 13:55:40,262 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=MY_FRONTEND,destinationType=Topic,destinationName=World.V1,endpoint=Consumer,clientId=broker-name-2-world,consumerId=Durable(broker-name-2-world_world) | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport: tcp:///MY_IP:PORT@61616


Summary :
It restored 2 durable subscriber even though there is just one for that topic.
Sets up connection, deactivate correct subscriber,activate correct subscriber

No more information for the wrongly mapped offline subscriber

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[jbonofre]

Not yet (I was busy with on other Apache projects). I'm working on
ActiveMQ during the week end, I will try to reproduce.

I keep you posted.

Regards
JB

On 15/11/2019 11:34, Ashish Vaishno wrote:

> Hello JB,
>
> Were you able to reproduce the issue?
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

[Ashish Vaishno]

Hello JB,

Just a info, i am creating topics from config as well.

Today, I tried the below
Replaced
<filteredKahaDB perDestination="true">
by
<filteredKahaDB topic="Hello.V1">
.....
....
....<filteredKahaDB topic="World.V1">
in my mkahadb config. I will still not able to solve the issue

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[jbonofre]

Hi,

I guess it works using KahaDB (without filtered) ?

Regards
JB

On 18/11/2019 16:08, Ashish Vaishno wrote:

> Hello JB,
>
> Just a info, i am creating topics from config as well.
>
> Today, I tried the below
> Replaced
> <filteredKahaDB perDestination="true">
> by
> <filteredKahaDB topic="Hello.V1">
> .....
> ....
> ....<filteredKahaDB topic="World.V1">
> in my mkahadb config. I will still not able to solve the issue
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

[Ashish Vaishno]

Hello JB,

Can you send me the config that you are using for persistence ? Have you
enabled any parameters ?

All the below situations lead me to an inconsistent state :
1. while using <filteredKahaDB perDestination="true"> with mkahadb
2. while using just kahadb
   <persistenceAdapter>
     <kahaDB directory="activemq-data" journalMaxFileLength="32mb"/>
   </persistenceAdapter>
3. while using <filteredKahaDB topic=">"> with mkahadb

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[jbonofre]

Hi

nothing special, a regular kahadb instance (with some tuning but nothing
major).

Regards
JB

On 19/11/2019 12:47, Ashish Vaishno wrote:

> Hello JB,
>
> Can you send me the config that you are using for persistence ? Have you
> enabled any parameters ?
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com

[Ashish Vaishno]

Hello JB,

Do i need to create a Jira issue for the same or the issue created here is
more than enough?

Once you have a working config where you dont face the issue, please let me
know

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

[Tim Bain]

Ashish,

If you convert from mKahaDB to the standard KahaDB configuration (in a
dev/test environment), does the problem disappear?

If so, please write a bug in JIRA for the problem. (this mailing list
thread is not sufficient to get a fix implemented). If not, we'll have to
keep trying to isolate what's different between your setup and JB's, and
we'll ask you to write the JIRA issue once we've zeroed in on it.

Thanks,
Tim

On Tue, Nov 19, 2019, 8:45 AM Ashish Vaishno <[hidden email]> wrote:

> Hello JB,
>
> Do i need to create a Jira issue for the same or the issue created here is
> more than enough?
>
> Once you have a working config where you dont face the issue, please let me
> know
>
> Regards
> Ashish
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>