offline Durable Topic Subscribers issue in activemq 5.15.8

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Tim Bain
Also, to be clear, I don't believe that KahaDB has anything to do with this
problem. The issue appears to be that the client ID assigned to the
subscription on reconnection doesn't match the previous connection's value,
which is independent of the persistence store type.

Tim

On Thu, Nov 21, 2019, 6:57 AM Tim Bain <[hidden email]> wrote:

> Could you please share that configuration?
>
> I didn't understand from your earlier message that you were creating your
> subscriptions using Camel, which adds another technology, and another
> potential source of bugs, into the mix.
>
> Are both of your consumers staying up while the broker is restarted? And
> are they running in the same JVM?
>
> Tim
>
> On Thu, Nov 21, 2019, 6:26 AM Ashish Vaishno <[hidden email]>
> wrote:
>
>> Hello Tim,
>>
>> I am setting the clientid in the file integration.camel.routes.xml, so
>> that
>> on each restart my durable susbcriber is always recognized and unique.
>>
>> Regards
>> Ashish
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
This post was updated on .
In reply to this post by Tim Bain
Hello Tim,

In reference to https://activemq.apache.org/sample-camel-routes 
Please find attached the config  camel.xml
<http://activemq.2283324.n4.nabble.com/file/t379708/camel.xml

my brokers are running in different jvm. i have a master slave setup in the
backend, so at one time only one if active.


Regarding your latest post :
When i enable debug logs, then i can clearly see that broker recovers a random susbcription(which is no where in the config files of activemq). So I always end you in a place where 2 client id (same name) are mapped to different topics/susbcription name. :(

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Tim Bain
That file doesn't use "Hello" or "World" anywhere in it, and appears to
have only a single route within it. Can you help me to understand how it
lines up with the question you originally asked, where you have two
separate durable subscriptions, for Hello and World, where one is working
and the other is not?

Thanks,
Tim

On Thu, Nov 21, 2019 at 8:10 AM Ashish Vaishno <[hidden email]>
wrote:

> Hello Tim,
>
> In reference to https://activemq.apache.org/sample-camel-routes
> Please find attached the config  camel.xml
> <http://activemq.2283324.n4.nabble.com/file/t379708/camel.xml>
>
> my brokers are running in different jvm. i have a master slave setup in the
> backend, so at one time only one if active.
>
> Regards
> Ashish
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
This post was updated on .
Hello Tim,

Sorry for the confusion. In my setup there are a lot of brokers which have a
similar config. WRT to hello/world example that i refered, the config would
look like
backend-broker-1 :
<route id="hello_mirror-standard-frontend-standard-01">
      <from
uri="frontend-standard-01:topic:Hello.v1?clientId=backend-broker-1-hello_mirror&amp;durableSubscriptionName=hello"/>
      <to uri="backend-broker-name-1:queue:hello.mirror"/>
    </route>

backend-broker-2:
<route id="hello_mirror-standard-frontend-standard-01">
      <from
uri="frontend-standard-01:topic:World.v1?clientId=backend-broker-2-world_mirror&amp;durableSubscriptionName=world"/>
      <to uri="backend-broker-name-1:queue:world.mirror"/>
    </route>

backend-broker-3:
<route id="hello_mirror-standard-frontend-standard-01">
      <from
uri="frontend-standard-01:topic:HowAreYou.v1?clientId=backend-broker-3-howareyou_mirror&amp;durableSubscriptionName=howareyou"/>
      <to uri="backend-broker-name-1:queue:howareyou.mirror"/>
    </route>
...

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

jbonofre
Hi

What kind of component is frontend-standard ? Is it a extend you did ?

Regards
JB

On 25/11/2019 09:57, Ashish Vaishno wrote:

> Hello Tim,
>
> Sorry for the confusion. In my setup there are a lot of brokers which have a
> similar config. WRT to hello/world example that i refered, the config would
> look like
> broker-1 :
> <route id="hello_mirror-standard-frontend-standard-01">
>       <from
> uri="frontend-standard-01:topic:Hello.v1?clientId=backend-broker-name-1-hello_mirror&amp;durableSubscriptionName=hello"/>
>       <to uri="backend-broker-name-1:queue:hello.mirror"/>
>     </route>
>
> broker-2:
> <route id="hello_mirror-standard-frontend-standard-01">
>       <from
> uri="frontend-standard-01:topic:World.v1?clientId=backend-broker-name-1-world_mirror&amp;durableSubscriptionName=world"/>
>       <to uri="backend-broker-name-1:queue:world.mirror"/>
>     </route>
>
> broker-3:
> <route id="hello_mirror-standard-frontend-standard-01">
>       <from
> uri="frontend-standard-01:topic:HowAreYou.v1?clientId=backend-broker-name-1-howareyou_mirror&amp;durableSubscriptionName=howareyou"/>
>       <to uri="backend-broker-name-1:queue:howareyou.mirror"/>
>     </route>
> ...
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

jbonofre
In reply to this post by Ashish Vaishno
By the way, in activemq.xml, in the broker settings, do you have
offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule
defined or do you use the default ?

Regards
JB

On 25/11/2019 09:57, Ashish Vaishno wrote:

> Hello Tim,
>
> Sorry for the confusion. In my setup there are a lot of brokers which have a
> similar config. WRT to hello/world example that i refered, the config would
> look like
> broker-1 :
> <route id="hello_mirror-standard-frontend-standard-01">
>       <from
> uri="frontend-standard-01:topic:Hello.v1?clientId=backend-broker-name-1-hello_mirror&amp;durableSubscriptionName=hello"/>
>       <to uri="backend-broker-name-1:queue:hello.mirror"/>
>     </route>
>
> broker-2:
> <route id="hello_mirror-standard-frontend-standard-01">
>       <from
> uri="frontend-standard-01:topic:World.v1?clientId=backend-broker-name-1-world_mirror&amp;durableSubscriptionName=world"/>
>       <to uri="backend-broker-name-1:queue:world.mirror"/>
>     </route>
>
> broker-3:
> <route id="hello_mirror-standard-frontend-standard-01">
>       <from
> uri="frontend-standard-01:topic:HowAreYou.v1?clientId=backend-broker-name-1-howareyou_mirror&amp;durableSubscriptionName=howareyou"/>
>       <to uri="backend-broker-name-1:queue:howareyou.mirror"/>
>     </route>
> ...
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
Hello JB,

I have not set offlineDurableSubscriberTaskSchedule or
offlineDurableSubscriberTimeout explicitly in my activemq.xml file.

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

jbonofre
Hi,

honestly, I don't understand the problem, and I'm not able to reproduce
with simple Camel routes (running in Karaf).

Regards
JB

On 25/11/2019 14:05, Ashish Vaishno wrote:

> Hello JB,
>
> I have not set offlineDurableSubscriberTaskSchedule or
> offlineDurableSubscriberTimeout explicitly in my activemq.xml file.
>
> Regards
> Ashish
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Tim Bain
OK, so I want to make sure I understand this correctly.

You have two brokers (or is that two failover pairs?): frontend-standard-01
and backend-broker-name-1. The brokers are not configured in a network of
brokers, though you're using your Camel routes to move messages between
them. You have some number of Camel routes running in a separate JVM (i.e.
not within any ActiveMQ broker). Does each Camel route run in the same JVM,
or are they each in their own JVM? When the broker is restarted or a
failover occurs, the JVM containing the Camel routes remains up, and
attempts to reconnect to the broker immediately and automatically, and
somewhere in this reconnection logic the subscription name is being
constructed incorrectly. Is that all accurate?

JB, do you know whether the subscription name gets formed by the broker or
by the client? If the latter, it's possible that this is actually a Camel
bug, but I don't know enough about where that's done to know whether this
is on the ActiveMQ side or the Camel side.

Tim

On Mon, Nov 25, 2019 at 6:24 AM Jean-Baptiste Onofré <[hidden email]>
wrote:

> Hi,
>
> honestly, I don't understand the problem, and I'm not able to reproduce
> with simple Camel routes (running in Karaf).
>
> Regards
> JB
>
> On 25/11/2019 14:05, Ashish Vaishno wrote:
> > Hello JB,
> >
> > I have not set offlineDurableSubscriberTaskSchedule or
> > offlineDurableSubscriberTimeout explicitly in my activemq.xml file.
> >
> > Regards
> > Ashish
> >
> >
> >
> > --
> > Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

jbonofre
Hi Tim,

The subscription is firstly client provided (and fallback to client id
if not specified).

Regards
JB

On 27/11/2019 07:59, Tim Bain wrote:

> OK, so I want to make sure I understand this correctly.
>
> You have two brokers (or is that two failover pairs?): frontend-standard-01
> and backend-broker-name-1. The brokers are not configured in a network of
> brokers, though you're using your Camel routes to move messages between
> them. You have some number of Camel routes running in a separate JVM (i.e.
> not within any ActiveMQ broker). Does each Camel route run in the same JVM,
> or are they each in their own JVM? When the broker is restarted or a
> failover occurs, the JVM containing the Camel routes remains up, and
> attempts to reconnect to the broker immediately and automatically, and
> somewhere in this reconnection logic the subscription name is being
> constructed incorrectly. Is that all accurate?
>
> JB, do you know whether the subscription name gets formed by the broker or
> by the client? If the latter, it's possible that this is actually a Camel
> bug, but I don't know enough about where that's done to know whether this
> is on the ActiveMQ side or the Camel side.
>
> Tim
>
> On Mon, Nov 25, 2019 at 6:24 AM Jean-Baptiste Onofré <[hidden email]>
> wrote:
>
>> Hi,
>>
>> honestly, I don't understand the problem, and I'm not able to reproduce
>> with simple Camel routes (running in Karaf).
>>
>> Regards
>> JB
>>
>> On 25/11/2019 14:05, Ashish Vaishno wrote:
>>> Hello JB,
>>>
>>> I have not set offlineDurableSubscriberTaskSchedule or
>>> offlineDurableSubscriberTimeout explicitly in my activemq.xml file.
>>>
>>> Regards
>>> Ashish
>>>
>>>
>>>
>>> --
>>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
In reply to this post by Tim Bain
Hello Tim,

Your assumption is correct. And yes,*" reconnection logic the subscription
name is being
constructed incorrectly "*

And yes : All my camel routes associated to a single broker run in a single
jvm.

Regards
Ashish






--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
Hello Tim/JB,

Also one more information. I tried to reproduce the issue with 1 frontend
(with 1 topic) and 1 backend (with 1 camel route) and I was not able to
reproduce it.

I saw the issue after 7/8 backend brokers with a lot of routes.

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Tim Bain
What does "7/8 backend brokers with a lot of routes" mean? My description
of your setup specified either a broker or a master-slave pair of brokers
for the backend, and you agreed that my description was correct. How did we
get from there to "7/8" backend brokers? Also, can you re-confirm that the
Camel routes are running outside of the JVM of any of the brokers, as I
described in my understanding of your setup? You confirmed that they all
run in the same JVM, but can you confirm that it's a non-broker, external
JVM?

If it wasn't for what you wrote about "7/8 backend brokers" (which sounds
like a NoB, even though your confirmation of my description from a few days
ago should have ruled that out), my assumption would be that there's a bug
in Camel where they form the subscription names in a non-deterministic way
when there are more than one in the same JVM, but I want to better
understand the "7/8 backend brokers" bit and to rule out the possibility of
a NoB before we go down that path.

Tim


On Fri, Nov 29, 2019 at 3:29 AM Ashish Vaishno <[hidden email]>
wrote:

> Hello Tim/JB,
>
> Also one more information. I tried to reproduce the issue with 1 frontend
> (with 1 topic) and 1 backend (with 1 camel route) and I was not able to
> reproduce it.
>
> I saw the issue after 7/8 backend brokers with a lot of routes.
>
> Regards
> Ashish
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
Hello Tim,

What I meant wrt to 7/8 broker is as follows:

1. I have one frontend broker which only have topics. These topics are then
subscribed by the backend broker
2. Each backend broker can have many routes, which means that a single
backend broker(with n number of queues) can subscribe to a lot of topics
which has its own subscription id.
3. I was not able to see the issue with one backend broker. So I started to
create more backend brokers. and when it reached 7 or 8 backend brokers, I
was able to see the issue.
4. So *7/8* only means after starting 7 to 8 backend broker, I was able to
see the issue.
5. So I just wanted to inform you that if you have one frontend broker and
one backend broker, then you might now be able to reproduce the issue

I am attaching a image that might be able to explain my setup
<http://activemq.2283324.n4.nabble.com/file/t379708/Capture1.png>

All my camel routes associated to a single broker run in a single jvm.

We do not have any non-brokers/applications which subscribe to the topics.
All the subscriptions are only made by the backend broker queues.

Regards
Ashish  



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: offline Durable Topic Subscribers issue in activemq 5.15.8

Ashish Vaishno
Hello Tim,

Did you see my comments?

Regards
Ashish



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
12