Load issue with ActiveMQ

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Load issue with ActiveMQ

Vikas Agarwal
Hi,

We are using ActiveMQ for 3+ years, however, we didn't test it under heavy
load. Recently, we started using ActiveMQ in another project where
sometimes load increases exponentially as we are listening to twitter
stream which can pile up a lot of messages depending of occurrence of some
global event like iPhone 6 launch. Now, what we are seeing in ActiveMQ
admin UI is that one of the queues gets piled up with thousands of message
ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ crashes
in between of my sleeping hours. :( And the strange thing is that when try
to view the log file for the reason of the crash, I am not able to find the
activemq.log file at all. We earlier faced ActiveMQ crash issues, but that
was due to producer flow control and we disabled that to fix the issue and
main point is that in those cases we never had the case where the log file
was missing. I have to restart the process to fix the issue and that too,
twice because on first restart, it shows error about missing or corrupt
data log.

So, please suggest me where to look for such issue or what should be my
next move for debugging. I am stuck with missing log file.

For information here are the memory settings:

<systemUsage>
    <systemUsage>
        <memoryUsage>
            <memoryUsage limit="*256 mb*"/>
        </memoryUsage>
        <storeUsage>
            <storeUsage limit="*100 gb*"/>
        </storeUsage>
        <tempUsage>
            <tempUsage limit="*50 gb*"/>
        </tempUsage>
    </systemUsage>
</systemUsage>

And


....

....
<policyEntry memoryLimit="*10mb*" producerFlowControl="true" queue="&gt;">

....
</policyEntry>

ActiveMQ process is started with* -Xms2G and -Xmx2G* which earlier used to
be the default  *-Xms1G and -Xmx1G*

--
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

James Green-3
On 3 November 2014 09:42, Vikas Agarwal <[hidden email]> wrote:

> Hi,
>
> We are using ActiveMQ for 3+ years, however, we didn't test it under heavy
> load. Recently, we started using ActiveMQ in another project where
> sometimes load increases exponentially as we are listening to twitter
> stream which can pile up a lot of messages depending of occurrence of some
> global event like iPhone 6 launch. Now, what we are seeing in ActiveMQ
> admin UI is that one of the queues gets piled up with thousands of message
> ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ crashes
>

Do you mean:

1. The queues have 50-100,000 messages each, or
2. The queues have thousands of messages each of which may be 50-100k in
size?


> in between of my sleeping hours. :( And the strange thing is that when try
> to view the log file for the reason of the crash, I am not able to find the
> activemq.log file at all. We earlier faced ActiveMQ crash issues, but that
>

Ensure you have INFO logging switched on and that AMQ writes to this log on
start-up. Also, that log could be rotated - ensure the process has
permission to write to it.


> was due to producer flow control and we disabled that to fix the issue and
> main point is that in those cases we never had the case where the log file
> was missing. I have to restart the process to fix the issue and that too,
> twice because on first restart, it shows error about missing or corrupt
> data log.
>
> So, please suggest me where to look for such issue or what should be my
> next move for debugging. I am stuck with missing log file.
>
> For information here are the memory settings:
>
> <systemUsage>
>     <systemUsage>
>         <memoryUsage>
>             <memoryUsage limit="*256 mb*"/>
>         </memoryUsage>
>         <storeUsage>
>             <storeUsage limit="*100 gb*"/>
>         </storeUsage>
>         <tempUsage>
>             <tempUsage limit="*50 gb*"/>
>         </tempUsage>
>     </systemUsage>
> </systemUsage>
>

You might want to start here and work out the model of usage you're seeing:
http://activemq.apache.org/javalangoutofmemory.html

 For instance: thousands of messages, few clients; or thousands of messages
and lots of clients.
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Vikas Agarwal
I mean 50,000-100,000 messages are there in queue after which it crashes
and this range is not definitive, it some crashes below a pile of 50,000
messages too. Actually, I am not able to predict when exactly it is
crashing, if its is crashing due to CPU usage or memory usage or IO.

Logs are behaving fine before the crash, so I believe everything is fine
with permissions and log level. Log rotation would not remove the
activemq.log completely. Further, after crash, activemq process is still
alive (checked using ps command), however, Admin UI (port 8161) is not
available after crash.

On Mon, Nov 3, 2014 at 7:03 PM, James Green <[hidden email]>
wrote:

> On 3 November 2014 09:42, Vikas Agarwal <[hidden email]> wrote:
>
> > Hi,
> >
> > We are using ActiveMQ for 3+ years, however, we didn't test it under
> heavy
> > load. Recently, we started using ActiveMQ in another project where
> > sometimes load increases exponentially as we are listening to twitter
> > stream which can pile up a lot of messages depending of occurrence of
> some
> > global event like iPhone 6 launch. Now, what we are seeing in ActiveMQ
> > admin UI is that one of the queues gets piled up with thousands of
> message
> > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ crashes
> >
>
> Do you mean:
>
> 1. The queues have 50-100,000 messages each, or
> 2. The queues have thousands of messages each of which may be 50-100k in
> size?
>
>
> > in between of my sleeping hours. :( And the strange thing is that when
> try
> > to view the log file for the reason of the crash, I am not able to find
> the
> > activemq.log file at all. We earlier faced ActiveMQ crash issues, but
> that
> >
>
> Ensure you have INFO logging switched on and that AMQ writes to this log on
> start-up. Also, that log could be rotated - ensure the process has
> permission to write to it.
>
>
> > was due to producer flow control and we disabled that to fix the issue
> and
> > main point is that in those cases we never had the case where the log
> file
> > was missing. I have to restart the process to fix the issue and that too,
> > twice because on first restart, it shows error about missing or corrupt
> > data log.
> >
> > So, please suggest me where to look for such issue or what should be my
> > next move for debugging. I am stuck with missing log file.
> >
> > For information here are the memory settings:
> >
> > <systemUsage>
> >     <systemUsage>
> >         <memoryUsage>
> >             <memoryUsage limit="*256 mb*"/>
> >         </memoryUsage>
> >         <storeUsage>
> >             <storeUsage limit="*100 gb*"/>
> >         </storeUsage>
> >         <tempUsage>
> >             <tempUsage limit="*50 gb*"/>
> >         </tempUsage>
> >     </systemUsage>
> > </systemUsage>
> >
>
> You might want to start here and work out the model of usage you're seeing:
> http://activemq.apache.org/javalangoutofmemory.html
>
>  For instance: thousands of messages, few clients; or thousands of messages
> and lots of clients.
>



--
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Tim Bain
Most developers I know would define "crash" to be "the process
terminated".  Since that doesn't seem to be the case here (and since you
applied the term to an instance of Producer Flow Control in a previous
situation, which is most definitely not an ActiveMQ crash nor an ActiveMQ
error of any sort), I assume that's not your definition of the term.  Can
you describe the symptoms you're seeing, without using the word "crash", to
get everyone on the same page about what your problem actually is?

On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <[hidden email]>
wrote:

> I mean 50,000-100,000 messages are there in queue after which it crashes
> and this range is not definitive, it some crashes below a pile of 50,000
> messages too. Actually, I am not able to predict when exactly it is
> crashing, if its is crashing due to CPU usage or memory usage or IO.
>
> Logs are behaving fine before the crash, so I believe everything is fine
> with permissions and log level. Log rotation would not remove the
> activemq.log completely. Further, after crash, activemq process is still
> alive (checked using ps command), however, Admin UI (port 8161) is not
> available after crash.
>
> On Mon, Nov 3, 2014 at 7:03 PM, James Green <[hidden email]>
> wrote:
>
> > On 3 November 2014 09:42, Vikas Agarwal <[hidden email]> wrote:
> >
> > > Hi,
> > >
> > > We are using ActiveMQ for 3+ years, however, we didn't test it under
> > heavy
> > > load. Recently, we started using ActiveMQ in another project where
> > > sometimes load increases exponentially as we are listening to twitter
> > > stream which can pile up a lot of messages depending of occurrence of
> > some
> > > global event like iPhone 6 launch. Now, what we are seeing in ActiveMQ
> > > admin UI is that one of the queues gets piled up with thousands of
> > message
> > > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ
> crashes
> > >
> >
> > Do you mean:
> >
> > 1. The queues have 50-100,000 messages each, or
> > 2. The queues have thousands of messages each of which may be 50-100k in
> > size?
> >
> >
> > > in between of my sleeping hours. :( And the strange thing is that when
> > try
> > > to view the log file for the reason of the crash, I am not able to find
> > the
> > > activemq.log file at all. We earlier faced ActiveMQ crash issues, but
> > that
> > >
> >
> > Ensure you have INFO logging switched on and that AMQ writes to this log
> on
> > start-up. Also, that log could be rotated - ensure the process has
> > permission to write to it.
> >
> >
> > > was due to producer flow control and we disabled that to fix the issue
> > and
> > > main point is that in those cases we never had the case where the log
> > file
> > > was missing. I have to restart the process to fix the issue and that
> too,
> > > twice because on first restart, it shows error about missing or corrupt
> > > data log.
> > >
> > > So, please suggest me where to look for such issue or what should be my
> > > next move for debugging. I am stuck with missing log file.
> > >
> > > For information here are the memory settings:
> > >
> > > <systemUsage>
> > >     <systemUsage>
> > >         <memoryUsage>
> > >             <memoryUsage limit="*256 mb*"/>
> > >         </memoryUsage>
> > >         <storeUsage>
> > >             <storeUsage limit="*100 gb*"/>
> > >         </storeUsage>
> > >         <tempUsage>
> > >             <tempUsage limit="*50 gb*"/>
> > >         </tempUsage>
> > >     </systemUsage>
> > > </systemUsage>
> > >
> >
> > You might want to start here and work out the model of usage you're
> seeing:
> > http://activemq.apache.org/javalangoutofmemory.html
> >
> >  For instance: thousands of messages, few clients; or thousands of
> messages
> > and lots of clients.
> >
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Vikas Agarwal
By crash, I mean message producers are not able to connect to ActiveMQ and
queue listeners stop processing messages because of the same reason. I have
to restart ActiveMQ to make everything working fine.

On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <[hidden email]> wrote:

> Most developers I know would define "crash" to be "the process
> terminated".  Since that doesn't seem to be the case here (and since you
> applied the term to an instance of Producer Flow Control in a previous
> situation, which is most definitely not an ActiveMQ crash nor an ActiveMQ
> error of any sort), I assume that's not your definition of the term.  Can
> you describe the symptoms you're seeing, without using the word "crash", to
> get everyone on the same page about what your problem actually is?
>
> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <[hidden email]>
> wrote:
>
> > I mean 50,000-100,000 messages are there in queue after which it crashes
> > and this range is not definitive, it some crashes below a pile of 50,000
> > messages too. Actually, I am not able to predict when exactly it is
> > crashing, if its is crashing due to CPU usage or memory usage or IO.
> >
> > Logs are behaving fine before the crash, so I believe everything is fine
> > with permissions and log level. Log rotation would not remove the
> > activemq.log completely. Further, after crash, activemq process is still
> > alive (checked using ps command), however, Admin UI (port 8161) is not
> > available after crash.
> >
> > On Mon, Nov 3, 2014 at 7:03 PM, James Green <[hidden email]>
> > wrote:
> >
> > > On 3 November 2014 09:42, Vikas Agarwal <[hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > We are using ActiveMQ for 3+ years, however, we didn't test it under
> > > heavy
> > > > load. Recently, we started using ActiveMQ in another project where
> > > > sometimes load increases exponentially as we are listening to twitter
> > > > stream which can pile up a lot of messages depending of occurrence of
> > > some
> > > > global event like iPhone 6 launch. Now, what we are seeing in
> ActiveMQ
> > > > admin UI is that one of the queues gets piled up with thousands of
> > > message
> > > > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ
> > crashes
> > > >
> > >
> > > Do you mean:
> > >
> > > 1. The queues have 50-100,000 messages each, or
> > > 2. The queues have thousands of messages each of which may be 50-100k
> in
> > > size?
> > >
> > >
> > > > in between of my sleeping hours. :( And the strange thing is that
> when
> > > try
> > > > to view the log file for the reason of the crash, I am not able to
> find
> > > the
> > > > activemq.log file at all. We earlier faced ActiveMQ crash issues, but
> > > that
> > > >
> > >
> > > Ensure you have INFO logging switched on and that AMQ writes to this
> log
> > on
> > > start-up. Also, that log could be rotated - ensure the process has
> > > permission to write to it.
> > >
> > >
> > > > was due to producer flow control and we disabled that to fix the
> issue
> > > and
> > > > main point is that in those cases we never had the case where the log
> > > file
> > > > was missing. I have to restart the process to fix the issue and that
> > too,
> > > > twice because on first restart, it shows error about missing or
> corrupt
> > > > data log.
> > > >
> > > > So, please suggest me where to look for such issue or what should be
> my
> > > > next move for debugging. I am stuck with missing log file.
> > > >
> > > > For information here are the memory settings:
> > > >
> > > > <systemUsage>
> > > >     <systemUsage>
> > > >         <memoryUsage>
> > > >             <memoryUsage limit="*256 mb*"/>
> > > >         </memoryUsage>
> > > >         <storeUsage>
> > > >             <storeUsage limit="*100 gb*"/>
> > > >         </storeUsage>
> > > >         <tempUsage>
> > > >             <tempUsage limit="*50 gb*"/>
> > > >         </tempUsage>
> > > >     </systemUsage>
> > > > </systemUsage>
> > > >
> > >
> > > You might want to start here and work out the model of usage you're
> > seeing:
> > > http://activemq.apache.org/javalangoutofmemory.html
> > >
> > >  For instance: thousands of messages, few clients; or thousands of
> > messages
> > > and lots of clients.
> > >
> >
> >
> >
> > --
> > Regards,
> > Vikas Agarwal
> > 91 – 9928301411
> >
> > InfoObjects, Inc.
> > Execution Matters
> > http://www.infoobjects.com
> > 2041 Mission College Boulevard, #280
> > Santa Clara, CA 95054
> > +1 (408) 988-2000 Work
> > +1 (408) 716-2726 Fax
> >
>



--
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Gaurav Sharma
A few qns to help with the situation:

1. what's the hardware where you're running the broker - #machines, type of machines, config, etc?

2. how much heap are you giving the amq processes? anything else running on the broker nodes or are they dedicated to amq?

3. are messages persistent? if so, what's the backing store? running transactions? how big is the message payload roughly?

4. how many producers and consumers and how do they connect? what clients are you using? what kinds of destinations and how many are there?

5. ran any perf/bench tests against this cluster to figure how far you can go with your workload type? if you don't measure, no way to tune and improve and iterate.

amq is a very flexible distributed messaging system but like any of the others, does best if you pay great attention to detail.


On Nov 3, 2014, at 20:54, Vikas Agarwal <[hidden email]> wrote:

By crash, I mean message producers are not able to connect to ActiveMQ and
queue listeners stop processing messages because of the same reason. I have
to restart ActiveMQ to make everything working fine.

> On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <[hidden email]> wrote:
>
> Most developers I know would define "crash" to be "the process
> terminated".  Since that doesn't seem to be the case here (and since you
> applied the term to an instance of Producer Flow Control in a previous
> situation, which is most definitely not an ActiveMQ crash nor an ActiveMQ
> error of any sort), I assume that's not your definition of the term.  Can
> you describe the symptoms you're seeing, without using the word "crash", to
> get everyone on the same page about what your problem actually is?
>
> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <[hidden email]>
> wrote:
>
>> I mean 50,000-100,000 messages are there in queue after which it crashes
>> and this range is not definitive, it some crashes below a pile of 50,000
>> messages too. Actually, I am not able to predict when exactly it is
>> crashing, if its is crashing due to CPU usage or memory usage or IO.
>>
>> Logs are behaving fine before the crash, so I believe everything is fine
>> with permissions and log level. Log rotation would not remove the
>> activemq.log completely. Further, after crash, activemq process is still
>> alive (checked using ps command), however, Admin UI (port 8161) is not
>> available after crash.
>>
>> On Mon, Nov 3, 2014 at 7:03 PM, James Green <[hidden email]>
>> wrote:
>>
>>>> On 3 November 2014 09:42, Vikas Agarwal <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We are using ActiveMQ for 3+ years, however, we didn't test it under
>>> heavy
>>>> load. Recently, we started using ActiveMQ in another project where
>>>> sometimes load increases exponentially as we are listening to twitter
>>>> stream which can pile up a lot of messages depending of occurrence of
>>> some
>>>> global event like iPhone 6 launch. Now, what we are seeing in
> ActiveMQ
>>>> admin UI is that one of the queues gets piled up with thousands of
>>> message
>>>> ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ
>> crashes
>>>
>>> Do you mean:
>>>
>>> 1. The queues have 50-100,000 messages each, or
>>> 2. The queues have thousands of messages each of which may be 50-100k
> in
>>> size?
>>>
>>>
>>>> in between of my sleeping hours. :( And the strange thing is that
> when
>>> try
>>>> to view the log file for the reason of the crash, I am not able to
> find
>>> the
>>>> activemq.log file at all. We earlier faced ActiveMQ crash issues, but
>>> that
>>>
>>> Ensure you have INFO logging switched on and that AMQ writes to this
> log
>> on
>>> start-up. Also, that log could be rotated - ensure the process has
>>> permission to write to it.
>>>
>>>
>>>> was due to producer flow control and we disabled that to fix the
> issue
>>> and
>>>> main point is that in those cases we never had the case where the log
>>> file
>>>> was missing. I have to restart the process to fix the issue and that
>> too,
>>>> twice because on first restart, it shows error about missing or
> corrupt
>>>> data log.
>>>>
>>>> So, please suggest me where to look for such issue or what should be
> my
>>>> next move for debugging. I am stuck with missing log file.
>>>>
>>>> For information here are the memory settings:
>>>>
>>>> <systemUsage>
>>>>    <systemUsage>
>>>>        <memoryUsage>
>>>>            <memoryUsage limit="*256 mb*"/>
>>>>        </memoryUsage>
>>>>        <storeUsage>
>>>>            <storeUsage limit="*100 gb*"/>
>>>>        </storeUsage>
>>>>        <tempUsage>
>>>>            <tempUsage limit="*50 gb*"/>
>>>>        </tempUsage>
>>>>    </systemUsage>
>>>> </systemUsage>
>>>
>>> You might want to start here and work out the model of usage you're
>> seeing:
>>> http://activemq.apache.org/javalangoutofmemory.html
>>>
>>> For instance: thousands of messages, few clients; or thousands of
>> messages
>>> and lots of clients.
>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax



--
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Vikas Agarwal
In reply to this post by Vikas Agarwal
For more information, I see following messages in logs (whenever, ActiveMQ
fails to respond to producer and consumer but logs are available)

2014-11-04 00:08:03,893 | WARN  | Transport Connection to: tcp://<ip>:45550
failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///<ip>:45550


And on client side

WARN  org.apache.activemq.transport.failover.FailoverTransport  - Transport
(tcp://<ip>:61616) failed, reason:  java.io.IOException: Wire format
negotiation timeout: peer did not send his wire format., attempting to
automatically reconnect


Strange things is that with this state, everything works sometimes while
does not work at rest of the time.

On Tue, Nov 4, 2014 at 10:24 AM, Vikas Agarwal <[hidden email]>
wrote:

> By crash, I mean message producers are not able to connect to ActiveMQ and
> queue listeners stop processing messages because of the same reason. I have
> to restart ActiveMQ to make everything working fine.
>
> On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <[hidden email]> wrote:
>
>> Most developers I know would define "crash" to be "the process
>> terminated".  Since that doesn't seem to be the case here (and since you
>> applied the term to an instance of Producer Flow Control in a previous
>> situation, which is most definitely not an ActiveMQ crash nor an ActiveMQ
>> error of any sort), I assume that's not your definition of the term.  Can
>> you describe the symptoms you're seeing, without using the word "crash",
>> to
>> get everyone on the same page about what your problem actually is?
>>
>> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <[hidden email]>
>> wrote:
>>
>> > I mean 50,000-100,000 messages are there in queue after which it crashes
>> > and this range is not definitive, it some crashes below a pile of 50,000
>> > messages too. Actually, I am not able to predict when exactly it is
>> > crashing, if its is crashing due to CPU usage or memory usage or IO.
>> >
>> > Logs are behaving fine before the crash, so I believe everything is fine
>> > with permissions and log level. Log rotation would not remove the
>> > activemq.log completely. Further, after crash, activemq process is still
>> > alive (checked using ps command), however, Admin UI (port 8161) is not
>> > available after crash.
>> >
>> > On Mon, Nov 3, 2014 at 7:03 PM, James Green <[hidden email]>
>> > wrote:
>> >
>> > > On 3 November 2014 09:42, Vikas Agarwal <[hidden email]>
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > We are using ActiveMQ for 3+ years, however, we didn't test it under
>> > > heavy
>> > > > load. Recently, we started using ActiveMQ in another project where
>> > > > sometimes load increases exponentially as we are listening to
>> twitter
>> > > > stream which can pile up a lot of messages depending of occurrence
>> of
>> > > some
>> > > > global event like iPhone 6 launch. Now, what we are seeing in
>> ActiveMQ
>> > > > admin UI is that one of the queues gets piled up with thousands of
>> > > message
>> > > > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ
>> > crashes
>> > > >
>> > >
>> > > Do you mean:
>> > >
>> > > 1. The queues have 50-100,000 messages each, or
>> > > 2. The queues have thousands of messages each of which may be 50-100k
>> in
>> > > size?
>> > >
>> > >
>> > > > in between of my sleeping hours. :( And the strange thing is that
>> when
>> > > try
>> > > > to view the log file for the reason of the crash, I am not able to
>> find
>> > > the
>> > > > activemq.log file at all. We earlier faced ActiveMQ crash issues,
>> but
>> > > that
>> > > >
>> > >
>> > > Ensure you have INFO logging switched on and that AMQ writes to this
>> log
>> > on
>> > > start-up. Also, that log could be rotated - ensure the process has
>> > > permission to write to it.
>> > >
>> > >
>> > > > was due to producer flow control and we disabled that to fix the
>> issue
>> > > and
>> > > > main point is that in those cases we never had the case where the
>> log
>> > > file
>> > > > was missing. I have to restart the process to fix the issue and that
>> > too,
>> > > > twice because on first restart, it shows error about missing or
>> corrupt
>> > > > data log.
>> > > >
>> > > > So, please suggest me where to look for such issue or what should
>> be my
>> > > > next move for debugging. I am stuck with missing log file.
>> > > >
>> > > > For information here are the memory settings:
>> > > >
>> > > > <systemUsage>
>> > > >     <systemUsage>
>> > > >         <memoryUsage>
>> > > >             <memoryUsage limit="*256 mb*"/>
>> > > >         </memoryUsage>
>> > > >         <storeUsage>
>> > > >             <storeUsage limit="*100 gb*"/>
>> > > >         </storeUsage>
>> > > >         <tempUsage>
>> > > >             <tempUsage limit="*50 gb*"/>
>> > > >         </tempUsage>
>> > > >     </systemUsage>
>> > > > </systemUsage>
>> > > >
>> > >
>> > > You might want to start here and work out the model of usage you're
>> > seeing:
>> > > http://activemq.apache.org/javalangoutofmemory.html
>> > >
>> > >  For instance: thousands of messages, few clients; or thousands of
>> > messages
>> > > and lots of clients.
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Vikas Agarwal
>> > 91 – 9928301411
>> >
>> > InfoObjects, Inc.
>> > Execution Matters
>> > http://www.infoobjects.com
>> > 2041 Mission College Boulevard, #280
>> > Santa Clara, CA 95054
>> > +1 (408) 988-2000 Work
>> > +1 (408) 716-2726 Fax
>> >
>>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>


--
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Tim Bain
Can you try increasing the wire format's maxInactivityDurationInitialDelay
as described by Christian in a similar-sounding problem last year:
http://activemq.2283324.n4.nabble.com/Wire-Format-Negotiation-Timeout-Say-what-td4666972.html

That may not be the final solution (there might be underlying network
problems that need to be addressed by your admins), but it should help
diagnose if that's what's going on...

On Mon, Nov 3, 2014 at 10:17 PM, Vikas Agarwal <[hidden email]>
wrote:

> For more information, I see following messages in logs (whenever, ActiveMQ
> fails to respond to producer and consumer but logs are available)
>
> 2014-11-04 00:08:03,893 | WARN  | Transport Connection to: tcp://<ip>:45550
> failed: java.io.EOFException |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///<ip>:45550
>
>
> And on client side
>
> WARN  org.apache.activemq.transport.failover.FailoverTransport  - Transport
> (tcp://<ip>:61616) failed, reason:  java.io.IOException: Wire format
> negotiation timeout: peer did not send his wire format., attempting to
> automatically reconnect
>
>
> Strange things is that with this state, everything works sometimes while
> does not work at rest of the time.
>
> On Tue, Nov 4, 2014 at 10:24 AM, Vikas Agarwal <[hidden email]>
> wrote:
>
> > By crash, I mean message producers are not able to connect to ActiveMQ
> and
> > queue listeners stop processing messages because of the same reason. I
> have
> > to restart ActiveMQ to make everything working fine.
> >
> > On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <[hidden email]> wrote:
> >
> >> Most developers I know would define "crash" to be "the process
> >> terminated".  Since that doesn't seem to be the case here (and since you
> >> applied the term to an instance of Producer Flow Control in a previous
> >> situation, which is most definitely not an ActiveMQ crash nor an
> ActiveMQ
> >> error of any sort), I assume that's not your definition of the term.
> Can
> >> you describe the symptoms you're seeing, without using the word "crash",
> >> to
> >> get everyone on the same page about what your problem actually is?
> >>
> >> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <[hidden email]>
> >> wrote:
> >>
> >> > I mean 50,000-100,000 messages are there in queue after which it
> crashes
> >> > and this range is not definitive, it some crashes below a pile of
> 50,000
> >> > messages too. Actually, I am not able to predict when exactly it is
> >> > crashing, if its is crashing due to CPU usage or memory usage or IO.
> >> >
> >> > Logs are behaving fine before the crash, so I believe everything is
> fine
> >> > with permissions and log level. Log rotation would not remove the
> >> > activemq.log completely. Further, after crash, activemq process is
> still
> >> > alive (checked using ps command), however, Admin UI (port 8161) is not
> >> > available after crash.
> >> >
> >> > On Mon, Nov 3, 2014 at 7:03 PM, James Green <[hidden email]
> >
> >> > wrote:
> >> >
> >> > > On 3 November 2014 09:42, Vikas Agarwal <[hidden email]>
> >> wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > We are using ActiveMQ for 3+ years, however, we didn't test it
> under
> >> > > heavy
> >> > > > load. Recently, we started using ActiveMQ in another project where
> >> > > > sometimes load increases exponentially as we are listening to
> >> twitter
> >> > > > stream which can pile up a lot of messages depending of occurrence
> >> of
> >> > > some
> >> > > > global event like iPhone 6 launch. Now, what we are seeing in
> >> ActiveMQ
> >> > > > admin UI is that one of the queues gets piled up with thousands of
> >> > > message
> >> > > > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ
> >> > crashes
> >> > > >
> >> > >
> >> > > Do you mean:
> >> > >
> >> > > 1. The queues have 50-100,000 messages each, or
> >> > > 2. The queues have thousands of messages each of which may be
> 50-100k
> >> in
> >> > > size?
> >> > >
> >> > >
> >> > > > in between of my sleeping hours. :( And the strange thing is that
> >> when
> >> > > try
> >> > > > to view the log file for the reason of the crash, I am not able to
> >> find
> >> > > the
> >> > > > activemq.log file at all. We earlier faced ActiveMQ crash issues,
> >> but
> >> > > that
> >> > > >
> >> > >
> >> > > Ensure you have INFO logging switched on and that AMQ writes to this
> >> log
> >> > on
> >> > > start-up. Also, that log could be rotated - ensure the process has
> >> > > permission to write to it.
> >> > >
> >> > >
> >> > > > was due to producer flow control and we disabled that to fix the
> >> issue
> >> > > and
> >> > > > main point is that in those cases we never had the case where the
> >> log
> >> > > file
> >> > > > was missing. I have to restart the process to fix the issue and
> that
> >> > too,
> >> > > > twice because on first restart, it shows error about missing or
> >> corrupt
> >> > > > data log.
> >> > > >
> >> > > > So, please suggest me where to look for such issue or what should
> >> be my
> >> > > > next move for debugging. I am stuck with missing log file.
> >> > > >
> >> > > > For information here are the memory settings:
> >> > > >
> >> > > > <systemUsage>
> >> > > >     <systemUsage>
> >> > > >         <memoryUsage>
> >> > > >             <memoryUsage limit="*256 mb*"/>
> >> > > >         </memoryUsage>
> >> > > >         <storeUsage>
> >> > > >             <storeUsage limit="*100 gb*"/>
> >> > > >         </storeUsage>
> >> > > >         <tempUsage>
> >> > > >             <tempUsage limit="*50 gb*"/>
> >> > > >         </tempUsage>
> >> > > >     </systemUsage>
> >> > > > </systemUsage>
> >> > > >
> >> > >
> >> > > You might want to start here and work out the model of usage you're
> >> > seeing:
> >> > > http://activemq.apache.org/javalangoutofmemory.html
> >> > >
> >> > >  For instance: thousands of messages, few clients; or thousands of
> >> > messages
> >> > > and lots of clients.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Vikas Agarwal
> >> > 91 – 9928301411
> >> >
> >> > InfoObjects, Inc.
> >> > Execution Matters
> >> > http://www.infoobjects.com
> >> > 2041 Mission College Boulevard, #280
> >> > Santa Clara, CA 95054
> >> > +1 (408) 988-2000 Work
> >> > +1 (408) 716-2726 Fax
> >> >
> >>
> >
> >
> >
> > --
> > Regards,
> > Vikas Agarwal
> > 91 – 9928301411
> >
> > InfoObjects, Inc.
> > Execution Matters
> > http://www.infoobjects.com
> > 2041 Mission College Boulevard, #280
> > Santa Clara, CA 95054
> > +1 (408) 988-2000 Work
> > +1 (408) 716-2726 Fax
> >
> >
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
Reply | Threaded
Open this post in threaded view
|

Re: Load issue with ActiveMQ

Vikas Agarwal
Thanks for pointer Tim. However, I have disabled inactivity check by setting

wireFormat.maxInactivityDuration=0

because earlier we were facing connection breaking because of inactivity.

However, I am able to debug the issue. This time I looked into one of the
rolled log file and found that there was a few errors saying "too many open
files". This triggered me to think over monitoring current threads and I
found that number of threads crossed the limits of default system limit
(ulimit -a) 1024. We were running a few more processes on the same machine
with the same user which are actually using few hundreds of threads which
caused the limits to reach quickly. So, I increased the system limits for
threads and open files by copying the limit configuration for yarn user
from other machine which increased the values as

<username> - nofile 32768
<username> - nproc 65536

And after setting these values and restarting the machine, I observed the
system for 2-3 days and it worked. System didn't show the crash behaviour I
mentioned.


Thanks everyone for your valuable time and suggestions. :)

On Tue, Nov 4, 2014 at 10:36 PM, Tim Bain <[hidden email]> wrote:

> Can you try increasing the wire format's maxInactivityDurationInitialDelay
> as described by Christian in a similar-sounding problem last year:
>
> http://activemq.2283324.n4.nabble.com/Wire-Format-Negotiation-Timeout-Say-what-td4666972.html
>
> That may not be the final solution (there might be underlying network
> problems that need to be addressed by your admins), but it should help
> diagnose if that's what's going on...
>
> On Mon, Nov 3, 2014 at 10:17 PM, Vikas Agarwal <[hidden email]>
> wrote:
>
> > For more information, I see following messages in logs (whenever,
> ActiveMQ
> > fails to respond to producer and consumer but logs are available)
> >
> > 2014-11-04 00:08:03,893 | WARN  | Transport Connection to:
> tcp://<ip>:45550
> > failed: java.io.EOFException |
> > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> > Transport: tcp:///<ip>:45550
> >
> >
> > And on client side
> >
> > WARN  org.apache.activemq.transport.failover.FailoverTransport  -
> Transport
> > (tcp://<ip>:61616) failed, reason:  java.io.IOException: Wire format
> > negotiation timeout: peer did not send his wire format., attempting to
> > automatically reconnect
> >
> >
> > Strange things is that with this state, everything works sometimes while
> > does not work at rest of the time.
> >
> > On Tue, Nov 4, 2014 at 10:24 AM, Vikas Agarwal <[hidden email]>
> > wrote:
> >
> > > By crash, I mean message producers are not able to connect to ActiveMQ
> > and
> > > queue listeners stop processing messages because of the same reason. I
> > have
> > > to restart ActiveMQ to make everything working fine.
> > >
> > > On Tue, Nov 4, 2014 at 12:00 AM, Tim Bain <[hidden email]>
> wrote:
> > >
> > >> Most developers I know would define "crash" to be "the process
> > >> terminated".  Since that doesn't seem to be the case here (and since
> you
> > >> applied the term to an instance of Producer Flow Control in a previous
> > >> situation, which is most definitely not an ActiveMQ crash nor an
> > ActiveMQ
> > >> error of any sort), I assume that's not your definition of the term.
> > Can
> > >> you describe the symptoms you're seeing, without using the word
> "crash",
> > >> to
> > >> get everyone on the same page about what your problem actually is?
> > >>
> > >> On Mon, Nov 3, 2014 at 10:15 AM, Vikas Agarwal <[hidden email]
> >
> > >> wrote:
> > >>
> > >> > I mean 50,000-100,000 messages are there in queue after which it
> > crashes
> > >> > and this range is not definitive, it some crashes below a pile of
> > 50,000
> > >> > messages too. Actually, I am not able to predict when exactly it is
> > >> > crashing, if its is crashing due to CPU usage or memory usage or IO.
> > >> >
> > >> > Logs are behaving fine before the crash, so I believe everything is
> > fine
> > >> > with permissions and log level. Log rotation would not remove the
> > >> > activemq.log completely. Further, after crash, activemq process is
> > still
> > >> > alive (checked using ps command), however, Admin UI (port 8161) is
> not
> > >> > available after crash.
> > >> >
> > >> > On Mon, Nov 3, 2014 at 7:03 PM, James Green <
> [hidden email]
> > >
> > >> > wrote:
> > >> >
> > >> > > On 3 November 2014 09:42, Vikas Agarwal <[hidden email]>
> > >> wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > We are using ActiveMQ for 3+ years, however, we didn't test it
> > under
> > >> > > heavy
> > >> > > > load. Recently, we started using ActiveMQ in another project
> where
> > >> > > > sometimes load increases exponentially as we are listening to
> > >> twitter
> > >> > > > stream which can pile up a lot of messages depending of
> occurrence
> > >> of
> > >> > > some
> > >> > > > global event like iPhone 6 launch. Now, what we are seeing in
> > >> ActiveMQ
> > >> > > > admin UI is that one of the queues gets piled up with thousands
> of
> > >> > > message
> > >> > > > ranging from 50k-100k. Now, sometimes (almost daily) the
> ActiveMQ
> > >> > crashes
> > >> > > >
> > >> > >
> > >> > > Do you mean:
> > >> > >
> > >> > > 1. The queues have 50-100,000 messages each, or
> > >> > > 2. The queues have thousands of messages each of which may be
> > 50-100k
> > >> in
> > >> > > size?
> > >> > >
> > >> > >
> > >> > > > in between of my sleeping hours. :( And the strange thing is
> that
> > >> when
> > >> > > try
> > >> > > > to view the log file for the reason of the crash, I am not able
> to
> > >> find
> > >> > > the
> > >> > > > activemq.log file at all. We earlier faced ActiveMQ crash
> issues,
> > >> but
> > >> > > that
> > >> > > >
> > >> > >
> > >> > > Ensure you have INFO logging switched on and that AMQ writes to
> this
> > >> log
> > >> > on
> > >> > > start-up. Also, that log could be rotated - ensure the process has
> > >> > > permission to write to it.
> > >> > >
> > >> > >
> > >> > > > was due to producer flow control and we disabled that to fix the
> > >> issue
> > >> > > and
> > >> > > > main point is that in those cases we never had the case where
> the
> > >> log
> > >> > > file
> > >> > > > was missing. I have to restart the process to fix the issue and
> > that
> > >> > too,
> > >> > > > twice because on first restart, it shows error about missing or
> > >> corrupt
> > >> > > > data log.
> > >> > > >
> > >> > > > So, please suggest me where to look for such issue or what
> should
> > >> be my
> > >> > > > next move for debugging. I am stuck with missing log file.
> > >> > > >
> > >> > > > For information here are the memory settings:
> > >> > > >
> > >> > > > <systemUsage>
> > >> > > >     <systemUsage>
> > >> > > >         <memoryUsage>
> > >> > > >             <memoryUsage limit="*256 mb*"/>
> > >> > > >         </memoryUsage>
> > >> > > >         <storeUsage>
> > >> > > >             <storeUsage limit="*100 gb*"/>
> > >> > > >         </storeUsage>
> > >> > > >         <tempUsage>
> > >> > > >             <tempUsage limit="*50 gb*"/>
> > >> > > >         </tempUsage>
> > >> > > >     </systemUsage>
> > >> > > > </systemUsage>
> > >> > > >
> > >> > >
> > >> > > You might want to start here and work out the model of usage
> you're
> > >> > seeing:
> > >> > > http://activemq.apache.org/javalangoutofmemory.html
> > >> > >
> > >> > >  For instance: thousands of messages, few clients; or thousands of
> > >> > messages
> > >> > > and lots of clients.
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Vikas Agarwal
> > >> > 91 – 9928301411
> > >> >
> > >> > InfoObjects, Inc.
> > >> > Execution Matters
> > >> > http://www.infoobjects.com
> > >> > 2041 Mission College Boulevard, #280
> > >> > Santa Clara, CA 95054
> > >> > +1 (408) 988-2000 Work
> > >> > +1 (408) 716-2726 Fax
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Vikas Agarwal
> > > 91 – 9928301411
> > >
> > > InfoObjects, Inc.
> > > Execution Matters
> > > http://www.infoobjects.com
> > > 2041 Mission College Boulevard, #280
> > > Santa Clara, CA 95054
> > > +1 (408) 988-2000 Work
> > > +1 (408) 716-2726 Fax
> > >
> > >
> >
> >
> > --
> > Regards,
> > Vikas Agarwal
> > 91 – 9928301411
> >
> > InfoObjects, Inc.
> > Execution Matters
> > http://www.infoobjects.com
> > 2041 Mission College Boulevard, #280
> > Santa Clara, CA 95054
> > +1 (408) 988-2000 Work
> > +1 (408) 716-2726 Fax
> >
>



--
Regards,
Vikas Agarwal
91 – 9928301411

InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax