Quantcast

Artemis 1.5.1 : org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction messages under some load

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Artemis 1.5.1 : org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction messages under some load

mlange
Currently I'm able to start some load testing using Artemis; The load is nowhere near what I expect to need, but that's currently not the issue.

Of course I test both Artemis performance and the producers / consumers. The performance itself is not at stake though, there's plenty of room for improvements anyway.

Using JMeter I sent a total of 50.000 messages using 50 threads to my servicemix which puts the messages on the Artemis broker. Next a bunch of processes get a message, does it's work and puts it on the next queue (/endpoint)

After some time, I noticed this: message repeatedly in the log:

WARN  [org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction 32,212,560,158, ignoring message on position PagePositionImpl [pageNr=12, messageNr=1851, recordID=-1] on address=<the actual queue name>

What does this message mean exactly? A google on AMQ2222029 does not yield any usable results.
Am I losing messages? What can I do to fix / avoid this, whatever it is I'm seeing? Please shed some light on what is happening.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Artemis 1.5.1 : org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction messages under some load

clebertsuconic
when you send messages in paging, they get written into page and the
journal in case you send transactionally.

if you restart the system, or kill, in case you're sending paged.. the
record will be written on the page files but it may be lost from the
journal.

Once you restart the page-tx is not there and you will see this warning.



If you over-use paging, I recommend to send not transactionally though.

On Fri, Feb 3, 2017 at 2:25 PM, mlange <[hidden email]> wrote:

> Currently I'm able to start some load testing using Artemis; The load is
> nowhere near what I expect to need, but that's currently not the issue.
>
> Of course I test both Artemis performance and the producers / consumers. The
> performance itself is not at stake though, there's plenty of room for
> improvements anyway.
>
> Using JMeter I sent a total of 50.000 messages using 50 threads to my
> servicemix which puts the messages on the Artemis broker. Next a bunch of
> processes get a message, does it's work and puts it on the next queue
> (/endpoint)
>
> After some time, I noticed this: message repeatedly in the log:
>
> WARN  [org.apache.activemq.artemis.core.server] AMQ222029: Could not locate
> page transaction 32,212,560,158, ignoring message on position
> PagePositionImpl [pageNr=12, messageNr=1851, recordID=-1] on address=<the
> actual queue name>
>
> What does this message mean exactly? A google on AMQ2222029 does not yield
> any usable results.
> Am I losing messages? What can I do to fix / avoid this, whatever it is I'm
> seeing? Please shed some light on what is happening.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Artemis-1-5-1-org-apache-activemq-artemis-core-server-AMQ222029-Could-not-locate-page-transaction-med-tp4721587.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Artemis 1.5.1 : org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction messages under some load

clebertsuconic
In reply to this post by mlange
> Using JMeter I sent a total of 50.000 messages using 50 threads to my
servicemix which puts the messages on the Artemis broker. Next a bunch of
processes get a message, does it's work and puts it on the next queue
(/endpoint)


Also, make sure that each thread is using its own Session / Producer / Consumer.

On Fri, Feb 3, 2017 at 2:25 PM, mlange <[hidden email]> wrote:

> Currently I'm able to start some load testing using Artemis; The load is
> nowhere near what I expect to need, but that's currently not the issue.
>
> Of course I test both Artemis performance and the producers / consumers. The
> performance itself is not at stake though, there's plenty of room for
> improvements anyway.
>
> Using JMeter I sent a total of 50.000 messages using 50 threads to my
> servicemix which puts the messages on the Artemis broker. Next a bunch of
> processes get a message, does it's work and puts it on the next queue
> (/endpoint)
>
> After some time, I noticed this: message repeatedly in the log:
>
> WARN  [org.apache.activemq.artemis.core.server] AMQ222029: Could not locate
> page transaction 32,212,560,158, ignoring message on position
> PagePositionImpl [pageNr=12, messageNr=1851, recordID=-1] on address=<the
> actual queue name>
>
> What does this message mean exactly? A google on AMQ2222029 does not yield
> any usable results.
> Am I losing messages? What can I do to fix / avoid this, whatever it is I'm
> seeing? Please shed some light on what is happening.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Artemis-1-5-1-org-apache-activemq-artemis-core-server-AMQ222029-Could-not-locate-page-transaction-med-tp4721587.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Artemis 1.5.1 : org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction messages under some load

mlange
Strangely, the situation is a bit complicated:

In a prior test I received messages that the broker could not start a thread due to not having enough heap available. It would not even shutdown, because it could not start the shutdown thread. Eventually the broker got killed, which then was taken over by the backup.

After this test, I manually removed all queues, but did not delete the journals. Also, I changed artemis.profile to use 4096M heap (up from 1024).

All producers and consumers were stopped prior to the removal of all the queues. The producers and consumers were started and the test repeated...

The test setup is like this:
- We use servicemix to provide the webservice(s), like this blueprint:
<blueprint
        xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:camel="http://camel.apache.org/schema/blueprint"
        xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
        xsi:schemaLocation="
                http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
                http://camel.apache.org/schema/blueprint/cxf http://camel.apache.org/schema/blueprint/cxf/camel-cxf.xsd
                http://camel.apache.org/schema/blueprint  http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

        <reference id="broker1" interface="javax.jms.ConnectionFactory" filter="(osgi.jndi.service.name=jms/broker1)" />

        <camelContext xmlns="http://camel.apache.org/schema/blueprint" id="Webservices" xmlns:ns="http://my.test/webservices/perform_load_test_request/1">
                <route id="ULT3-Incoming" streamCache="true">
                        <from uri="cxf:bean:MyLoadTest_Endpoint?dataFormat=PAYLOAD" id="PerformLoadTest3-Webservice"/>
                        <transacted />
                        <to uri="activemq:my.test.queue?connectionFactory=broker1&amp;jmsMessageType=Text" id="PerformLoadTestTest3-ToQueue"/>
                </route>
        </camelContext>

        <cxf:cxfEndpoint id="MyLoadTest_Endpoint" wsdlURL="wsdl/perform_load_test_1.1.wsdl" address="http://0.0.0.0:8383/webservices/perform_load_test_01" />
</blueprint>

the broker connection is setup like this:
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">

    <bean id="activemqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
        <property name="brokerURL" value="failover:(tcp://brokerhost1:10010,tcp://brokerhost2:10010)" />
        <property name="userName" value="xxxxx" />
        <property name="password" value="yyyyy" />
    </bean>

    <bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
        <property name="maxConnections" value="8" />
        <property name="connectionFactory" ref="activemqConnectionFactory" />
    </bean>

    <bean id="resourceManager" class="org.apache.activemq.pool.ActiveMQResourceManager" init-method="recoverResource">
        <property name="transactionManager" ref="transactionManager" />
        <property name="connectionFactory" ref="activemqConnectionFactory" />
        <property name="resourceName" value="activemq.localhost" />
    </bean>

    <reference id="transactionManager" interface="javax.transaction.TransactionManager" />

    <service ref="pooledConnectionFactory" interface="javax.jms.ConnectionFactory">
        <service-properties>
            <entry key="name" value="broker1" />
            <entry key="osgi.jndi.service.name" value="jms/broker1" />
        </service-properties>
    </service>

</blueprint>

This producer seems to do the job fine. The consumers/producers are running in a separate java process that uses code I have only limited influence upon, but I reckon each process has it's own (1) session with "many" (in the case of this process 40) threads.

I see this message (Could not locate page transaction) only on the 1st queue, but do not seem to loose any messages. (That's good, reliability takes precedence over performance, although performance is quite important as well)

Could the message also happen due to the default prefetch of 1000 messages?
Is there actually any harm?
Should I increase the global-max-size directive in broker.xml? And is 400MB (I reckon the value is in KB, so the default is 100MB) the safe choice with a 4GB heap?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Artemis 1.5.1 : org.apache.activemq.artemis.core.server] AMQ222029: Could not locate page transaction messages under some load

clebertsuconic
it seems a result from a previous running broker. It means the TX is
not available on the journal and it will be ignored.. I think it's
safe since you had these issues while running the broker.


I would start fresh and re-run your tests.

On Mon, Feb 6, 2017 at 11:10 AM, mlange <[hidden email]> wrote:

> Strangely, the situation is a bit complicated:
>
> In a prior test I received messages that the broker could not start a thread
> due to not having enough heap available. It would not even shutdown, because
> it could not start the shutdown thread. Eventually the broker got killed,
> which then was taken over by the backup.
>
> After this test, I manually removed all queues, but did not delete the
> journals. Also, I changed artemis.profile to use 4096M heap (up from 1024).
>
> All producers and consumers were stopped prior to the removal of all the
> queues. The producers and consumers were started and the test repeated...
>
> The test setup is like this:
> - We use servicemix to provide the webservice(s), like this blueprint:
> <blueprint
>         xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
>         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>         xmlns:camel="http://camel.apache.org/schema/blueprint"
>         xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
>         xsi:schemaLocation="
>                 http://www.osgi.org/xmlns/blueprint/v1.0.0
> http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
>                 http://camel.apache.org/schema/blueprint/cxf
> http://camel.apache.org/schema/blueprint/cxf/camel-cxf.xsd
>                 http://camel.apache.org/schema/blueprint
> http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
>
>         <reference id="broker1" interface="javax.jms.ConnectionFactory"
> filter="(osgi.jndi.service.name=jms/broker1)" />
>
>         <camelContext xmlns="http://camel.apache.org/schema/blueprint"
> id="Webservices"
> xmlns:ns="http://my.test/webservices/perform_load_test_request/1">
>                 <route id="ULT3-Incoming" streamCache="true">
>                         <from
> uri="cxf:bean:MyLoadTest_Endpoint?dataFormat=PAYLOAD"
> id="PerformLoadTest3-Webservice"/>
>                         <transacted />
>                         <to
> uri="activemq:my.test.queue?connectionFactory=broker1&amp;jmsMessageType=Text"
> id="PerformLoadTestTest3-ToQueue"/>
>                 </route>
>         </camelContext>
>
>         <cxf:cxfEndpoint id="MyLoadTest_Endpoint"
> wsdlURL="wsdl/perform_load_test_1.1.wsdl"
> address="http://0.0.0.0:8383/webservices/perform_load_test_01" />
> </blueprint>
>
> the broker connection is setup like this:
> <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
>
>     <bean id="activemqConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
>         <property name="brokerURL"
> value="failover:(tcp://brokerhost1:10010,tcp://brokerhost2:10010)" />
>         <property name="userName" value="xxxxx" />
>         <property name="password" value="yyyyy" />
>     </bean>
>
>     <bean id="pooledConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactory">
>         <property name="maxConnections" value="8" />
>         <property name="connectionFactory" ref="activemqConnectionFactory"
> />
>     </bean>
>
>     <bean id="resourceManager"
> class="org.apache.activemq.pool.ActiveMQResourceManager"
> init-method="recoverResource">
>         <property name="transactionManager" ref="transactionManager" />
>         <property name="connectionFactory" ref="activemqConnectionFactory"
> />
>         <property name="resourceName" value="activemq.localhost" />
>     </bean>
>
>     <reference id="transactionManager"
> interface="javax.transaction.TransactionManager" />
>
>     <service ref="pooledConnectionFactory"
> interface="javax.jms.ConnectionFactory">
>         <service-properties>
>             <entry key="name" value="broker1" />
>             <entry key="osgi.jndi.service.name" value="jms/broker1" />
>         </service-properties>
>     </service>
>
> </blueprint>
>
> This producer seems to do the job fine. The consumers/producers are running
> in a separate java process that uses code I have only limited influence
> upon, but I reckon each process has it's own (1) session with "many" (in the
> case of this process 40) threads.
>
> I see this message (Could not locate page transaction) only on the 1st
> queue, but do not seem to loose any messages. (That's good, reliability
> takes precedence over performance, although performance is quite important
> as well)
>
> Could the message also happen due to the default prefetch of 1000 messages?
> Is there actually any harm?
> Should I increase the global-max-size directive in broker.xml? And is 400MB
> (I reckon the value is in KB, so the default is 100MB) the safe choice with
> a 4GB heap?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Artemis-1-5-1-org-apache-activemq-artemis-core-server-AMQ222029-Could-not-locate-page-transaction-med-tp4721587p4721634.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Loading...