Encourage more users to embed activemq?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Encourage more users to embed activemq?

Kevin Burton
When we initially deployed activemq we took the normal route of using it
with an init.d and run the daemon like we do apache, cassandra, etc.  Like
a daemon.

However, I found that we lean pretty hard on the actual implementation of
ActiveMQ and need to go above and beyond what ActiveMQ provides.

For example, JMX was too slow for us, so I implemented a basic REST API
that allows us to fetch queue metadata.  We went from 15000ms to 20ms to
fetch all the metadata.

We’re also able to embed other daemons this way such as our own servlets,
metrics, etc.

Seems like it might be a good idea to encourage ActiveMQ users to embed AMQ
in their own application and look at using BrokerService as more of an API
vs a daemon.



--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: Encourage more users to embed activemq?

Tim Bain
If you embed it in your app, then you lose the ability to cycle your app
without taking down your broker (which is a bad thing if you use
non-persistent messaging as we do).  Clearly that trade-off is a win in
your situation, but your situation's a bit of an outlier in terms of
performance, so the forces that drive you to embed may not be applicable to
users with more mainstream performance requirements.

With that being said, I'd be really interested in the ability to expose a
RESTful API around the broker so you can get JMX stats faster than JMX will
allow, because we've been hampered by how slow JMX is.  We're trying to
poll dozens/hundreds of destination/consumer/producer MBeans every
second to log out status (to help us look for sources of lag within the
system), and JMX is completely unable to keep up, so we'd love a better
option.  I'm not sure if that would be done by embedding ActiveMQ within an
app that provides only that RESTful API or by rolling the API into the core
broker executable, but if you're still thinking of open-sourcing that code
(whether as part of the ActiveMQ project or as a companion project), we'd
be interested in using it.

Tim

On Fri, Apr 17, 2015 at 11:50 AM, Kevin Burton <[hidden email]> wrote:

> When we initially deployed activemq we took the normal route of using it
> with an init.d and run the daemon like we do apache, cassandra, etc.  Like
> a daemon.
>
> However, I found that we lean pretty hard on the actual implementation of
> ActiveMQ and need to go above and beyond what ActiveMQ provides.
>
> For example, JMX was too slow for us, so I implemented a basic REST API
> that allows us to fetch queue metadata.  We went from 15000ms to 20ms to
> fetch all the metadata.
>
> We’re also able to embed other daemons this way such as our own servlets,
> metrics, etc.
>
> Seems like it might be a good idea to encourage ActiveMQ users to embed AMQ
> in their own application and look at using BrokerService as more of an API
> vs a daemon.
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Reply | Threaded
Open this post in threaded view
|

Re: Encourage more users to embed activemq?

David Jencks
In reply to this post by Kevin Burton
From my highly osgi-centric viewpoint this seems like a good use of running activemq in an osgi environment exposing services that can be used by e.g. your REST stuff to publish them externally.  Adding servlets would be easy using the http whiteboard service….

thanks
david jencks

On Apr 17, 2015, at 1:50 PM, Kevin Burton <[hidden email]> wrote:

> When we initially deployed activemq we took the normal route of using it
> with an init.d and run the daemon like we do apache, cassandra, etc.  Like
> a daemon.
>
> However, I found that we lean pretty hard on the actual implementation of
> ActiveMQ and need to go above and beyond what ActiveMQ provides.
>
> For example, JMX was too slow for us, so I implemented a basic REST API
> that allows us to fetch queue metadata.  We went from 15000ms to 20ms to
> fetch all the metadata.
>
> We’re also able to embed other daemons this way such as our own servlets,
> metrics, etc.
>
> Seems like it might be a good idea to encourage ActiveMQ users to embed AMQ
> in their own application and look at using BrokerService as more of an API
> vs a daemon.
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>

Reply | Threaded
Open this post in threaded view
|

Re: Encourage more users to embed activemq?

Kevin Burton
In reply to this post by Tim Bain
On Fri, Apr 17, 2015 at 11:53 AM, Tim Bain <[hidden email]> wrote:

> If you embed it in your app, then you lose the ability to cycle your app
> without taking down your broker (which is a bad thing if you use
> non-persistent messaging as we do).


Oh. to clarify.  I built our own foo-activemq service.  So it benefits from
our framework but it’s pretty stand alone.

This way I can embed my own stuff alongside activemq but you can also
reboot it.

We have like 15 daemons anyway and that code is pretty reliable so maybe
this would be harder for others.


> With that being said, I'd be really interested in the ability to expose a
> RESTful API around the broker so you can get JMX stats faster than JMX will
> allow, because we've been hampered by how slow JMX is.


Jolokia is looking pretty good so far.  I know there was discussion about
not embedding hawt.io by default in ActiveMQ but maybe that wasn’t the best
strategy.

I’m not sure anyone actually uses the current web UI but maybe I’m wrong.


> We're trying to
> poll dozens/hundreds of destination/consumer/producer MBeans every
> second to log out status (to help us look for sources of lag within the
> system), and JMX is completely unable to keep up, so we'd love a better
> option.


Jolokia is looking decent.  I didn’t test it in our environment because for
one of our uses I only needed two fields and I might even decide to go with
a binary protocol. I want to fetch it rapidly… like once every 500ms so I
can quickly see if any of our queues have a message and then start
subscribing to them.


> I'm not sure if that would be done by embedding ActiveMQ within an
> app that provides only that RESTful API or by rolling the API into the core
> broker executable, but if you're still thinking of open-sourcing that code
> (whether as part of the ActiveMQ project or as a companion project), we'd
> be interested in using it.
>

Ug.  I’m totally fine with OSSing things but the problem we have now is
that our app is very modular, but we’re not easily able to break off
sub-modules and publish them because git submodules are evil and broken and
just don’t work.

We git a lot of rapid iteration by having all the same code in the same
repository and just rapidly iterating.  the problem is that it means we
can’t easily contribute back to Open Source.

I will need a way to fix this in the future though. That clearly doesn’t
scale.


--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>