DOS attack on activemq setup

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

DOS attack on activemq setup

xabhi
Hi,

I was thinking about ways in which I cause DOS attack on activemq and how to prevent it.

I can bring the setup down by:
1. creating large number of connections - restrict based on connectionID?
2. large number of destinations
3. large number of subscriptions, consumers, producers, wildcard subscriptions etc - restrict wildcard subscription, limit no of consumer/producer?
4. Sending large number of persistent/non-persistent messages with huge sizes - limit msgsize that can be sent?

I don't know how to implement each of them and would like to get ActiveMQ community's thought on how to prevent these scenarios (either by hacking into/enriching activemq code - Plugins ?). What are other ways to create a DOS attack on activemq?

I know ActiveMQ provides basic authentication/authorization (username/password) to restrict some of these cases like authorization policy for destinations based on user name, groups.

What I am talking about is an unintentional DOS attack- where an legitimate application/client goes berserk to bug in code etc. and creates large number of connections or does a wildcard subscription and start receiving all messages etc.

I would like to get thought on how to prevent each of the cases I pointed before.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

Hadrian Zbarcea
Are you thinking about something like the producer flow control [1]?

Hadrian

[1] http://activemq.apache.org/producer-flow-control.html

On 04/09/2015 07:01 AM, xabhi wrote:

> Hi,
>
> I was thinking about ways in which I cause DOS attack on activemq and how to
> prevent it.
>
> I can bring the setup down by:
> 1. creating large number of connections - restrict based on connectionID?
> 2. large number of destinations
> 3. large number of subscriptions, consumers, producers, wildcard
> subscriptions etc - restrict wildcard subscription, limit no of
> consumer/producer?
> 4. Sending large number of persistent/non-persistent messages with huge
> sizes - limit msgsize that can be sent?
>
> I don't know how to implement each of them and would like to get ActiveMQ
> community's thought on how to prevent these scenarios (either by hacking
> into/enriching activemq code - Plugins ?). What are other ways to create a
> DOS attack on activemq?
>
> I know ActiveMQ provides basic authentication/authorization
> (username/password) to restrict some of these cases like authorization
> policy for destinations based on user name, groups.
>
> What I am talking about is an unintentional DOS attack- where an legitimate
> application/client goes berserk to bug in code etc. and creates large number
> of connections or does a wildcard subscription and start receiving all
> messages etc.
>
> I would like to get thought on how to prevent each of the cases I pointed
> before.
>
> Thanks,
> Abhi
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

gtully
In reply to this post by xabhi
some work has been done in this area.

there are connection limits (on a transport connector) and message
size limits on openwire.

there are currently no destination or producer/consumer number limits.
Flow control helps but in a very dynamic environment it is difficult
to get appropriate limits that won't negatively impact some
destinations over others.

something like a wildcard subscription that drains queues is possible.

the authorization helps, but if you have authorisation you can cause havok.

The place to address this issues is in a plugin and the key would be
identifying the threats that make sense to bundle together so that
there can be plugins that address scenarios.

recently i added the option to disable durable subs for folks that
want to exclusively use virtual topics in a high through put env but
in the main, the core is not the place for these checks.

In a plugin you get an opportunity to track and put limits on all
aspects of the broker. It will take a bunch of iterations over
usecases to determine what set of limits make sense but a good start
point would be to be able to limit everything and to be able to
control composite and wildcard destinations with some sort of regular
expression or something.

hope this helps ;-)



On 9 April 2015 at 12:01, xabhi <[hidden email]> wrote:

> Hi,
>
> I was thinking about ways in which I cause DOS attack on activemq and how to
> prevent it.
>
> I can bring the setup down by:
> 1. creating large number of connections - restrict based on connectionID?
> 2. large number of destinations
> 3. large number of subscriptions, consumers, producers, wildcard
> subscriptions etc - restrict wildcard subscription, limit no of
> consumer/producer?
> 4. Sending large number of persistent/non-persistent messages with huge
> sizes - limit msgsize that can be sent?
>
> I don't know how to implement each of them and would like to get ActiveMQ
> community's thought on how to prevent these scenarios (either by hacking
> into/enriching activemq code - Plugins ?). What are other ways to create a
> DOS attack on activemq?
>
> I know ActiveMQ provides basic authentication/authorization
> (username/password) to restrict some of these cases like authorization
> policy for destinations based on user name, groups.
>
> What I am talking about is an unintentional DOS attack- where an legitimate
> application/client goes berserk to bug in code etc. and creates large number
> of connections or does a wildcard subscription and start receiving all
> messages etc.
>
> I would like to get thought on how to prevent each of the cases I pointed
> before.
>
> Thanks,
> Abhi
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
Thanks gary for the insights here.

>>there are connection limits (on a transport connector) and message size limits on openwire.
If I specify these options, are these checked for every new connection and messages that arrive at broker.
I am asking this because my ActiveMQ setup recieves around 40 million messages throughout the day and so do you think a check on message size for every message can create a performance problem here??

How much overhead do you think a plugin of this sort would add on normal broker functioning?

My concern is while attempting to secure my broker from unintentional DOS kind of attack, I don't want to overdo these things and degrade normal functioning as well :)

Thanks again for all the help.
-Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

rajdavies


10 April 2015 07:12
Thanks gary for the insights here.

limits on openwire.
If I specify these options, are these checked for every new connection and
messages that arrive at broker.
Yes - there are also checks on the connector for max number of message consumers and producers.
I am asking this because my ActiveMQ setup recieves around 40 million
messages throughout the day and so do you think a check on message size for
every message can create a performance problem here??
No - its so tiny you wouldn't be able to measure its impact against everything else going on in the JVM, let alone the ActiveMQ broker

How much overhead do you think a plugin of this sort would add on normal
broker functioning?
The checks already built in (max connections,consumers,producers, message size) are always used anyway. If you write your own broker plugin, as a guess the I would think the over head would be minimal - but you'd need to benchmark it - YMMV.

My concern is while attempting to secure my broker from unintentional DOS
kind of attack, I don't want to overdo these things and degrade normal
functioning as well :)

Thanks again for all the help.
-Abhi




--
View this message in context: http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694683.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
9 April 2015 12:55
some work has been done in this area.

there are connection limits (on a transport connector) and message
size limits on openwire.

there are currently no destination or producer/consumer number limits.
Flow control helps but in a very dynamic environment it is difficult
to get appropriate limits that won't negatively impact some
destinations over others.

something like a wildcard subscription that drains queues is possible.

the authorization helps, but if you have authorisation you can cause havok.

The place to address this issues is in a plugin and the key would be
identifying the threats that make sense to bundle together so that
there can be plugins that address scenarios.

recently i added the option to disable durable subs for folks that
want to exclusively use virtual topics in a high through put env but
in the main, the core is not the place for these checks.

In a plugin you get an opportunity to track and put limits on all
aspects of the broker. It will take a bunch of iterations over
usecases to determine what set of limits make sense but a good start
point would be to be able to limit everything and to be able to
control composite and wildcard destinations with some sort of regular
expression or something.

hope this helps ;-)


9 April 2015 12:01
Hi,

I was thinking about ways in which I cause DOS attack on activemq and how to
prevent it.

I can bring the setup down by:
1. creating large number of connections - restrict based on connectionID?
2. large number of destinations
3. large number of subscriptions, consumers, producers, wildcard
subscriptions etc - restrict wildcard subscription, limit no of
consumer/producer?
4. Sending large number of persistent/non-persistent messages with huge
sizes - limit msgsize that can be sent?

I don't know how to implement each of them and would like to get ActiveMQ
community's thought on how to prevent these scenarios (either by hacking
into/enriching activemq code - Plugins ?). What are other ways to create a
DOS attack on activemq?

I know ActiveMQ provides basic authentication/authorization
(username/password) to restrict some of these cases like authorization
policy for destinations based on user name, groups.

What I am talking about is an unintentional DOS attack- where an legitimate
application/client goes berserk to bug in code etc. and creates large number
of connections or does a wildcard subscription and start receiving all
messages etc.

I would like to get thought on how to prevent each of the cases I pointed
before.

Thanks,
Abhi




--
View this message in context: http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

James Carman
In reply to this post by xabhi
Do you expose your broker to the outside world?

On Thursday, April 9, 2015, xabhi <[hidden email]> wrote:

> Hi,
>
> I was thinking about ways in which I cause DOS attack on activemq and how
> to
> prevent it.
>
> I can bring the setup down by:
> 1. creating large number of connections - restrict based on connectionID?
> 2. large number of destinations
> 3. large number of subscriptions, consumers, producers, wildcard
> subscriptions etc - restrict wildcard subscription, limit no of
> consumer/producer?
> 4. Sending large number of persistent/non-persistent messages with huge
> sizes - limit msgsize that can be sent?
>
> I don't know how to implement each of them and would like to get ActiveMQ
> community's thought on how to prevent these scenarios (either by hacking
> into/enriching activemq code - Plugins ?). What are other ways to create a
> DOS attack on activemq?
>
> I know ActiveMQ provides basic authentication/authorization
> (username/password) to restrict some of these cases like authorization
> policy for destinations based on user name, groups.
>
> What I am talking about is an unintentional DOS attack- where an legitimate
> application/client goes berserk to bug in code etc. and creates large
> number
> of connections or does a wildcard subscription and start receiving all
> messages etc.
>
> I would like to get thought on how to prevent each of the cases I pointed
> before.
>
> Thanks,
> Abhi
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
This post was updated on .
Thanks everyone for your responses.
I would like to know if there are any more ways to disrupt broker setup besides the ones I enumerated in my original post.

Is there any sample Plugin examples which I can refer to?

Hi James,
>>Do you expose your broker to the outside world?
My ActiveMQ setup is not exposed to outside world. I am paranoid about these aspects because it is a very critical service.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

Tim Bain
If you're using persistent messages in ActiveMQ to store messages to the
disk, and the user can lunch a DOS attack against the disk through another
application, the result will be a DOS attack against ActiveMQ.  Obviously
the ability to do this depends on what applications are located on the same
disk and the attacker's access to those applications.
On Apr 10, 2015 7:39 AM, "xabhi" <[hidden email]> wrote:

> Thanks everyone for your responses.
> I would like to know if there are any more ways to disrupt broker setup
> besides the ones I enumerated in my original post.
>
> Hi James,
> >>Do you expose your broker to the outside world?
> My ActiveMQ setup is not exposed to outside world. I am paranoid about
> these
> aspects because it is a very critical service.
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694695.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
In reply to this post by rajdavies
Hi,

>>Yes - there are also checks on the connector for max number of message consumers and producers.
I couldn't find the wiki which shows how to specify max number of message consumers and producers on transport connector. I found how to specify max number of connections here http://activemq.apache.org/tcp-transport-reference.html.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

Jakub Scholz
Maximal number of connections is a nice feature. But often what you really
need is the ability to limit the number of connections "per user", not per
connector. The overall maximal number of connections per connector would
protect your broker from crashing because it runs out of memory etc. But
the second per user limit can also make sure that one rogue application
doesn't use the overall limit for its self - and cause that other
applications will not be able to connect.

Jakub

On Fri, Apr 10, 2015 at 4:06 PM, xabhi <[hidden email]> wrote:

> Hi,
>
> >>Yes - there are also checks on the connector for max number of message
> consumers and producers.
> I couldn't find the wiki which shows how to specify max number of message
> consumers and producers on transport connector. I found how to specify max
> number of connections here
> http://activemq.apache.org/tcp-transport-reference.html
> <http://activemq.apache.org/tcp-transport-reference.html>  .
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694702.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

James Carman
In reply to this post by xabhi
If it is not exposed to the outside world, then are you afraid of an
unintentional "attack" from within or do you think someone intentionally
would try to bring it down?

On Friday, April 10, 2015, xabhi <[hidden email]> wrote:

> Thanks everyone for your responses.
> I would like to know if there are any more ways to disrupt broker setup
> besides the ones I enumerated in my original post.
>
> Hi James,
> >>Do you expose your broker to the outside world?
> My ActiveMQ setup is not exposed to outside world. I am paranoid about
> these
> aspects because it is a very critical service.
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694695.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

Tim Bain
I believe Abhi said previously that he's worried less about intentional
attacks and more about what misbehaving or poorly-written programs could do
that would kill the broker.

On Fri, Apr 10, 2015 at 9:47 AM, James Carman <[hidden email]>
wrote:

> If it is not exposed to the outside world, then are you afraid of an
> unintentional "attack" from within or do you think someone intentionally
> would try to bring it down?
>
> On Friday, April 10, 2015, xabhi <[hidden email]> wrote:
>
> > Thanks everyone for your responses.
> > I would like to know if there are any more ways to disrupt broker setup
> > besides the ones I enumerated in my original post.
> >
> > Hi James,
> > >>Do you expose your broker to the outside world?
> > My ActiveMQ setup is not exposed to outside world. I am paranoid about
> > these
> > aspects because it is a very critical service.
> >
> > Thanks,
> > Abhi
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694695.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

James Carman
I wouldn't characterize it as an "attack", then.  Anyway, just trying to
understand.  Thanks

On Friday, April 10, 2015, Tim Bain <[hidden email]> wrote:

> I believe Abhi said previously that he's worried less about intentional
> attacks and more about what misbehaving or poorly-written programs could do
> that would kill the broker.
>
> On Fri, Apr 10, 2015 at 9:47 AM, James Carman <[hidden email]
> <javascript:;>>
> wrote:
>
> > If it is not exposed to the outside world, then are you afraid of an
> > unintentional "attack" from within or do you think someone intentionally
> > would try to bring it down?
> >
> > On Friday, April 10, 2015, xabhi <[hidden email] <javascript:;>>
> wrote:
> >
> > > Thanks everyone for your responses.
> > > I would like to know if there are any more ways to disrupt broker setup
> > > besides the ones I enumerated in my original post.
> > >
> > > Hi James,
> > > >>Do you expose your broker to the outside world?
> > > My ActiveMQ setup is not exposed to outside world. I am paranoid about
> > > these
> > > aspects because it is a very critical service.
> > >
> > > Thanks,
> > > Abhi
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694695.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
In reply to this post by Tim Bain
Hi Tim, rajdavies
Could you please point me to documentation on how to restrict no. of consumers/producers on transport connector. I couldn't find them on activemq wiki.

All i could find was this page http://activemq.apache.org/tcp-transport-reference.html

Thanks
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

rajdavies
Can't find it in the docs - will fix when I get a chance;
Basically the properties on the TransportConnector are:

maximumConsumersAllowedPerConnection and
maximumProducersAllowedPerConnection

You can set them on the TransportConnector (used by the Broker)

e.g. url = "tcp://localhost:61616?maximumProducersAllowedPerConnection=200";

On 11 April 2015 at 09:52, xabhi <[hidden email]> wrote:

> Hi Tim, rajdavies
> Could you please point me to documentation on how to restrict no. of
> consumers/producers on transport connector. I couldn't find them on
> activemq
> wiki.
>
> All i could find was this page
> http://activemq.apache.org/tcp-transport-reference.html
> <http://activemq.apache.org/tcp-transport-reference.html>
>
> Thanks
> Abhi
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694782.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
I just thought of another case where broker can be crashed. This is a case where a rogue app sends copious amount of persistent messages with no consumer and causes disk space to fill up making the broker unavailable.

I am using multiKahaDB in my setup and have grouped it based on destination patterns (a logical grouping of related destinations). But the kind of scenario I have described can be prevented from affecting other applications If I can put size limit on each of multiKahaDB persistent adapter separately otherwise one multiKahaDB can use entire remaining space for itself. That way broker won't run out of space for all destinations but only a group of destinations thus preventing  the crash of entire setup.

Is there any KahaDB configuration already present which will help in this case?

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

gtully
Abhi,
that is a problem. The best we have at the moment is via
https://issues.apache.org/jira/browse/AMQ-2668

Having per adapter limits respected by the broker in the mkahadb case
would be a nice enhancement.


On 13 April 2015 at 15:10, xabhi <[hidden email]> wrote:

> I just thought of another case where broker can be crashed. This is a case
> where a rogue app sends copious amount of persistent messages with no
> consumer and causes disk space to fill up making the broker unavailable.
>
> I am using multiKahaDB in my setup and have grouped it based on destination
> patterns (a logical grouping of related destinations). But the kind of
> scenario I have described can be prevented from affecting other applications
> If I can put size limit on each of multiKahaDB persistent adapter separately
> otherwise one multiKahaDB can use entire remaining space for itself. That
> way broker won't run out of space for all destinations but only a group of
> destinations thus preventing  the crash of entire setup.
>
> Is there any KahaDB configuration already present which will help in this
> case?
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694820.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
Thanks gary!

I have logged a JIRA request for this enhancement - https://issues.apache.org/jira/browse/AMQ-5720
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

xabhi
In reply to this post by gtully
Hi gary,
In http://activemq.apache.org/per-destination-policies.html the memoryLimit option refers to only memoryUsage part of systemUsage as per my understanding and there is no way to configure the storeUsage compoment as of now. Just want to confirm my understanding here before I start using this option.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: DOS attack on activemq setup

gtully
correct, there is only storeUsageHighWaterMark

On 14 April 2015 at 12:01, xabhi <[hidden email]> wrote:

> Hi gary,
> In  http://activemq.apache.org/per-destination-policies.html
> <http://activemq.apache.org/per-destination-policies.html>   the
> *memoryLimit* option refers to only memoryUsage part of systemUsage as per
> my understanding and there is no way to configure the storeUsage compoment
> as of now. Just want to confirm my understanding here before I start using
> this option.
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DOS-attack-on-activemq-setup-tp4694598p4694869.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
12