nginx limit_req and limit_conn not working to prevent DoS attack

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

nginx limit_req and limit_conn not working to prevent DoS attack

blason
Hi All,

I am using nginx in our products. When I run goldeneye DoS attack script
against nginx, it is not able to defend against the attack and normal users
getting impacted.  

python goldeneye.py http://<ipaddress> -w 5 -s 10000 -m random -d

we are using below nginx limit_req options but didnt help. The nginx
documentation says that, these options are used to limit the request rating
limit per key. below is some sample configuration that we tried. The problem
is, when we use these nginx options, it still keeps nginx  busy responding
with 503 or some other error code for all those requests beyond the rate
limit . Hence any genuine user when trying to access webserver during the
attack time, not getting chance to access our server and timing out or
getting 500 error.

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;

    ...

    server {

        limit_req zone=one ;
        ...

        location /sampleurl/ {
           
        }

(Note: also tried limit_conn options and behavior is same).



Why should nginx respond back with any error code rather it should drop
connections !! otherwise it can't protect itself against any DoS attack.
please share the thoughts.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275796#msg-275796

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

Zhang Chao
Hi!

I don’t think just drop the connection is a good idea, client will never know what happens on the server end.
However, the code 444 may help you, nginx just close the connection in this case.

On 2 August 2017 at 09:30:01, Phani Sreenivasa Prasad ([hidden email]) wrote:

Hi All,

I am using nginx in our products. When I run goldeneye DoS attack script
against nginx, it is not able to defend against the attack and normal users
getting impacted.

python goldeneye.py http://<ipaddress> -w 5 -s 10000 -m random -d

we are using below nginx limit_req options but didnt help. The nginx
documentation says that, these options are used to limit the request rating
limit per key. below is some sample configuration that we tried. The problem
is, when we use these nginx options, it still keeps nginx busy responding
with 503 or some other error code for all those requests beyond the rate
limit . Hence any genuine user when trying to access webserver during the
attack time, not getting chance to access our server and timing out or
getting 500 error.

http {
limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;

...

server {

limit_req zone=one ;
...

location /sampleurl/ {

}

(Note: also tried limit_conn options and behavior is same).



Why should nginx respond back with any error code rather it should drop
connections !! otherwise it can't protect itself against any DoS attack.
please share the thoughts.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275796#msg-275796

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

blason
I assume it would help dropping connections . since we are setting rate
limit per ip and any client IP which is suspicious by sending requests in
bulk(lets say 10000 connections/requests), it makes sense to not to accept
connections/requests from that IP.

Thoughts ??

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275798#msg-275798

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

Anoop Alias
You can use an external tool to parse Nginx error log and block the IP in iptables/netfilter 

On Wed, Aug 2, 2017 at 7:43 AM, Phani Sreenivasa Prasad <[hidden email]> wrote:
I assume it would help dropping connections . since we are setting rate
limit per ip and any client IP which is suspicious by sending requests in
bulk(lets say 10000 connections/requests), it makes sense to not to accept
connections/requests from that IP.

Thoughts ??

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275798#msg-275798

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

lists@lazygranch.com
I don't know about iptables, but you can limit port 80 and 443 with ipfw. I run mine at 10 per IP. I've had corporations behind NAT trigger lesser limits. My point being you don't have to parse the log (swatch). 



Sent: August 1, 2017 8:26 PM
Reply-to: [hidden email]
Subject: Re: nginx limit_req and limit_conn not working to prevent DoS attack

You can use an external tool to parse Nginx error log and block the IP in iptables/netfilter 

On Wed, Aug 2, 2017 at 7:43 AM, Phani Sreenivasa Prasad <[hidden email]> wrote:
I assume it would help dropping connections . since we are setting rate
limit per ip and any client IP which is suspicious by sending requests in
bulk(lets say 10000 connections/requests), it makes sense to not to accept
connections/requests from that IP.

Thoughts ??

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275798#msg-275798

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

blason
In reply to this post by Anoop Alias
Yes. Firewall would be another option. But before to that, i would like to
try out all options at nginx level if one or other would resolve the issue
at nginx layer itself.

cant we put accept() filters? or
how the deny option works? can we use deny option to not to accept any new
connections if number of connections already exceeds max limit from a client
IP.?
are there any third party modules available for nginx to embed firewall
functionality? something reliable !!

My objective is, using limit_conn directive, when number of connections
exceeding limit, instead of sending 503, or 444, just do not accept any new
connections from that specific IP only(if a client is opening 10000
connections at a time, it should be fine to not accept connections from that
IP citing the reason that it could be malicious).

Thoughts !!

Thanks.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275801#msg-275801

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

lists@lazygranch.com
The trouble is nginx does a fair amount of work before blocking the IP address, unless things have changed. My recollection is it parses the whole request. Obviously it doesn't send any data. So you are better off blocking with the firewall.

You do need to know your audience. Something related to a university could generate a number of simultaneous users behind one IP. In my case Boeing triggered the limit.


  Original Message  
From: [hidden email]
Sent: August 1, 2017 9:08 PM
To: [hidden email]
Reply-to: [hidden email]
Subject: Re: nginx limit_req and limit_conn not working to prevent DoS attack

Yes. Firewall would be another option. But before to that, i would like to
try out all options at nginx level if one or other would resolve the issue
at nginx layer itself.

cant we put accept() filters? or
how the deny option works? can we use deny option to not to accept any new
connections if number of connections already exceeds max limit from a client
IP.?
are there any third party modules available for nginx to embed firewall
functionality? something reliable !!

My objective is, using limit_conn directive, when number of connections
exceeding limit, instead of sending 503, or 444, just do not accept any new
connections from that specific IP only(if a client is opening 10000
connections at a time, it should be fine to not accept connections from that
IP citing the reason that it could be malicious).

Thoughts !!

Thanks.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275801#msg-275801

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

nginx mailing list
In reply to this post by blason
The original confusion came from the fact you slided away from the basic mantra of the Unix philosophy stating 'Make each program do one thing well'.

nginx is a Web server, which generalized itself into a stream server. It serves content and manages access (protects it).
What you are trying to achieve is turning nginx into a firewall, which it is not.
A content server does not simply cut connections. It behaves and responds to requests. That is standard.
All you can do at the connection level is limiting their number (cf. limit_conn).

It has been suggested you used iptables, as it is a firewall. At the software level, I would rather recommend nftables.
Some log analyzers could help you make the interface between a content server and a software firewall, such as fail2ban.

You could also go for hardware (D)DoS protection, depending on the scale of your needs.

​There is nothing to be surprised of, the product you are using merely doing the job. it has been made for​
---
B. R.

On Wed, Aug 2, 2017 at 6:27 AM, Gary Sellani <[hidden email]> wrote:
The trouble is nginx does a fair amount of work before blocking the IP address, unless things have changed. My recollection is it parses the whole request. Obviously it doesn't send any data. So you are better off blocking with the firewall.

You do need to know your audience. Something related to a university could generate a number of simultaneous users behind one IP. In my case Boeing triggered the limit.


  Original Message  
From: [hidden email]
Sent: August 1, 2017 9:08 PM
To: [hidden email]
Reply-to: [hidden email]
Subject: Re: nginx limit_req and limit_conn not working to prevent DoS attack

Yes. Firewall would be another option. But before to that, i would like to
try out all options at nginx level if one or other would resolve the issue
at nginx layer itself.

cant we put accept() filters? or
how the deny option works? can we use deny option to not to accept any new
connections if number of connections already exceeds max limit from a client
IP.?
are there any third party modules available for nginx to embed firewall
functionality? something reliable !!

My objective is, using limit_conn directive, when number of connections
exceeding limit, instead of sending 503, or 444, just do not accept any new
connections from that specific IP only(if a client is opening 10000
connections at a time, it should be fine to not accept connections from that
IP citing the reason that it could be malicious).

Thoughts !!

Thanks.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275796,275801#msg-275801

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: nginx limit_req and limit_conn not working to prevent DoS attack

Valery Kholodkov
I think the confusion is still there and it is in the term 'firewall'.

While nginx is no good for level 3 firewall, also known as netfilter,
it's perfect for application level firewall and you already mentioned it
by saying that 'it manages access (protects it)'.

People turn nginx into application level firewall, no problem. The
question is rather if you want to pay the price of scrubbing the request
in the userland and not philosophical considerations.

Yet at certain scale and confidence level, indeed a
vertically-integrated solution might be more adequate.

On 02-08-17 10:15, B.R. via nginx wrote:

> The original confusion came from the fact you slided away from the basic
> mantra of the Unix philosophy stating 'Make each program do one thing well'.
>
> nginx is a Web server, which generalized itself into a stream server. It
> serves content and manages access (protects it).
> What you are trying to achieve is turning nginx into a firewall, which
> it is not.
> A content server does not simply cut connections. It behaves and
> responds to requests. That is standard.
> All you can do at the connection level is limiting their number (cf.
> limit_conn).
>
> It has been suggested you used iptables, as it is a firewall. At the
> software level, I would rather recommend nftables.
> Some log analyzers could help you make the interface between a content
> server and a software firewall, such as fail2ban.
>
> You could also go for hardware (D)DoS protection, depending on the scale
> of your needs.
>
> ​There is nothing to be surprised of, the product you are using merely
> doing the job. it has been made for​
> ---
> *B. R.*
>
> On Wed, Aug 2, 2017 at 6:27 AM, Gary Sellani <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     The trouble is nginx does a fair amount of work before blocking the
>     IP address, unless things have changed. My recollection is it parses
>     the whole request. Obviously it doesn't send any data. So you are
>     better off blocking with the firewall.
>
>     You do need to know your audience. Something related to a university
>     could generate a number of simultaneous users behind one IP. In my
>     case Boeing triggered the limit.
>
>
>       Original Message
>     From: [hidden email] <mailto:[hidden email]>
>     Sent: August 1, 2017 9:08 PM
>     To: [hidden email] <mailto:[hidden email]>
>     Reply-to: [hidden email] <mailto:[hidden email]>
>     Subject: Re: nginx limit_req and limit_conn not working to prevent
>     DoS attack
>
>     Yes. Firewall would be another option. But before to that, i would
>     like to
>     try out all options at nginx level if one or other would resolve the
>     issue
>     at nginx layer itself.
>
>     cant we put accept() filters? or
>     how the deny option works? can we use deny option to not to accept
>     any new
>     connections if number of connections already exceeds max limit from
>     a client
>     IP.?
>     are there any third party modules available for nginx to embed firewall
>     functionality? something reliable !!
>
>     My objective is, using limit_conn directive, when number of connections
>     exceeding limit, instead of sending 503, or 444, just do not accept
>     any new
>     connections from that specific IP only(if a client is opening 10000
>     connections at a time, it should be fine to not accept connections
>     from that
>     IP citing the reason that it could be malicious).
>
>     Thoughts !!
>
>     Thanks.
>
>     Posted at Nginx Forum:
>     https://forum.nginx.org/read.php?2,275796,275801#msg-275801
>     <https://forum.nginx.org/read.php?2,275796,275801#msg-275801>
>
>     _______________________________________________
>     nginx mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://mailman.nginx.org/mailman/listinfo/nginx
>     <http://mailman.nginx.org/mailman/listinfo/nginx>
>     _______________________________________________
>     nginx mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://mailman.nginx.org/mailman/listinfo/nginx
>     <http://mailman.nginx.org/mailman/listinfo/nginx>
>
>
>
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Loading...