nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Lukas
Dear all

Fascinated by nginx, I attempted to integrate it with modsecurity.

Unfortunately, ever when modsecurity is enabled, nginx reports a
sefault in sysmessages.

Searching the web did not reveal any solution, i.e. I switched off
SecAudit* and even started modsecurity without rules -- it continued
crashing.

Thank you for any hint on solving this issue.

Please find next information related to my setup including some logs.

wbr, Lukas

==

My current setup:

Platform: Linux/4.3.3 running on Debian/wheezy

nginx: self-compiled from sources according to
https://blog.stickleback.dk/nginx-modsec-on-ubuntu-14-04-lts/

modsecurity: installed and configured according to
https://www.howtoforge.com/tutorial/install-nginx-with-mod_security-on-ubuntu-15-04/

Relevant Logs:

$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.9.9
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1e 11 Feb 2013
TLS SNI support enabled
configure arguments: --user=www-data --group=www-data --with-pcre-jit
  --with-ipv6 --with-http_ssl_module
  --add-module=../modsecurity-2.9.0/nginx/modsecurity
  --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid
  --error-log-path=/var/log/nginx/error.log
  --http-log-path=/var/log/nginx/access.log

$ tail error.log
2016/01/10 13:13:34 [notice] 10256#0: ModSecurity: LIBXML compiled version="2.8.0"
2016/01/10 13:13:34 [notice] 10256#0: ModSecurity: Status engine is
    currently disabled, enable it by set SecStatusEngine to On.
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity for nginx
    (STABLE)/2.9.0 (http://www.modsecurity.org/) configured.
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: APR compiled
    version="1.4.6"; loaded version="1.4.6"
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: PCRE compiled
    version="8.30 "; loaded version="8.30 2012-02-04"
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: LIBXML compiled version="2.8.0"
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: Status engine is
    currently disabled, enable it by set SecStatusEngine to On.
2016/01/10 13:13:38 [alert] 10261#0: worker process 10267 exited on signal 11
2016/01/10 13:13:38 [alert] 10261#0: worker process 10264 exited on signal 11
2016/01/10 13:13:38 [alert] 10261#0: worker process 10265 exited on signal 11

$ dmesg
[605432.202671] nginx[10267]: segfault at 70 ip 08093ba1 sp bfc9a7c0 error 4 in nginx[8048000+123000]
[605432.385414] nginx[10264]: segfault at 70 ip 08093ba1 sp bfc9a7c0 error 4 in nginx[8048000+123000]
[605432.409089] nginx[10265]: segfault at 70 ip 08093ba1 sp bfc9a7c0 error 4 in nginx[8048000+123000]

--
Lukas Ruf       <http://www.lpr.ch> | Ad Personam
Consecom  <http://www.consecom.com> | Ad Laborem

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Rainer Duffner

> Am 10.01.2016 um 14:39 schrieb Lukas <[hidden email]>:
>
> Dear all
>
> Fascinated by nginx, I attempted to integrate it with modsecurity.
>
> Unfortunately, ever when modsecurity is enabled, nginx reports a
> sefault in sysmessages.
>
> Searching the web did not reveal any solution, i.e. I switched off
> SecAudit* and even started modsecurity without rules -- it continued
> crashing.
>
> Thank you for any hint on solving this issue.
>
> Please find next information related to my setup including some logs.



By chance, I tried to get this to work just yesterday and also got only SIGSEGV from it.
(nginx 1.8, FreeBSD 10.1-amd64, ap22-mod_security-2.9.0, all from my own repository)


I found this:

https://github.com/SpiderLabs/ModSecurity/issues/839

So, you need to set

proxy_force_ranges on;

in the location you want to protect with mod_security.

It didn’t segfault any more after this - but I haven’t had time to check how well it actually works.




Rainer
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Felipe Zimmerle
In reply to this post by Lukas
Hi Lukas,

You may want to use the ModSecurity's nginx_refactoring branch instead of the master branch. Here is the link to the branch:


Br.,
Felipe Zimmerle
Lead dev for ModSecurity


On Sun, Jan 10, 2016 at 10:39 AM Lukas <[hidden email]> wrote:
Dear all

Fascinated by nginx, I attempted to integrate it with modsecurity.

Unfortunately, ever when modsecurity is enabled, nginx reports a
sefault in sysmessages.

Searching the web did not reveal any solution, i.e. I switched off
SecAudit* and even started modsecurity without rules -- it continued
crashing.

Thank you for any hint on solving this issue.

Please find next information related to my setup including some logs.

wbr, Lukas

==

My current setup:

Platform: Linux/4.3.3 running on Debian/wheezy

nginx: self-compiled from sources according to
https://blog.stickleback.dk/nginx-modsec-on-ubuntu-14-04-lts/

modsecurity: installed and configured according to
https://www.howtoforge.com/tutorial/install-nginx-with-mod_security-on-ubuntu-15-04/

Relevant Logs:

$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.9.9
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1e 11 Feb 2013
TLS SNI support enabled
configure arguments: --user=www-data --group=www-data --with-pcre-jit
  --with-ipv6 --with-http_ssl_module
  --add-module=../modsecurity-2.9.0/nginx/modsecurity
  --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid
  --error-log-path=/var/log/nginx/error.log
  --http-log-path=/var/log/nginx/access.log

$ tail error.log
2016/01/10 13:13:34 [notice] 10256#0: ModSecurity: LIBXML compiled version="2.8.0"
2016/01/10 13:13:34 [notice] 10256#0: ModSecurity: Status engine is
    currently disabled, enable it by set SecStatusEngine to On.
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity for nginx
    (STABLE)/2.9.0 (http://www.modsecurity.org/) configured.
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: APR compiled
    version="1.4.6"; loaded version="1.4.6"
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: PCRE compiled
    version="8.30 "; loaded version="8.30 2012-02-04"
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: LIBXML compiled version="2.8.0"
2016/01/10 13:13:35 [notice] 10260#0: ModSecurity: Status engine is
    currently disabled, enable it by set SecStatusEngine to On.
2016/01/10 13:13:38 [alert] 10261#0: worker process 10267 exited on signal 11
2016/01/10 13:13:38 [alert] 10261#0: worker process 10264 exited on signal 11
2016/01/10 13:13:38 [alert] 10261#0: worker process 10265 exited on signal 11

$ dmesg
[605432.202671] nginx[10267]: segfault at 70 ip 08093ba1 sp bfc9a7c0 error 4 in nginx[8048000+123000]
[605432.385414] nginx[10264]: segfault at 70 ip 08093ba1 sp bfc9a7c0 error 4 in nginx[8048000+123000]
[605432.409089] nginx[10265]: segfault at 70 ip 08093ba1 sp bfc9a7c0 error 4 in nginx[8048000+123000]

--
Lukas Ruf       <http://www.lpr.ch> | Ad Personam
Consecom  <http://www.consecom.com> | Ad Laborem

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Lukas
In reply to this post by Rainer Duffner
> Rainer Duffner <[hidden email]> [2016-01-10 14:46]:
>
>
> > Am 10.01.2016 um 14:39 schrieb Lukas <[hidden email]>:
> >
> > Unfortunately, ever when modsecurity is enabled, nginx reports a
> > sefault in sysmessages.
> >
> > Searching the web did not reveal any solution, i.e. I switched off
> > SecAudit* and even started modsecurity without rules -- it continued
> > crashing.
> >
>
>
> By chance, I tried to get this to work just yesterday and also got only SIGSEGV from it.
> (nginx 1.8, FreeBSD 10.1-amd64, ap22-mod_security-2.9.0, all from my own repository)
>
>
> I found this:
>
> https://github.com/SpiderLabs/ModSecurity/issues/839
>
> So, you need to set
>
> proxy_force_ranges on;
>
> in the location you want to protect with mod_security.
>
> It didn???t segfault any more after this - but I haven???t had time to check how well it actually works.
>

Thanks for your hint, Rainer.  Now it has not crahed anymore.....

wbr
Lukas

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Lukas
In reply to this post by Felipe Zimmerle
Hi Felipe

> Felipe Zimmerle <[hidden email]> [2016-01-10 14:49]:
>
> You may want to use the ModSecurity's nginx_refactoring branch instead of
> the master branch. Here is the link to the branch:
>
> https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring
>

Thanks for your hint.

I found that recommendation.  Since I also read that it would not be
fully compatible with OWASP/CRS I have not given it a try.

What is the situation regrading OWASP/CRS?

wbr
Lukas

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Felipe Zimmerle
Hi Lukas,

On Sun, Jan 10, 2016 at 11:05 AM Lukas <[hidden email]> wrote:
I found that recommendation.  Since I also read that it would not be
fully compatible with OWASP/CRS I have not given it a try.

What is the situation regrading OWASP/CRS?

Currently there are three different versions of ModSecurity for nginx:

- Version 2.9.0: That is the last released version, I think that is the one that you are using.
- nginx_refactoring: That version contains some fixes on the top of v2.9.0, but those fixes may lead to instabilities depending on your configuration.
- ModSecurity-connector: That is something that still under development and we have some work to do, to be exactly:


Only use the ModSecurity-connector if you understands well the ModSecurity rules and the consequences of the missing pieces. 

Further information about libModSecurity can be found here:

Br.,
Felipe.

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Lukas
Hi Felipe

> Felipe Zimmerle <[hidden email]> [2016-01-11 17:12]:
>
> On Sun, Jan 10, 2016 at 11:05 AM Lukas <[hidden email]> wrote:
>
> > I found that recommendation.  Since I also read that it would not be
> > fully compatible with OWASP/CRS I have not given it a try.
> >
> > What is the situation regrading OWASP/CRS?
> >
>
> Currently there are three different versions of ModSecurity for nginx:
>
> - Version 2.9.0: That is the last released version, I think that is the one
> that you are using.
> - nginx_refactoring: That version contains some fixes on the top of v2.9.0,
> but those fixes may lead to instabilities depending on your configuration.
> - ModSecurity-connector: That is something that still under development and
> we have some work to do, to be exactly:
>
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20documentation
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20features
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20operators
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20transformation
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20variables
>
> Only use the ModSecurity-connector if you understands well the ModSecurity
> rules and the consequences of the missing pieces.
>
> Further information about libModSecurity can be found here:
> http://blog.zimmerle.org/2016/01/an-overview-of-upcoming-libmodsecurity.html
> or:
> https://www.trustwave.com/Resources/SpiderLabs-Blog/An-Overview-of-the-Upcoming-libModSecurity/
>

Thanks for pointing this out.

What worries me a "little bit" is that nginx started crashing with an
Out-of-Memory Exception when ModSecurity 2.9.0 with OWASP/CRS was
activated.

Have others experienced similar problems?

Isn't there at least a run-time control in nginx that kills
subprocesses like ModSecurity as soon as they start overconsuming
resources/execution time?

Thanks.

wbr
Lukas

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Robert Paprocki
ModSecurity isn't a sub-process, it's compiled into the nginx binary and runs as part of the worker process(es). Nginx doesn't have a concept of spawning children in the manner you're referencing, so there's nothing to be monitored wrt. resource consumption. Any resource monitoring would be done by the kernel, and the target would be nginx itself.

If you're running into an OOM condition with the nginx worker process, it sounds like a leak within one of the modules (possible, but not definitely, ModSecurity, if it only happens when you load the OWASP CRS).



On Tue, Jan 19, 2016 at 3:10 PM, Lukas <[hidden email]> wrote:
Hi Felipe

> Felipe Zimmerle <[hidden email]> [2016-01-11 17:12]:
>
> On Sun, Jan 10, 2016 at 11:05 AM Lukas <[hidden email]> wrote:
>
> > I found that recommendation.  Since I also read that it would not be
> > fully compatible with OWASP/CRS I have not given it a try.
> >
> > What is the situation regrading OWASP/CRS?
> >
>
> Currently there are three different versions of ModSecurity for nginx:
>
> - Version 2.9.0: That is the last released version, I think that is the one
> that you are using.
> - nginx_refactoring: That version contains some fixes on the top of v2.9.0,
> but those fixes may lead to instabilities depending on your configuration.
> - ModSecurity-connector: That is something that still under development and
> we have some work to do, to be exactly:
>
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20documentation
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20features
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20operators
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20transformation
> https://github.com/SpiderLabs/ModSecurity/labels/libmodsec%20-%20missing%20variables
>
> Only use the ModSecurity-connector if you understands well the ModSecurity
> rules and the consequences of the missing pieces.
>
> Further information about libModSecurity can be found here:
> http://blog.zimmerle.org/2016/01/an-overview-of-upcoming-libmodsecurity.html
> or:
> https://www.trustwave.com/Resources/SpiderLabs-Blog/An-Overview-of-the-Upcoming-libModSecurity/
>

Thanks for pointing this out.

What worries me a "little bit" is that nginx started crashing with an
Out-of-Memory Exception when ModSecurity 2.9.0 with OWASP/CRS was
activated.

Have others experienced similar problems?

Isn't there at least a run-time control in nginx that kills
subprocesses like ModSecurity as soon as they start overconsuming
resources/execution time?

Thanks.

wbr
Lukas

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Lukas
Hi Robert

> Robert Paprocki <[hidden email]> [2016-01-20 00:14]:
>
> ModSecurity isn't a sub-process, it's compiled into the nginx binary and
> runs as part of the worker process(es). Nginx doesn't have a concept of
> spawning children in the manner you're referencing, so there's nothing to
> be monitored wrt. resource consumption. Any resource monitoring would be
> done by the kernel, and the target would be nginx itself.
>
Thanks for clarifying.

> If you're running into an OOM condition with the nginx worker process, it
> sounds like a leak within one of the modules (possible, but not definitely,
> ModSecurity, if it only happens when you load the OWASP CRS).
>

I have not had the time to test different variants yet.  The
proxy-setup, however, works perfectly fine with "ModSecurityEnabled
off;" but crashes otherwise.

My current config:

  server {
    listen 443 ssl;
    listen [::]:443 ssl;

    root /var/www/html;

    index index.html index.htm index.nginx-debian.html;

    server_name foobar;

    ssl on;
    ssl_certificate     crt.stack.pem;
    ssl_certificate_key key.pem;

    ssl_session_timeout 5m;
    ssl_session_cache  builtin:1000  shared:SSL:10m;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    location / {

      ModSecurityEnabled off;
      ModSecurityConfig modsecurity/modsecurity_crs_10_setup.conf;

      proxy_force_ranges on;
      proxy_set_header Host                $host;
      proxy_set_header X-Real-IP           $remote_addr;
      proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto   $scheme;

      proxy_pass      <a href="http://ip.ad.dr.ess:80;">http://ip.ad.dr.ess:80;
      proxy_redirect  http://ip.ad.dr.ess:80 https://$host$request_uri;

      client_max_body_size       10m;
      client_body_buffer_size    128k;

      client_body_temp_path      /var/cache/nginx/client_body_temp;

      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;

      proxy_buffer_size          4k;
      proxy_buffers              4 32k;
      proxy_busy_buffers_size    64k;
      proxy_temp_file_write_size 64k;

      proxy_temp_path            /var/cache/nginx/proxy_temp;

    }

    location ~ /\.ht {
      deny all;
    }

    access_log  /var/log/nginx/access.log upstreamlog;
    error_log   /var/log/nginx/error.log debug;
  }

Thanks.

wbr.
Lukas

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Lukas
In reply to this post by Lukas
Dear all

> Lukas <[hidden email]> [2016-01-10 14:39]:
>
> Fascinated by nginx, I attempted to integrate it with modsecurity.
>
> Unfortunately, ever when modsecurity is enabled, nginx reports a
> sefault in sysmessages.
>

I tried debugging the issue a bit further (from a user perspective)
with common web-page and CalDAV with the following results:

* nginx with modsecurity switched off works perfectly as a proxy nginx
* nginx with modsecurity switched on with one owasp rule-set
  (modsecurity_crs_20_protocol_violations.conf) works for common
  web-pages with multi-media content (quick test without any errors
  reported)
* nginx with modsecurity switched on with one owasp rule-set
  (modsecurity_crs_20_protocol_violations.conf) does not work for
  CalDAV.
  error.log: 2016/01/23 01:19:07 [emerg] 4844#0: *7 posix_memalign(16,
  4096) failed (12: Cannot allocate memory) while logging request
* nginx with modsecurity switched on without any ruleset
  does not work for CalDAV -- same error
* nginx with modsecurity switched off without any ruleset
  does work for CalDAV perfectly.

With modsecurity switched on, an Out-of-Memory exception took place
always reporting:

[876715.533926] nginx invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[876715.533930] nginx cpuset=/ mems_allowed=0
[876715.533936] CPU: 0 PID: 4844 Comm: nginx Not tainted 4.3.3-consecom-ag #1
[876715.533937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS debian/1.7.5-1-0-g506b58d-dirty-20140812_231322-gandalf 04/01/2014
[876715.533939]  f5a53ed0 d52542a6 f5a6b7c0 d5110792 d55a6db0 f5a6bab4 000280da 00000000
[876715.533943]  00000000 ffffffff 0d3f1361 00031d5e f4929cb8 00200282 f4929cb8 f4929cb0
[876715.533946]  d50babb7 00200206 d525956e 00000002 00000002 f5020840 f5020bc4 d55a5702
[876715.533949] Call Trace:
[876715.533955]  [<d52542a6>] ? dump_stack+0x3e/0x58
[876715.533959]  [<d5110792>] ? dump_header.isra.8+0x65/0x1be
[876715.533963]  [<d50babb7>] ? delayacct_end+0x47/0xa0
[876715.533967]  [<d525956e>] ? ___ratelimit+0x7e/0xe0
[876715.533970]  [<d50d0fa9>] ? oom_kill_process+0x1d9/0x380
[876715.533973]  [<d51e9d3a>] ? security_capable_noaudit+0x3a/0x60
[876715.533977]  [<d5047b0b>] ? has_ns_capability_noaudit+0xb/0x20
[876715.533979]  [<d50d0b76>] ? oom_badness+0x96/0x100
[876715.533981]  [<d50d1402>] ? out_of_memory+0x252/0x320
[876715.533984]  [<d50d4f5e>] ? __alloc_pages_nodemask+0x77e/0x7a0
[876715.533989]  [<d50efd24>] ? handle_mm_fault+0xd54/0xf50
[876715.533990]  [<d50f2cef>] ? vma_merge+0x1bf/0x280
[876715.533992]  [<d50f414a>] ? do_brk+0x1ca/0x2b0
[876715.533995]  [<d5037657>] ? __do_page_fault+0x137/0x3a0
[876715.533998]  [<d50379f0>] ? vmalloc_sync_all+0x130/0x130
[876715.534001]  [<d54d3566>] ? error_code+0x5a/0x60
[876715.534003]  [<d50379f0>] ? vmalloc_sync_all+0x130/0x130
[876715.534004] Mem-Info:
[876715.534008] active_anon:543864 inactive_anon:208884 isolated_anon:0
[876715.534008]  active_file:54 inactive_file:77 isolated_file:0
[876715.534008]  unevictable:0 dirty:1 writeback:0 unstable:0
[876715.534008]  slab_reclaimable:326 slab_unreclaimable:997
[876715.534008]  mapped:88 shmem:4 pagetables:957 bounce:0
[876715.534008]  free:21502 free_pcp:289 free_cma:0
[876715.534014] DMA free:12152kB min:64kB low:80kB high:96kB active_anon:1676kB inactive_anon:1928kB active_file:8kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15916kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:76kB kernel_stack:8kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:120 all_unreclaimable? yes
[876715.534016] lowmem_reserve[]: 0 839 3023 3023
[876715.534021] Normal free:73380kB min:3528kB low:4408kB high:5292kB active_anon:386788kB inactive_anon:386844kB active_file:208kB inactive_file:276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:892920kB managed:859928kB mlocked:0kB dirty:4kB writeback:0kB mapped:324kB shmem:0kB slab_reclaimable:1288kB slab_unreclaimable:3912kB kernel_stack:672kB pagetables:3808kB unstable:0kB bounce:0kB free_pcp:564kB local_pcp:564kB free_cma:0kB writeback_tmp:0kB pages_scanned:115004 all_unreclaimable? yes
[876715.534022] lowmem_reserve[]: 0 0 17471 17471
[876715.534027] HighMem free:476kB min:512kB low:2808kB high:5104kB active_anon:1786992kB inactive_anon:446764kB active_file:0kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2236296kB managed:2236296kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:16kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:592kB local_pcp:592kB free_cma:0kB writeback_tmp:0kB pages_scanned:7836 all_unreclaimable? yes
[876715.534028] lowmem_reserve[]: 0 0 0 0
[876715.534030] DMA: 4*4kB (E) 7*8kB (UE) 5*16kB (UEM) 3*32kB (U) 2*64kB (EM) 2*128kB (EM) 3*256kB (UEM) 1*512kB (E) 2*1024kB (UE) 2*2048kB (UE) 1*4096kB (M) = 12152kB
[876715.534039] Normal: 149*4kB (UEM) 108*8kB (UEM) 63*16kB (UE) 32*32kB (UEM) 10*64kB (UE) 11*128kB (UEM) 5*256kB (UE) 2*512kB (EM) 2*1024kB (UM) 3*2048kB (UEM) 14*4096kB (M) = 73380kB
[876715.534047] HighMem: 1*4kB (U) 1*8kB (U) 1*16kB (M) 2*32kB (UM) 0*64kB 1*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 476kB
[876715.534054] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=4096kB

Thanks for any hints

Lukas


--
Lukas Ruf       <http://www.lpr.ch> | Ad Personam
Consecom  <http://www.consecom.com> | Ad Laborem

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx/1.9.9 with modsecurity/2.9.0 crashes with segfault and worker process exited on signal 11

Robert Paprocki
The modsec devel team is working hard on the new libmodsecurity. You may just be better off waiting for them to put the finishing touches on that project. Nginx + modsec 2.9 likely will get no dev attention moving forward, given that the whole system is being revamped now.

Sent from my iPhone

> On Jan 22, 2016, at 16:44, Lukas <[hidden email]> wrote:
>
> Dear all
>
>> Lukas <[hidden email]> [2016-01-10 14:39]:
>>
>> Fascinated by nginx, I attempted to integrate it with modsecurity.
>>
>> Unfortunately, ever when modsecurity is enabled, nginx reports a
>> sefault in sysmessages.
>
> I tried debugging the issue a bit further (from a user perspective)
> with common web-page and CalDAV with the following results:
>
> * nginx with modsecurity switched off works perfectly as a proxy nginx
> * nginx with modsecurity switched on with one owasp rule-set
>  (modsecurity_crs_20_protocol_violations.conf) works for common
>  web-pages with multi-media content (quick test without any errors
>  reported)
> * nginx with modsecurity switched on with one owasp rule-set
>  (modsecurity_crs_20_protocol_violations.conf) does not work for
>  CalDAV.
>  error.log: 2016/01/23 01:19:07 [emerg] 4844#0: *7 posix_memalign(16,
>  4096) failed (12: Cannot allocate memory) while logging request
> * nginx with modsecurity switched on without any ruleset
>  does not work for CalDAV -- same error
> * nginx with modsecurity switched off without any ruleset
>  does work for CalDAV perfectly.
>
> With modsecurity switched on, an Out-of-Memory exception took place
> always reporting:
>
> [876715.533926] nginx invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
> [876715.533930] nginx cpuset=/ mems_allowed=0
> [876715.533936] CPU: 0 PID: 4844 Comm: nginx Not tainted 4.3.3-consecom-ag #1
> [876715.533937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS debian/1.7.5-1-0-g506b58d-dirty-20140812_231322-gandalf 04/01/2014
> [876715.533939]  f5a53ed0 d52542a6 f5a6b7c0 d5110792 d55a6db0 f5a6bab4 000280da 00000000
> [876715.533943]  00000000 ffffffff 0d3f1361 00031d5e f4929cb8 00200282 f4929cb8 f4929cb0
> [876715.533946]  d50babb7 00200206 d525956e 00000002 00000002 f5020840 f5020bc4 d55a5702
> [876715.533949] Call Trace:
> [876715.533955]  [<d52542a6>] ? dump_stack+0x3e/0x58
> [876715.533959]  [<d5110792>] ? dump_header.isra.8+0x65/0x1be
> [876715.533963]  [<d50babb7>] ? delayacct_end+0x47/0xa0
> [876715.533967]  [<d525956e>] ? ___ratelimit+0x7e/0xe0
> [876715.533970]  [<d50d0fa9>] ? oom_kill_process+0x1d9/0x380
> [876715.533973]  [<d51e9d3a>] ? security_capable_noaudit+0x3a/0x60
> [876715.533977]  [<d5047b0b>] ? has_ns_capability_noaudit+0xb/0x20
> [876715.533979]  [<d50d0b76>] ? oom_badness+0x96/0x100
> [876715.533981]  [<d50d1402>] ? out_of_memory+0x252/0x320
> [876715.533984]  [<d50d4f5e>] ? __alloc_pages_nodemask+0x77e/0x7a0
> [876715.533989]  [<d50efd24>] ? handle_mm_fault+0xd54/0xf50
> [876715.533990]  [<d50f2cef>] ? vma_merge+0x1bf/0x280
> [876715.533992]  [<d50f414a>] ? do_brk+0x1ca/0x2b0
> [876715.533995]  [<d5037657>] ? __do_page_fault+0x137/0x3a0
> [876715.533998]  [<d50379f0>] ? vmalloc_sync_all+0x130/0x130
> [876715.534001]  [<d54d3566>] ? error_code+0x5a/0x60
> [876715.534003]  [<d50379f0>] ? vmalloc_sync_all+0x130/0x130
> [876715.534004] Mem-Info:
> [876715.534008] active_anon:543864 inactive_anon:208884 isolated_anon:0
> [876715.534008]  active_file:54 inactive_file:77 isolated_file:0
> [876715.534008]  unevictable:0 dirty:1 writeback:0 unstable:0
> [876715.534008]  slab_reclaimable:326 slab_unreclaimable:997
> [876715.534008]  mapped:88 shmem:4 pagetables:957 bounce:0
> [876715.534008]  free:21502 free_pcp:289 free_cma:0
> [876715.534014] DMA free:12152kB min:64kB low:80kB high:96kB active_anon:1676kB inactive_anon:1928kB active_file:8kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15916kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:76kB kernel_stack:8kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:120 all_unreclaimable? yes
> [876715.534016] lowmem_reserve[]: 0 839 3023 3023
> [876715.534021] Normal free:73380kB min:3528kB low:4408kB high:5292kB active_anon:386788kB inactive_anon:386844kB active_file:208kB inactive_file:276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:892920kB managed:859928kB mlocked:0kB dirty:4kB writeback:0kB mapped:324kB shmem:0kB slab_reclaimable:1288kB slab_unreclaimable:3912kB kernel_stack:672kB pagetables:3808kB unstable:0kB bounce:0kB free_pcp:564kB local_pcp:564kB free_cma:0kB writeback_tmp:0kB pages_scanned:115004 all_unreclaimable? yes
> [876715.534022] lowmem_reserve[]: 0 0 17471 17471
> [876715.534027] HighMem free:476kB min:512kB low:2808kB high:5104kB active_anon:1786992kB inactive_anon:446764kB active_file:0kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2236296kB managed:2236296kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:16kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:592kB local_pcp:592kB free_cma:0kB writeback_tmp:0kB pages_scanned:7836 all_unreclaimable? yes
> [876715.534028] lowmem_reserve[]: 0 0 0 0
> [876715.534030] DMA: 4*4kB (E) 7*8kB (UE) 5*16kB (UEM) 3*32kB (U) 2*64kB (EM) 2*128kB (EM) 3*256kB (UEM) 1*512kB (E) 2*1024kB (UE) 2*2048kB (UE) 1*4096kB (M) = 12152kB
> [876715.534039] Normal: 149*4kB (UEM) 108*8kB (UEM) 63*16kB (UE) 32*32kB (UEM) 10*64kB (UE) 11*128kB (UEM) 5*256kB (UE) 2*512kB (EM) 2*1024kB (UM) 3*2048kB (UEM) 14*4096kB (M) = 73380kB
> [876715.534047] HighMem: 1*4kB (U) 1*8kB (U) 1*16kB (M) 2*32kB (UM) 0*64kB 1*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 476kB
> [876715.534054] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=4096kB
>
> Thanks for any hints
>
> Lukas
>
>
> --
> Lukas Ruf       <http://www.lpr.ch> | Ad Personam
> Consecom  <http://www.consecom.com> | Ad Laborem
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx