Memory issue

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Memory issue

anish10dec
Hi,

We suspect an issue on cpanel server from last update nginx.

Every overnight, there is many nginx reloads due to stat generation process
:

We see an ever increasing memory use of ngixn worker process, usually it
says around 1-2%, we now see it cumulating without coming back to normal
each overnight, we are now after 3 days at 7% memory.

Thanks

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273274#msg-273274

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
We only user nginx as proxy on concerned server

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273275#msg-273275

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
In reply to this post by anish10dec
We made a lot of tests, removing brotli, geoip, we can't solve the issue

i suspect :
nginx: cache manager process
simple bug !

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273303#msg-273303

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
In reply to this post by anish10dec
After upgrade and recompilation, issue is much less important, it increase
only by 0.17% for an antire stats processing cycle, but issue remain
unsolved....



[root@web1 ~]# nginx -V
nginx version: nginx/1.11.12
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with LibreSSL 2.5.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/etc/nginx/modules --with-pcre=./pcre-8.40 --with-pcre-jit
--with-zlib=./zlib-1.2.11 --with-openssl=./libressl-2.5.1
--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log
--http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nobody
--group=nobody --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module --with-http_dav_module
--with-http_flv_module --with-http_mp4_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-http_auth_request_module --add-dynamic-module=naxsi-http2/naxsi_src
--with-file-aio --with-threads --with-stream --with-stream_ssl_module
--with-http_slice_module --with-compat --with-http_v2_module
--with-http_geoip_module=dynamic
--add-dynamic-module=ngx_pagespeed-release-1.11.33.4-beta
--add-dynamic-module=/usr/local/rvm/gems/ruby-2.3.1/gems/passenger-5.1.2/src/nginx_module
--add-dynamic-module=ngx_brotli --add-dynamic-module=echo-nginx-module-0.60
--add-dynamic-module=headers-more-nginx-module-0.32
--add-dynamic-module=ngx_http_redis-0.3.8
--add-dynamic-module=redis2-nginx-module
--add-dynamic-module=srcache-nginx-module-0.31
--add-dynamic-module=ngx_devel_kit-0.3.0
--add-dynamic-module=set-misc-nginx-module-0.31
--add-dynamic-module=ModSecurity-nginx --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
--with-ld-opt=-Wl,-E


#Core Functionality

user  nobody;
worker_processes  8;
pid        /var/run/nginx.pid;
pcre_jit on;
error_log /var/log/nginx/error_log;
#error_log /home/abackup/debug.log debug;
worker_rlimit_nofile 300000;

#Load Dynamic Modules
include /etc/nginx/modules.d/*.load;

events {
worker_connections 8192;
use epoll;
multi_accept on;
accept_mutex off;
}

#Settings For other core modules like for example the stream module
include /etc/nginx/conf.d/main_custom_include.conf;

#Settings for the http core module
include /etc/nginx/conf.d/http_settings_custom.conf;

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273304#msg-273304

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
In reply to this post by anish10dec
Uprgraded to last nginx version :
memory still increase it seems

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273411#msg-273411

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Maxim Dounin
Hello!

On Wed, Apr 05, 2017 at 01:02:15AM -0400, JohnCarne wrote:

> Uprgraded to last nginx version :
> memory still increase it seems

You may have better luck describing your issue: what you do, what
you see as a result, and why you think this is an issue.  Posting
messages saying "I still have an issue" is unlikely to help.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
We described it properly when opening ticket, I reformulate :

Usually, 1 nginx worker process consumes 1.16-2% of RAM maximum on this
server, and it remain stable.
For some days after nginx upgrades, every overnight, during daily stat
generation process of cpanel which happens on overnight like set, there is
many nginx reloads due to stat generation (= normal), but this is now
causing an ever increasing memory use of RAM by nginx worker process,
usually it stays around 1-2% RAM, we now see it cumulating after stat
generation process increasing itself at begin with 1-2% RAM each time, which
will lead after some weaks to a saturated server in term of RAM if nginx is
not started. When we saw the issue first time, Nginx was consuming 12% of
server RAM considering we have 128 GB RAM on this shared hosting server.

After recent nginx upgrade :
The increase is around 0.20% daily, instead of 1-2% RAM

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273436#msg-273436

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
In reply to this post by anish10dec
It looks like i don't speak english properly to be understood, others will
open a thread on this issue, and may be explain better

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273437#msg-273437

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Aleksandar Lazic


Am 06-04-2017 07:14, schrieb JohnCarne:
> It looks like i don't speak english properly to be understood, others
> will
> open a thread on this issue, and may be explain better

Well how about to remove the additionally modules and watch if the
memory issue still exists.

###
--add-dynamic-module=ngx_pagespeed-release-1.11.33.4-beta
--add-dynamic-module=/usr/local/rvm/gems/ruby-2.3.1/gems/passenger-5.1.2/src/nginx_module
--add-dynamic-module=ngx_brotli
--add-dynamic-module=echo-nginx-module-0.60
--add-dynamic-module=headers-more-nginx-module-0.32
--add-dynamic-module=ngx_http_redis-0.3.8
--add-dynamic-module=redis2-nginx-module
--add-dynamic-module=srcache-nginx-module-0.31
--add-dynamic-module=ngx_devel_kit-0.3.0
--add-dynamic-module=set-misc-nginx-module-0.31
--add-dynamic-module=ModSecurity-nginx
###

Maybe some of this modules are the reasons for the memory issue.

Regards
Aleks
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
I let dev Anoop answer to you... he has a clue about the issue :

https://github.com/SpiderLabs/ModSecurity-nginx/issues/45

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273444#msg-273444

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Anoop Alias
If a module is dynamic loadable has issue and if we do not load the module , will it still cause the error ?

In the case above , ModSecurity-nginx was compiled as a dynamic module and not loaded .


On Thu, Apr 6, 2017 at 3:31 PM, JohnCarne <[hidden email]> wrote:
I let dev Anoop answer to you... he has a clue about the issue :

https://github.com/SpiderLabs/ModSecurity-nginx/issues/45

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273444#msg-273444

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Maxim Dounin
In reply to this post by anish10dec
Hello!

On Wed, Apr 05, 2017 at 09:32:41PM -0400, JohnCarne wrote:

> We described it properly when opening ticket, I reformulate :
>
> Usually, 1 nginx worker process consumes 1.16-2% of RAM maximum on this
> server, and it remain stable.
> For some days after nginx upgrades, every overnight, during daily stat
> generation process of cpanel which happens on overnight like set, there is
> many nginx reloads due to stat generation (= normal), but this is now
> causing an ever increasing memory use of RAM by nginx worker process,
> usually it stays around 1-2% RAM, we now see it cumulating after stat
> generation process increasing itself at begin with 1-2% RAM each time, which
> will lead after some weaks to a saturated server in term of RAM if nginx is
> not started. When we saw the issue first time, Nginx was consuming 12% of
> server RAM considering we have 128 GB RAM on this shared hosting server.
>
> After recent nginx upgrade :
> The increase is around 0.20% daily, instead of 1-2% RAM

So, you observe one nginx worker process consuming about 12% of
your server RAM, that is, more than 10GB of memory, correct?  You
may want to provide something like "ps alx | grep nginx" output
to illustrate the problem.

You may start with the following basic steps:

- Check your "nginx -V" output and nginx configuration; disable
  3rd party modules if there are any, and check if the problem
  persists.  In many cases various obscure problems are introduced
  by bugs in 3rd party modules.

- Make sure you are talking about a single worker process memory
  consumption, and not overral memory consumption of all nginx
  worker processes.  Multiple configuration reloads can leave
  multiple nginx worker processes in the "shutting down..." state
  for a long time which depends on the particular workload, and it
  is not a surprise you need memory if you do lots of configuration
  reloads.

- Check your nginx configuration to see if there natural reasons
  to consume memory - multiple connections and large buffers
  configured, thousands of complex location configurtions, large
  shared memory zones, and so on.

- Try to find out what exactly causes increased memory
  consumption.  The "stats generation process" you write about is
  not something nginx does by itself, and it is completely unknown
  what it means for anyone except you.

If the above won't be enough for you to identify the problem,
consider providing additional information about the observed
proble, including "ps alx" output which demonstrates the problem,
"nginx -V" output, and full nginx configuration (shown with
"nginx -T").

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
cpanel stat generation cause thet nginx makes a lot of reload to grab new
file descriptor... no issue on that

Issue is nginx, I show you situation now with 2.58% used for 1 work, which
is same value for others, but gloably, nginx uses now 2.58%, this number is
increasing slowly at the rythm of nginx reloads asked...


[root@web1 ~]# ps alx | grep nginx
5    99  711213  913692  20   0 3917936 3397132 ep_pol S ?          0:22
nginx: worker process
5    99  711224  913692  20   0 3918128 3397300 ep_pol S ?          0:24
nginx: worker process
5    99  711229  913692  20   0 3918392 3397456 ep_pol S ?          0:26
nginx: worker process
5    99  711238  913692  20   0 3918128 3397228 ep_pol S ?          0:20
nginx: worker process
5    99  711245  913692  20   0 3917936 3397144 ep_pol S ?          0:23
nginx: worker process
5    99  711248  913692  20   0 3918096 3397296 ep_pol S ?          0:18
nginx: worker process
5    99  711252  913692  20   0 3918392 3397392 ep_pol S ?          0:21
nginx: worker process
5    99  711255  913692  20   0 3918128 3397132 -   R    ?          0:19
nginx: worker process
5    99  711257  913692  20   0 3917580 3394632 ep_pol S ?          0:00
nginx: cache manager process
0     0  767011  766950  20   0 112652   956 pipe_w S+   pts/2      0:00
grep --color=auto nginx
5     0  913692       1  20   0 3917576 3396176 sigsus Ss ?        74:26
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf


Output of nginx -T is taking 100's of pages, i can't pu it here...

[root@web1 ~]# nginx -V
nginx version: nginx/1.11.13
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with LibreSSL 2.5.2
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/etc/nginx/modules --with-pcre=./pcre-8.40 --with-pcre-jit
--with-zlib=./zlib-1.2.11 --with-openssl=./libressl-2.5.2
--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log
--http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nobody
--group=nobody --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module --with-http_dav_module
--with-http_flv_module --with-http_mp4_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-http_auth_request_module --add-dynamic-module=naxsi-http2/naxsi_src
--with-file-aio --with-threads --with-stream --with-stream_ssl_module
--with-http_slice_module --with-compat --with-http_v2_module
--with-http_geoip_module=dynamic
--add-dynamic-module=ngx_pagespeed-release-1.11.33.4-beta
--add-dynamic-module=/usr/local/rvm/gems/ruby-2.3.1/gems/passenger-5.1.2/src/nginx_module
--add-dynamic-module=ngx_brotli --add-dynamic-module=echo-nginx-module-0.60
--add-dynamic-module=headers-more-nginx-module-0.32
--add-dynamic-module=ngx_http_redis-0.3.8
--add-dynamic-module=redis2-nginx-module
--add-dynamic-module=srcache-nginx-module-0.31
--add-dynamic-module=ngx_devel_kit-0.3.0
--add-dynamic-module=set-misc-nginx-module-0.31
--add-dynamic-module=ModSecurity-nginx --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
--with-ld-opt=-Wl,-E
[root@web1 ~]#


Anoop is dev of Xtendweb stack using nginx core, he is investigating, and it
seems his solution if to use a pre-approved nginx core :
https://openresty.org/en/
and that it could solve many issues with 3rd party modules which we need to
use absolutely...

Indeed, we are all tired to do 3 upgrades /month

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273456#msg-273456

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Valentin V. Bartenev-3
On Thursday 06 April 2017 11:32:45 JohnCarne wrote:
> cpanel stat generation cause thet nginx makes a lot of reload to grab new
> file descriptor... no issue on that
>
[..]

JFYI, reloading nginx isn't required to reopen log files.
See for details: http://nginx.org/en/docs/control.html#logs

  wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
Thanks for info, Anoop will read this, he is subscribed

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273458#msg-273458

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Lucas Rolff-2
In reply to this post by anish10dec
> cpanel stat generation cause thet nginx makes a lot of reload to grab new file descriptor... no issue on that

Even though this is off-topic - if you issue a lot of reloads during cPanel stat generation, your hooks are configured wrong, since Apache in cPanel only reloads *once* during the whole process.

> Issue is nginx, I show you situation now with 2.58% used for 1 work, which is same value for others, but gloably, nginx uses now 2.58%, this number is increasing slowly at the rythm of nginx reloads asked..

I happen to use nginx (mainline version) myself on a cPanel server - no custom modules, there's no memory leak in the latest versions of nginx - I stay happily at 0.2% in memory (32 gigabyte server)

> Anoop is dev of Xtendweb stack using nginx core, he is investigating, and it seems his solution if to use a pre-approved nginx core : https://openresty.org/en/

Using OpenResty wouldn't solve your issues - you can use the exact same modules as OpenResty does, in a normal nginx build (be aware that some modules such as the lua module, and the echo-nginx module isn't yet building correctly against nginx 1.11.11+)

In the end, include only the extra modules you require - because as far as I can tell, in the mainline version of nginx, nothing (at least for me) has caused memory issues with workers, even when reloading a bunch of times.

It might very well be that one of the 3rd-party modules have not been fully tested to work with 1.11.13







On 06/04/2017, 17.32, "nginx on behalf of JohnCarne" <[hidden email] on behalf of [hidden email]> wrote:

>cpanel stat generation cause thet nginx makes a lot of reload to grab new
>file descriptor... no issue on that
>
>Issue is nginx, I show you situation now with 2.58% used for 1 work, which
>is same value for others, but gloably, nginx uses now 2.58%, this number is
>increasing slowly at the rythm of nginx reloads asked...
>
>
>[root@web1 ~]# ps alx | grep nginx
>5    99  711213  913692  20   0 3917936 3397132 ep_pol S ?          0:22
>nginx: worker process
>5    99  711224  913692  20   0 3918128 3397300 ep_pol S ?          0:24
>nginx: worker process
>5    99  711229  913692  20   0 3918392 3397456 ep_pol S ?          0:26
>nginx: worker process
>5    99  711238  913692  20   0 3918128 3397228 ep_pol S ?          0:20
>nginx: worker process
>5    99  711245  913692  20   0 3917936 3397144 ep_pol S ?          0:23
>nginx: worker process
>5    99  711248  913692  20   0 3918096 3397296 ep_pol S ?          0:18
>nginx: worker process
>5    99  711252  913692  20   0 3918392 3397392 ep_pol S ?          0:21
>nginx: worker process
>5    99  711255  913692  20   0 3918128 3397132 -   R    ?          0:19
>nginx: worker process
>5    99  711257  913692  20   0 3917580 3394632 ep_pol S ?          0:00
>nginx: cache manager process
>0     0  767011  766950  20   0 112652   956 pipe_w S+   pts/2      0:00
>grep --color=auto nginx
>5     0  913692       1  20   0 3917576 3396176 sigsus Ss ?        74:26
>nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
>
>
>Output of nginx -T is taking 100's of pages, i can't pu it here...
>
>[root@web1 ~]# nginx -V
>nginx version: nginx/1.11.13
>built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
>built with LibreSSL 2.5.2
>TLS SNI support enabled
>configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
>--modules-path=/etc/nginx/modules --with-pcre=./pcre-8.40 --with-pcre-jit
>--with-zlib=./zlib-1.2.11 --with-openssl=./libressl-2.5.2
>--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log
>--http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid
>--lock-path=/var/run/nginx.lock
>--http-client-body-temp-path=/var/cache/nginx/client_temp
>--http-proxy-temp-path=/var/cache/nginx/proxy_temp
>--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
>--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
>--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nobody
>--group=nobody --with-http_ssl_module --with-http_realip_module
>--with-http_addition_module --with-http_sub_module --with-http_dav_module
>--with-http_flv_module --with-http_mp4_module --with-http_gunzip_module
>--with-http_gzip_static_module --with-http_random_index_module
>--with-http_secure_link_module --with-http_stub_status_module
>--with-http_auth_request_module --add-dynamic-module=naxsi-http2/naxsi_src
>--with-file-aio --with-threads --with-stream --with-stream_ssl_module
>--with-http_slice_module --with-compat --with-http_v2_module
>--with-http_geoip_module=dynamic
>--add-dynamic-module=ngx_pagespeed-release-1.11.33.4-beta
>--add-dynamic-module=/usr/local/rvm/gems/ruby-2.3.1/gems/passenger-5.1.2/src/nginx_module
>--add-dynamic-module=ngx_brotli --add-dynamic-module=echo-nginx-module-0.60
>--add-dynamic-module=headers-more-nginx-module-0.32
>--add-dynamic-module=ngx_http_redis-0.3.8
>--add-dynamic-module=redis2-nginx-module
>--add-dynamic-module=srcache-nginx-module-0.31
>--add-dynamic-module=ngx_devel_kit-0.3.0
>--add-dynamic-module=set-misc-nginx-module-0.31
>--add-dynamic-module=ModSecurity-nginx --with-cc-opt='-O2 -g -pipe -Wall
>-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
>--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
>--with-ld-opt=-Wl,-E
>[root@web1 ~]#
>
>
>Anoop is dev of Xtendweb stack using nginx core, he is investigating, and it
>seems his solution if to use a pre-approved nginx core :
>https://openresty.org/en/
>and that it could solve many issues with 3rd party modules which we need to
>use absolutely...
>
>Indeed, we are all tired to do 3 upgrades /month
>
>Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273456#msg-273456
>
>_______________________________________________
>nginx mailing list
>[hidden email]
>http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Maxim Dounin
In reply to this post by anish10dec
Hello!

On Thu, Apr 06, 2017 at 11:32:45AM -0400, JohnCarne wrote:

> cpanel stat generation cause thet nginx makes a lot of reload to grab new
> file descriptor... no issue on that
>
> Issue is nginx, I show you situation now with 2.58% used for 1 work, which
> is same value for others, but gloably, nginx uses now 2.58%, this number is
> increasing slowly at the rythm of nginx reloads asked...
>
>
> [root@web1 ~]# ps alx | grep nginx
> 5    99  711213  913692  20   0 3917936 3397132 ep_pol S ?          0:22
> nginx: worker process
> 5    99  711224  913692  20   0 3918128 3397300 ep_pol S ?          0:24
> nginx: worker process
> 5    99  711229  913692  20   0 3918392 3397456 ep_pol S ?          0:26
> nginx: worker process
> 5    99  711238  913692  20   0 3918128 3397228 ep_pol S ?          0:20
> nginx: worker process
> 5    99  711245  913692  20   0 3917936 3397144 ep_pol S ?          0:23
> nginx: worker process
> 5    99  711248  913692  20   0 3918096 3397296 ep_pol S ?          0:18
> nginx: worker process
> 5    99  711252  913692  20   0 3918392 3397392 ep_pol S ?          0:21
> nginx: worker process
> 5    99  711255  913692  20   0 3918128 3397132 -   R    ?          0:19
> nginx: worker process
> 5    99  711257  913692  20   0 3917580 3394632 ep_pol S ?          0:00
> nginx: cache manager process
> 0     0  767011  766950  20   0 112652   956 pipe_w S+   pts/2      0:00
> grep --color=auto nginx
> 5     0  913692       1  20   0 3917576 3396176 sigsus Ss ?        74:26
> nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf

Ok, so nginx master process grown to 3+ GB, and all workers are
mostly identical.  So the memory is certainly consumed by the
master process.  This may or may not be normal, depending on the
configuration used.

If you think that memory is consumed on configuration reloads, you
may want to provide "ps alx | grep nginx" after restarting nginx
(it is expected to be small), after one configuration reload (it
is expected to be larger due to another copy of the configuration
allocated) and after multiple configuration reloads (this will
demonstrate increased memory consumption, if any).

Note well: leaking memory on configuration reloads is a more or
less typical problem with various 3rd party modules.  IIRC,
various versions of modsecurity were previously reported doing
this.  As previously suggested, try without 3rd party modules to
see if it helps.

> Output of nginx -T is taking 100's of pages, i can't pu it here...

Well, not a problem.  It simply limits possibility of others to help
you.  Anyway, I suspect your problem is 3rd party modules, and I
don't think there is a room to investigate further before testing
if the problem persists without 3rd party modules.

[...]

> Indeed, we are all tired to do 3 upgrades /month

Note that the way you are asking others to help you is not likely
to attract many volunteers.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

anish10dec
Thanks for your feebacks. I do my best seriously to communicate what I can
!

We note this issue on very busy server only, it will be hard to remove all
modules on such busy server, and reconfig all, but not impossible. Anoop is
on the case on smallest servers, and succeed to see the issue at small
scale, he will troubleshoot with sometimes...

Yes, 1 of the 3rd party module is leaking memory for sure, no doubt about
this for me.

ps alx | grep nginx" after restarting nginx
This gives 1.16% as usual
+ 1 more reload, we get around 2.24%
+ more 100's more reloads, we gradually get to 2.58%

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273461#msg-273461

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

nginx mailing list
Idea coming right out of the blue: have you given a thought on compiling nginx (+ gradually modules) with valgrind? ​You should know pretty quickly if something is wrong.​
​Note the slowdown, though. Might not be a good idea on production, or if you do not secure some offload to somewhere else if it becomes messy.​
---
B. R.

On Thu, Apr 6, 2017 at 7:20 PM, JohnCarne <[hidden email]> wrote:
Thanks for your feebacks. I do my best seriously to communicate what I can
!

We note this issue on very busy server only, it will be hard to remove all
modules on such busy server, and reconfig all, but not impossible. Anoop is
on the case on smallest servers, and succeed to see the issue at small
scale, he will troubleshoot with sometimes...

Yes, 1 of the 3rd party module is leaking memory for sure, no doubt about
this for me.

ps alx | grep nginx" after restarting nginx
This gives 1.16% as usual
+ 1 more reload, we get around 2.24%
+ more 100's more reloads, we gradually get to 2.58%

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,273274,273461#msg-273461

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory issue

Aleksandar Lazic
In reply to this post by Maxim Dounin


Am 06-04-2017 19:05, schrieb Maxim Dounin:
> Hello!
>
> On Thu, Apr 06, 2017 at 11:32:45AM -0400, JohnCarne wrote:
>

[...]

> [...]
>
>> Indeed, we are all tired to do 3 upgrades /month
>
> Note that the way you are asking others to help you is not likely
> to attract many volunteers.

+1
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
12
Loading...