Possible memory leak?

classic Classic list List threaded Threaded
36 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Possible memory leak?

adrian.hilt
We're running Nginx version 1.15.8 but we've been seeing similar issues with
other versions too and on all of our servers that have a high number of
vhosts.

The issue is that when you do an nginx reload it ends up using almost 2x the
ram as it was previously.  Here is a test I ran.
--------------------------------------------------------------------------------
 21.3 MiB +   1.4 GiB =   1.4 GiB nginx (3)
 21.3 MiB +   1.4 GiB =   1.4 GiB nginx (3)
484.2 MiB +   1.4 GiB =   1.9 GiB nginx (3)
588.1 MiB +   1.4 GiB =   2.0 GiB nginx (3)
720.3 MiB +   1.4 GiB =   2.1 GiB nginx (3)
  1.4 GiB +   1.4 GiB =   2.8 GiB nginx (3)
 18.0 MiB +   2.7 GiB =   2.7 GiB nginx (3)
 20.8 MiB +   2.7 GiB =   2.7 GiB nginx (3)
 20.8 MiB +   2.7 GiB =   2.7 GiB nginx (3)
--------------------------------------------------------------------------------

I expect the ram usage to increase while the reload is happening but after
it's done shouldn't the ram usage go back to about the same level?

This issue is completely reproducible across all of our servers and if I do
a full restart, ram usage goes back down to normal.

Any thoughts?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283216#msg-283216

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Maxim Dounin
Hello!

On Thu, Feb 28, 2019 at 01:43:18PM -0500, wkbrad wrote:

> We're running Nginx version 1.15.8 but we've been seeing similar issues with
> other versions too and on all of our servers that have a high number of
> vhosts.
>
> The issue is that when you do an nginx reload it ends up using almost 2x the
> ram as it was previously.  Here is a test I ran.
> --------------------------------------------------------------------------------
>  21.3 MiB +   1.4 GiB =   1.4 GiB nginx (3)
>  21.3 MiB +   1.4 GiB =   1.4 GiB nginx (3)
> 484.2 MiB +   1.4 GiB =   1.9 GiB nginx (3)
> 588.1 MiB +   1.4 GiB =   2.0 GiB nginx (3)
> 720.3 MiB +   1.4 GiB =   2.1 GiB nginx (3)
>   1.4 GiB +   1.4 GiB =   2.8 GiB nginx (3)
>  18.0 MiB +   2.7 GiB =   2.7 GiB nginx (3)
>  20.8 MiB +   2.7 GiB =   2.7 GiB nginx (3)
>  20.8 MiB +   2.7 GiB =   2.7 GiB nginx (3)
> --------------------------------------------------------------------------------
>
> I expect the ram usage to increase while the reload is happening but after
> it's done shouldn't the ram usage go back to about the same level?
>
> This issue is completely reproducible across all of our servers and if I do
> a full restart, ram usage goes back down to normal.
>
> Any thoughts?

Configuration reload implies that master process parses the
configuration and creates new configuration structures in memory.  
That is, memory usage is expected to be 2x compared to a clean
startup assuming most of the memory is used for the configuration.

Once the configuration is correctly parsed and applied, master
process will free the old configuration.  At this point memory
usage is expected to be the same as after a clean start, but given
memory allocation details it is almost never the case.

For example, assuming the system allocator simply uses sbrk()
without any caching, after the configuration reload the new
configuration will use addresses higher than the original one
used, so allocator will not be able to release no-longer-needed
memory (previously used by the original configuration) to the
system.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
Maxim Dounin Wrote:
-------------------------------------------------------
> so allocator will not be able to release no-longer-needed
> memory (previously used by the original configuration) to the
> system.

Thanks Maxim!  That sounds like the very definition of a memory leak to me.
:)

But I'm not sure how accurate it is.  For example, if I reload Nginx again
it doesn't then use 3x or 4x more ram.  It stays at the same, but elevated,
level as the first reload.  So it does appear to be able to release that
memory, maybe just not on the first reload.

But why would that be and is there a solution to that kind of problem?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283222#msg-283222

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Maxim Dounin
Hello!

On Thu, Feb 28, 2019 at 03:54:24PM -0500, wkbrad wrote:

> Maxim Dounin Wrote:
> -------------------------------------------------------
> > so allocator will not be able to release no-longer-needed
> > memory (previously used by the original configuration) to the
> > system.
>
> Thanks Maxim!  That sounds like the very definition of a memory leak to me.
> :)

Well, if it does, you'll probably want to re-check your definition
of a memory leak.  This is a result of how a particular allocator
works, not a memory leak.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
Thanks Maxim!  I really appreciate your time!

As for what I was referring to a memory leak, I guess I'm just referring to
the effects I'm seeing across a number of systems with highly different
configurations.

I'm not an expert so I'll defer to you on that.  :)

My question still stands though, is there a way to solve that particular
issue? It is causing us problems when the ram that Nginx is using doubles.

Are you saying this is an issue with our OS ( CentOS 6 and 7 ), specific
Nginx build ( we use 2 different builds but they are the same version ), or
something else?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283227#msg-283227

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

RE: Possible memory leak?

Reinis Rozitis
> My question still stands though, is there a way to solve that particular issue? It is
> causing us problems when the ram that Nginx is using doubles.

Theoretically if that’s a problem you could instead of a reload send USR2 and QUIT to the nginx process (http://nginx.org/en/docs/control.html) which should spawn a new master and the gracefully quit the old one.

Correct me if I'm wrong (actually haven't tested for memory usage).

rr

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

nginx mailing list
That should do it, AFAIK a process cannot give back memory already allocated to the system.

I would be more than interested to know more about a different technique that doesn't involve a fork system call.

Em quinta-feira, 28 de fevereiro de 2019 19:28:00 BRT, Reinis Rozitis <[hidden email]> escreveu:


> My question still stands though, is there a way to solve that particular issue? It is
> causing us problems when the ram that Nginx is using doubles.

Theoretically if that’s a problem you could instead of a reload send USR2 and QUIT to the nginx process (http://nginx.org/en/docs/control.html) which should spawn a new master and the gracefully quit the old one.

Correct me if I'm wrong (actually haven't tested for memory usage).


rr

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: RE: Possible memory leak?

adrian.hilt
In reply to this post by Reinis Rozitis
Thanks Reinis!  That's some really great info and from the short tests that
I've run so far I think this is going to be the solution.

I used this command as the test:
pid="$(cat /run/nginx.pid)"; kill -USR2 $pid; sleep 10; kill -QUIT $pid

And here is what happened with the reload:
 37.0 MiB +   1.4 GiB =   1.4 GiB    nginx (3)
495.4 MiB +   1.4 GiB =   1.9 GiB    nginx (4)
606.6 MiB +   1.4 GiB =   2.0 GiB    nginx (4)
738.3 MiB +   1.4 GiB =   2.1 GiB    nginx (4)
 40.1 MiB +   1.7 GiB =   1.8 GiB    nginx (4)
 57.1 MiB +   2.8 GiB =   2.8 GiB    nginx (6)
 57.4 MiB +   2.8 GiB =   2.8 GiB    nginx (6)
  1.3 GiB +   1.4 GiB =   2.7 GiB    nginx (5)
 14.6 MiB +   1.4 GiB =   1.4 GiB    nginx (4)

Started at 1.4G and ended at 1.4G.  Yay!

I also tested whether it was reloading gracefully (i.e. not killing active
connections ) and indeed it is.  Yay again!

I'm going to run some more tests tomorrow and modify the systemd script on
one of our servers as another test.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283232#msg-283232

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
In reply to this post by adrian.hilt
Hi all,

I just wanted to share the details of what I've found about this issue.
Also thanks to Maxim Dounin and Reinis Rozitis who gave some really great
answers!

The more I look into this the more I'm convinced this is an issue with Nginx
itself.  I've tested this with 3 different builds now and all have the exact
same issue.

The first 2 types of servers I tested were both running Nginx 1.15.8 on
Centos 7 ( with 1 of them being on 6 ).  I tested about 10 of our over 100
servers.  This time I tested in a default install of Debian 9 with Nginix
version 1.10.3 and the issue exists there too.  I just wanted to test on
something completely different.

For the test, I created 50k very simple vhosts which used about 1G of RAM.
Here is the ps_mem output.
 94.3 MiB +   1.0 GiB =   1.1 GiB nginx (3)

After a normal reload it then uses 2x the ram:
186.3 MiB +   1.9 GiB =   2.1 GiB nginx (3)

And if I reload it again it briefly jumps up to about 4G during the reload
and then goes back down to 2G.

If I instead use the "upgrade" option.  In the case of Debian, service nginx
upgrade, then it reloads gracefully and goes back to using 1G again.
100.8 MiB +   1.0 GiB =   1.1 GiB nginx (3)

The difference between the "reload" and "upgrade" process is basically only
that reload sends a HUP signal to Nginx and upgrade sends a USR2 and then
QUIT signal.  What happens with all of those signals is entirely up to
Nginx.  It could even ignore them if chose too.

Additionally, I ran the same test with Apache.  Not because I want to
compare Nginx to Apache, they are different for a reason.  I just wanted to
test if this was a system issue.  So I did the same thing on Debian 9,
installed Apache and created 50k simple vhosts.  It used about 800M of ram
and reloading did not cause that to increase at all.

All of that leads me to these questions.

Why would anyone want to use the normal reload process to reload the Nginx
configuration?
Shouldn't we always be using the upgrade process instead?
Are there any downsides to doing that?
Has anyone else noticed these issues and have you found another fix?

Look forward to hearing back and thanks in advance!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283309#msg-283309

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Anoop Alias
Nginx does use more ram for the number of vhosts than Apache does 


The USR2 is for binary in-place upgrades and normally you should just send a SIGHUP, I have seen sometimes the USR2 leading multiple master process and showing some weird behaviour 

You can probably use worker_shutdown_timeout 10s; or something to get the workers to shut down in a more timebound manner

 

On Fri, Mar 8, 2019 at 12:03 AM wkbrad <[hidden email]> wrote:
Hi all,

I just wanted to share the details of what I've found about this issue.
Also thanks to Maxim Dounin and Reinis Rozitis who gave some really great
answers!

The more I look into this the more I'm convinced this is an issue with Nginx
itself.  I've tested this with 3 different builds now and all have the exact
same issue.

The first 2 types of servers I tested were both running Nginx 1.15.8 on
Centos 7 ( with 1 of them being on 6 ).  I tested about 10 of our over 100
servers.  This time I tested in a default install of Debian 9 with Nginix
version 1.10.3 and the issue exists there too.  I just wanted to test on
something completely different.

For the test, I created 50k very simple vhosts which used about 1G of RAM.
Here is the ps_mem output.
 94.3 MiB +   1.0 GiB =   1.1 GiB       nginx (3)

After a normal reload it then uses 2x the ram:
186.3 MiB +   1.9 GiB =   2.1 GiB       nginx (3)

And if I reload it again it briefly jumps up to about 4G during the reload
and then goes back down to 2G.

If I instead use the "upgrade" option.  In the case of Debian, service nginx
upgrade, then it reloads gracefully and goes back to using 1G again.
100.8 MiB +   1.0 GiB =   1.1 GiB       nginx (3)

The difference between the "reload" and "upgrade" process is basically only
that reload sends a HUP signal to Nginx and upgrade sends a USR2 and then
QUIT signal.  What happens with all of those signals is entirely up to
Nginx.  It could even ignore them if chose too.

Additionally, I ran the same test with Apache.  Not because I want to
compare Nginx to Apache, they are different for a reason.  I just wanted to
test if this was a system issue.  So I did the same thing on Debian 9,
installed Apache and created 50k simple vhosts.  It used about 800M of ram
and reloading did not cause that to increase at all.

All of that leads me to these questions.

Why would anyone want to use the normal reload process to reload the Nginx
configuration?
Shouldn't we always be using the upgrade process instead?
Are there any downsides to doing that?
Has anyone else noticed these issues and have you found another fix?

Look forward to hearing back and thanks in advance!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283309#msg-283309

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
Thanks, Anoop!  But I don't think you understood the point I was trying to
get across.  I was definitely not trying to compare nginx and apache memory
usage. Let's just ignore that part was ever said.  :)

I'm trying to understand why Nginx is using 2x the memory usage when the HUP
signal is sent, i.e. the normal reload process.

When you use the USR2/QUIT method, i.e. the binary upgrade process, it
doesn't do this.

It's a big problem on high vhost servers when you go from normally using 1G
of ram to using 2G and then 4G during subsequent reloads.

It's that brief 4G spike that initially caught my attention.  But then I
noticed that it was always using 2x more ram.  Whoa!

This is super easy to reproduce so I invite you to test it yourself.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283312#msg-283312

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Anoop Alias
Its simple..Nginx has a master process and number of worker process as you configure in nginx.conf . Its the workers that handle connections and each one does async

When you do a HUP, all the master process is doing is spawning n new workers and all new connections to port 80/443 are handled by the new workers, but remember that he old workers may still be doing some job and terminating it then and there means you are closing off some connections in a non-graceful way, so the master process keeps the old workers also active for sometime to let it gracefully finish all its doing 

So if the worker process is n , during reload it will become 2n and then n workers are gracefully shutdown which means if n workers use x memory , then during reload the memory become 2x 

You can set workers to a low value ,say 1 worker process if the system is limited in memory ,but the possibility of having 2n workers during reload cannot be avoided as its more like a feature and the 2x memory usage is an unwanted side effect of this feature

Having said that Nginx dev's can still look into why defining more vhost consume lot of memory while apache dont have this problem. I develop an automation script for a popular web control panel and most servers using the script have upto 10k vhost defined and the memory usage would be 4x times than apache for nginx with this much amount of vhosts defined . with ssl defs etc needed for each vhost we cannot reduce the number of vhosts also 


On Fri, Mar 8, 2019 at 8:05 AM wkbrad <[hidden email]> wrote:
Thanks, Anoop!  But I don't think you understood the point I was trying to
get across.  I was definitely not trying to compare nginx and apache memory
usage. Let's just ignore that part was ever said.  :)

I'm trying to understand why Nginx is using 2x the memory usage when the HUP
signal is sent, i.e. the normal reload process.

When you use the USR2/QUIT method, i.e. the binary upgrade process, it
doesn't do this.

It's a big problem on high vhost servers when you go from normally using 1G
of ram to using 2G and then 4G during subsequent reloads.

It's that brief 4G spike that initially caught my attention.  But then I
noticed that it was always using 2x more ram.  Whoa!

This is super easy to reproduce so I invite you to test it yourself.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283312#msg-283312

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

lists@lazygranch.com
In reply to this post by adrian.hilt
On Thu, 07 Mar 2019 13:33:39 -0500
"wkbrad" <[hidden email]> wrote:

> Hi all,
>
> I just wanted to share the details of what I've found about this
> issue. Also thanks to Maxim Dounin and Reinis Rozitis who gave some
> really great answers!
>
> The more I look into this the more I'm convinced this is an issue
> with Nginx itself.  I've tested this with 3 different builds now and
> all have the exact same issue.
>
> The first 2 types of servers I tested were both running Nginx 1.15.8
> on Centos 7 ( with 1 of them being on 6 ).  I tested about 10 of our
> over 100 servers.  This time I tested in a default install of Debian
> 9 with Nginix version 1.10.3 and the issue exists there too.  I just
> wanted to test on something completely different.
>
> For the test, I created 50k very simple vhosts which used about 1G of
> RAM. Here is the ps_mem output.
>  94.3 MiB +   1.0 GiB =   1.1 GiB nginx (3)
>
> After a normal reload it then uses 2x the ram:
> 186.3 MiB +   1.9 GiB =   2.1 GiB nginx (3)
>
> And if I reload it again it briefly jumps up to about 4G during the
> reload and then goes back down to 2G.
>
> If I instead use the "upgrade" option.  In the case of Debian,
> service nginx upgrade, then it reloads gracefully and goes back to
> using 1G again. 100.8 MiB +   1.0 GiB =   1.1 GiB nginx (3)
>
> The difference between the "reload" and "upgrade" process is
> basically only that reload sends a HUP signal to Nginx and upgrade
> sends a USR2 and then QUIT signal.  What happens with all of those
> signals is entirely up to Nginx.  It could even ignore them if chose
> too.
>
> Additionally, I ran the same test with Apache.  Not because I want to
> compare Nginx to Apache, they are different for a reason.  I just
> wanted to test if this was a system issue.  So I did the same thing
> on Debian 9, installed Apache and created 50k simple vhosts.  It used
> about 800M of ram and reloading did not cause that to increase at all.
>
> All of that leads me to these questions.
>
> Why would anyone want to use the normal reload process to reload the
> Nginx configuration?
> Shouldn't we always be using the upgrade process instead?
> Are there any downsides to doing that?
> Has anyone else noticed these issues and have you found another fix?
>
> Look forward to hearing back and thanks in advance!
>
> Posted at Nginx Forum:
> https://forum.nginx.org/read.php?2,283216,283309#msg-283309
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx



Well for what it's worth, here is my result.
centos 7 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux

sh-4.2# nginx -v
nginx version: nginx/1.14.0


sh-4.2# ps_mem | grep nginx
  4.7 MiB +   2.1 MiB =   6.7 MiB       nginx (2)
sh-4.2# systemctl reload nginx
sh-4.2# ps_mem | grep nginx
  1.7 MiB +   4.0 MiB =   5.7 MiB       nginx (2)
sh-4.2# systemctl restart nginx
sh-4.2# ps_mem | grep nginx
804.0 KiB +   3.5 MiB =   4.2 MiB       nginx (2)
sh-4.2# ps_mem | grep nginx
  2.9 MiB +   2.9 MiB =   5.8 MiB       nginx (2)
sh-4.2# ps_mem | grep nginx
  2.9 MiB +   2.9 MiB =   5.8 MiB       nginx (2)
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
In reply to this post by Anoop Alias
Hi Anoop!

I thought you might have been the nDeploy guy and I've been planning on
bringing this up with you too.  We actually have several servers licensed
with you.  :)

And they do have the same issue but you're still misunderstanding what the
problem is.

I completely understand that when the reload happens it should use 2x the
ram.  That's expected.  What is not expected is that the ram stays at that
level AFTER the reload is complete.

Let's look at an example from a live Xtendweb server.  Here is the ram usage
after a restart.
 30.5 MiB +   1.4 GiB =   1.5 GiB nginx (4)

And here is the ram usage after a reload.
 28.4 MiB +   2.8 GiB =   2.9 GiB nginx (4)

The reload is completely finished at that point with no workers in shutting
down state and it's now using 2x the ram.  Now if I use the binary reload
process next it goes back down.
 26.1 MiB +   1.5 GiB =   1.5 GiB nginx (4)

Again, I'm not talking about what SHOULD be happening.  It's totally normal
and expected for it to use 2x the ram DURING the reload.  It's not expected
for it to continue using 2x the ram AFTER the reload is finished.

Thanks!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283317#msg-283317

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
In reply to this post by lists@lazygranch.com
Thanks for that info.  It's definitely harder to notice the issue on small
servers like that.  But you are still seeing about a 50% increase in ram
usage there by your own tests.

The smallest server I've tested this on uses about 20M during the first
start and about 50M after a reload is completely finished.

Not so much of a problem for small servers but definitely a big problem for
large ones.  That said the issue is still there on small servers like you've
just seen.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283318#msg-283318

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Anoop Alias
In reply to this post by adrian.hilt
Sorry OT -  @wkbrad -  Please contact me off-the-list and we can discuss this further 

On Fri, Mar 8, 2019 at 9:09 PM wkbrad <[hidden email]> wrote:
Hi Anoop!

I thought you might have been the nDeploy guy and I've been planning on
bringing this up with you too.  We actually have several servers licensed
with you.  :)

And they do have the same issue but you're still misunderstanding what the
problem is.

I completely understand that when the reload happens it should use 2x the
ram.  That's expected.  What is not expected is that the ram stays at that
level AFTER the reload is complete.

Let's look at an example from a live Xtendweb server.  Here is the ram usage
after a restart.
 30.5 MiB +   1.4 GiB =   1.5 GiB       nginx (4)

And here is the ram usage after a reload.
 28.4 MiB +   2.8 GiB =   2.9 GiB       nginx (4)

The reload is completely finished at that point with no workers in shutting
down state and it's now using 2x the ram.  Now if I use the binary reload
process next it goes back down.
 26.1 MiB +   1.5 GiB =   1.5 GiB       nginx (4)

Again, I'm not talking about what SHOULD be happening.  It's totally normal
and expected for it to use 2x the ram DURING the reload.  It's not expected
for it to continue using 2x the ram AFTER the reload is finished.

Thanks!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283317#msg-283317

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

lists@lazygranch.com
In reply to this post by adrian.hilt
On Fri, 08 Mar 2019 10:42:28 -0500
"wkbrad" <[hidden email]> wrote:

> Thanks for that info.  It's definitely harder to notice the issue on
> small servers like that.  But you are still seeing about a 50%
> increase in ram usage there by your own tests.
>
> The smallest server I've tested this on uses about 20M during the
> first start and about 50M after a reload is completely finished.
>
> Not so much of a problem for small servers but definitely a big
> problem for large ones.  That said the issue is still there on small
> servers like you've just seen.
>
> Posted at Nginx Forum:
> https://forum.nginx.org/read.php?2,283216,283318#msg-283318
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

Actually the total RAM went down after a reload for my ps_mem in the
previous email. I repeated the test just using free, which could be a
polluted test, but the RAM went down again. I also did the ps_mem test
again and total RAM was reduced.

I'm not caching in nginx, if that makes a difference.

sh-4.2# free -m
              total        used        free      shared  buff/cache   available
Mem:           1838         276         175         104        1385        1259
Swap:             0           0           0
sh-4.2# systemctl reload nginx
sh-4.2# free -m
              total        used        free      shared  buff/cache   available
Mem:           1838         272         180         104        1385        1263
Swap:             0           0           0

And repeated ps_mem test:
sh-4.2# ps_mem | grep nginx
  2.3 MiB +   3.5 MiB =   5.8 MiB       nginx (2)
sh-4.2# systemctl reload nginx
sh-4.2# ps_mem | grep nginx
  1.8 MiB +   3.1 MiB =   4.9 MiB       nginx (2)


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

adrian.hilt
In reply to this post by adrian.hilt
Hi All,

I think I haven't been clear in what I'm seeing so let's start over.  :)  I
set up a very simple test on Centos 7 with a default install of Nginx
1.12.2.  Below is exactly what I did to produce the result and it's clear to
me that Nginx is using 2x the ram than it should be using after the first
reload.  Can anyone explain why the ram usage would double after doing a
config reload?

yum update
reboot
yum install epel-release
yum install nginx
systemctl enable nginx
systemctl start nginx
yum install ps_mem vim
cd /etc/nginx/
vim vhost.template
--------------------------------------------------------------------------------
server {
listen 80;
listen [::]:80;

server_name {{DOMAIN}};

root /var/www/html;
index index.html;

location / {
try_files $uri $uri/ =404;
}
}
--------------------------------------------------------------------------------
cd conf.d
for i in $(seq -w 1 50000); do sed 's/{{DOMAIN}}/dom'${i}'.com/'
../vhost.template > dom${i}.conf; done
systemctl restart nginx
ps_mem|grep nginx
--------------------------------------------------------------------------------
 13.8 MiB + 750.7 MiB = 764.5 MiB nginx (3)
--------------------------------------------------------------------------------
systemctl reload nginx; sleep 60; ps_mem |grep nginx
--------------------------------------------------------------------------------
 27.2 MiB +   1.4 GiB =   1.5 GiB nginx (3)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283344#msg-283344

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Anoop Alias
I am able to reproduce the issue @wkbrad is reporting

[root@server1 ~]# ps_mem|head -1 && ps_mem|grep nginx
 Private  +   Shared  =  RAM used       Program 
 25.3 MiB + 119.5 MiB = 144.9 MiB       nginx (3)
[root@server1 ~]# systemctl restart nginx
[root@server1 ~]# ps_mem|head -1 && ps_mem|grep nginx
 Private  +   Shared  =  RAM used       Program 
 24.2 MiB +  58.1 MiB =  82.2 MiB       nginx (4)                       -------------------------->  notice the sharedmemory usage is half os what is used before restart
[root@server1 ~]# ps_mem|head -1 && ps_mem|grep nginx
 Private  +   Shared  =  RAM used       Program 
 23.1 MiB +  57.9 MiB =  81.0 MiB       nginx (3)                       ---------------------------> the cache loader process exits and the ram usage remain same
[root@server1 ~]# nginx -s reload                                             ---------------------------> A graceful reload is performed on Nginx
[root@server1 ~]# ps_mem|head -1 && ps_mem|grep nginx
 Private  +   Shared  =  RAM used       Program 
 15.8 MiB + 118.8 MiB = 134.5 MiB       nginx (3)                     ----------------------------> The shared RAM size doubles and stay at this value till another restart is performed



##############################################################################

I think this is because the pmap shows 2 heaps after reload whereas there is only one right after the restart , An additional heap appears after reload

[root@server1 ~]# systemctl restart nginx
[root@server1 ~]# ps aux|grep nginx
root     22392  0.0  0.7 510316 62184 ?        Ss   13:49   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
[root@server1 ~]# pmap -X 22392|head -2 && pmap -X 22392|grep heap
22392:   nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
         Address Perm   Offset Device      Inode   Size   Rss   Pss Referenced Anonymous Swap Locked Mapping
        01b10000 rw-p 00000000  00:00          0  61224 58688 17187         80     58688    0      0 [heap]


Now after the reload


[root@server1 ~]# pmap -X 20983|head -2 && pmap -X 20983|grep heap
20983:   nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
         Address Perm   Offset Device      Inode   Size    Rss   Pss Referenced Anonymous Swap Locked Mapping
        02780000 rw-p 00000000  00:00          0  61224  61220 23118      51540     61220    0      0 [heap]
        0634a000 rw-p 00000000  00:00          0  57856  55360 19138      55360     55360    0      0 [heap]
###########################################################################


On Tue, Mar 12, 2019 at 2:07 AM wkbrad <[hidden email]> wrote:
Hi All,

I think I haven't been clear in what I'm seeing so let's start over.  :)  I
set up a very simple test on Centos 7 with a default install of Nginx
1.12.2.  Below is exactly what I did to produce the result and it's clear to
me that Nginx is using 2x the ram than it should be using after the first
reload.  Can anyone explain why the ram usage would double after doing a
config reload?

yum update
reboot
yum install epel-release
yum install nginx
systemctl enable nginx
systemctl start nginx
yum install ps_mem vim
cd /etc/nginx/
vim vhost.template
--------------------------------------------------------------------------------
server {
listen 80;
listen [::]:80;

server_name {{DOMAIN}};

root /var/www/html;
index index.html;

location / {
try_files $uri $uri/ =404;
}
}
--------------------------------------------------------------------------------
cd conf.d
for i in $(seq -w 1 50000); do sed 's/{{DOMAIN}}/dom'${i}'.com/'
../vhost.template > dom${i}.conf; done
systemctl restart nginx
ps_mem|grep nginx
--------------------------------------------------------------------------------
 13.8 MiB + 750.7 MiB = 764.5 MiB       nginx (3)
--------------------------------------------------------------------------------
systemctl reload nginx; sleep 60; ps_mem |grep nginx
--------------------------------------------------------------------------------
 27.2 MiB +   1.4 GiB =   1.5 GiB       nginx (3)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283216,283344#msg-283344

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


--
Anoop P Alias 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Possible memory leak?

Maxim Dounin
In reply to this post by adrian.hilt
Hello!

On Mon, Mar 11, 2019 at 04:37:50PM -0400, wkbrad wrote:

> I think I haven't been clear in what I'm seeing so let's start over.  :)  I
> set up a very simple test on Centos 7 with a default install of Nginx
> 1.12.2.  Below is exactly what I did to produce the result and it's clear to
> me that Nginx is using 2x the ram than it should be using after the first
> reload.  Can anyone explain why the ram usage would double after doing a
> config reload?

As I already tried to explained earlier in this thread, this is a
result of two things:

1) How nginx allocates memory when doing a configuration reload:
it creates a new configuration first, and then frees the old one.

2) How system memory allocator works.  Usually it cannot return
memory to the system if there are any remaining allocations above
the freed memory regions.  In some cases you can configure system
allocator to use mmap(), so it will be possible to free such
allocations, but it may a be a bad idea for other reasons.

As a result, if large amount of memory is used solely for the
configuration structures, memory occupied by the nginx master
process from the system point of view is roughly doubled after a
configuration reload.

Note that the memory in question is not leaked.  It is properly
freed by nginx, and it is available for future allocations within
nginx.  In worker processes, this memory will be used for various
run-time allocations, such as request buffers and so on.  In the
master process, this memory will be used on further configuration
reloads, so the master process will not grow any further.

If the amount of memory used for configuration structures is a
problem, you may want to re-think your configuration approach.  In
particular, large virtual hosting providers are known to use nginx
with small number of server{} blocks serving many different
domains.  Alternatively, you may want to build nginx with less
modules compiled in, as each module usually allocates at least
basic configuration structures in each server{} / location{} even
if not used.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
12