Nginx 1.12.1 Memory Consumption

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Nginx 1.12.1 Memory Consumption

Matthew Smith
Hello,

I have encountered what I consider to be an interesting behavior. We have Nginx 1.12.1 configured to do SSL termination as well as reverse proxy. Whenever there is a traffic spike (300 req/s > 1000 req/s, 3k active connections > 20k active connections), there is a corresponding spike in Nginx memory consumption. In this case 500M > 8G across 10 worker processes. What is interesting is that Nginx never seems to release this memory after the traffic returns to normal. Is this expected? What is Nginx using this memory for? Is there a configuration that will rotate the workers based on some metric in order to return memory to the system?

Requests per second:

Active connections:

Total Nginx memory usage:

Thanks,

Matt

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx 1.12.1 Memory Consumption

Maxim Dounin
Hello!

On Wed, Mar 14, 2018 at 05:05:42PM +0000, Matthew Smith wrote:

> I have encountered what I consider to be an interesting behavior. We have
> Nginx 1.12.1 configured to do SSL termination as well as reverse proxy.
> Whenever there is a traffic spike (300 req/s > 1000 req/s, 3k active
> connections > 20k active connections), there is a corresponding spike in
> Nginx memory consumption. In this case 500M > 8G across 10 worker
> processes. What is interesting is that Nginx never seems to release this
> memory after the traffic returns to normal. Is this expected? What is Nginx
> using this memory for? Is there a configuration that will rotate the
> workers based on some metric in order to return memory to the system?

All memory allocated for request handling nginx returns to
the allocator once a request is closed.  If you see memory not
actually released to the system, this may be one of the following:

- System allocator fails to return memory to the system.  Tuning
  system allocator might help here.

- Given that you are using nginx for SSL termination, this may be
  an OpenSSL [mis]feature called "buf-freelists".  It implies caching
  of up to 2 megabytes of allocated buffers per SSL context, and
  this may be a problem if there are multiple server{} blocks with
  SSL enabled.  Fix is to recompile OpenSSL with the
  "no-buf-freelists" option, or upgrade to OpenSSL 1.1.x where
  this feature is disabled.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx 1.12.1 Memory Consumption

Peter Booth
In reply to this post by Matthew Smith
Two questions: 

1. how are you measuring memory consumption?
2. How much physical memory do you have on your host?


Assuming that you are running on Linux, can you use pidstat -r -t -u -v -w -C “nginx”
to confirm the process’s memory consumption, 

and cat /var/meminfo to view a detailed description of how memory is being used onto entire host.


On Mar 14, 2018, at 1:05 PM, Matthew Smith <[hidden email]> wrote:

Hello,

I have encountered what I consider to be an interesting behavior. We have Nginx 1.12.1 configured to do SSL termination as well as reverse proxy. Whenever there is a traffic spike (300 req/s > 1000 req/s, 3k active connections > 20k active connections), there is a corresponding spike in Nginx memory consumption. In this case 500M > 8G across 10 worker processes. What is interesting is that Nginx never seems to release this memory after the traffic returns to normal. Is this expected? What is Nginx using this memory for? Is there a configuration that will rotate the workers based on some metric in order to return memory to the system?

Requests per second:

Active connections:

Total Nginx memory usage:

Thanks,

Matt
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx 1.12.1 Memory Consumption

Matthew Smith
Hello,

The host has 30G total memory. Nginx usage is being measured by summing the Pss values from /proc/$pid/smaps for all worker processes.

Do you have any suggestions for differentiating between the two issues that might prevent memory from being returned to the system?

Thanks!

On Thu, Mar 15, 2018 at 1:06 PM Peter Booth <[hidden email]> wrote:
Two questions: 

1. how are you measuring memory consumption?
2. How much physical memory do you have on your host?


Assuming that you are running on Linux, can you use pidstat -r -t -u -v -w -C “nginx”
to confirm the process’s memory consumption, 

and cat /var/meminfo to view a detailed description of how memory is being used onto entire host.


On Mar 14, 2018, at 1:05 PM, Matthew Smith <[hidden email]> wrote:

Hello,

I have encountered what I consider to be an interesting behavior. We have Nginx 1.12.1 configured to do SSL termination as well as reverse proxy. Whenever there is a traffic spike (300 req/s > 1000 req/s, 3k active connections > 20k active connections), there is a corresponding spike in Nginx memory consumption. In this case 500M > 8G across 10 worker processes. What is interesting is that Nginx never seems to release this memory after the traffic returns to normal. Is this expected? What is Nginx using this memory for? Is there a configuration that will rotate the workers based on some metric in order to return memory to the system?

Requests per second:

Active connections:

Total Nginx memory usage:

Thanks,

Matt
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx 1.12.1 Memory Consumption

Maxim Konovalov
Hi Matthew,

On 19/03/2018 17:38, Matthew Smith wrote:
> Hello,
>
> The host has 30G total memory. Nginx usage is being measured by
> summing the Pss values from /proc/$pid/smaps for all worker processes.
>
> Do you have any suggestions for differentiating between the two
> issues that might prevent memory from being returned to the system?
>
Did you read Maxim Dounin's comment about OpenSSL freelists issue?

http://mailman.nginx.org/pipermail/nginx/2018-March/055869.html

--
Maxim Konovalov
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx 1.12.1 Memory Consumption

Peter Booth
In reply to this post by Matthew Smith
I’d use wrk2 or httperf to recreate a spike that hits an http endpoint. If you don’t see a spike  but see one with https then you know ssl is one factor.

It’s also interesting that this happens st around 23000 connections. If you reduce workr count to one or two
And still see max connections around 23000 then it looks another factor is tcp resources.

Sent from my iPhone

On Mar 19, 2018, at 10:38 AM, Matthew Smith <[hidden email]> wrote:

Hello,

The host has 30G total memory. Nginx usage is being measured by summing the Pss values from /proc/$pid/smaps for all worker processes.

Do you have any suggestions for differentiating between the two issues that might prevent memory from being returned to the system?

Thanks!

On Thu, Mar 15, 2018 at 1:06 PM Peter Booth <[hidden email]> wrote:
Two questions: 

1. how are you measuring memory consumption?
2. How much physical memory do you have on your host?


Assuming that you are running on Linux, can you use pidstat -r -t -u -v -w -C “nginx”
to confirm the process’s memory consumption, 

and cat /var/meminfo to view a detailed description of how memory is being used onto entire host.


On Mar 14, 2018, at 1:05 PM, Matthew Smith <[hidden email]> wrote:

Hello,

I have encountered what I consider to be an interesting behavior. We have Nginx 1.12.1 configured to do SSL termination as well as reverse proxy. Whenever there is a traffic spike (300 req/s > 1000 req/s, 3k active connections > 20k active connections), there is a corresponding spike in Nginx memory consumption. In this case 500M > 8G across 10 worker processes. What is interesting is that Nginx never seems to release this memory after the traffic returns to normal. Is this expected? What is Nginx using this memory for? Is there a configuration that will rotate the workers based on some metric in order to return memory to the system?

Requests per second:

Active connections:

Total Nginx memory usage:

Thanks,

Matt
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx