worker_connections allocates a lot of memory

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

worker_connections allocates a lot of memory

vergil
Hi,

I noticed that worker_connections while is defined as the maximum number of
connections per worker, nginx pre-allocated enough memory to handle all
possible worker_connections, even before they are actually needed.

For example, setting worker_connections to 10485760 causes nginx to take
4.3GB of memory upon init.

Is this how its supposed to be?
Is there a way to tell nginx to only allocate memory as needed?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288358,288358#msg-288358

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: worker_connections allocates a lot of memory

Marcin Wanat

Hi,

I noticed that worker_connections while is defined as the maximum number of
connections per worker, nginx pre-allocated enough memory to handle all
possible worker_connections, even before they are actually needed.

For example, setting worker_connections to 10485760 causes nginx to take
4.3GB of memory upon init.

Is this how its supposed to be?
Is there a way to tell nginx to only allocate memory as needed?

I think you do not need to worry about it.
You will run out of available ports, file descriptors or system memory needed to allocate
these resources way before you will notice this.

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: worker_connections allocates a lot of memory

Maxim Dounin
In reply to this post by vergil
Hello!

On Tue, Jun 16, 2020 at 03:42:04AM -0400, shaharmor wrote:

> I noticed that worker_connections while is defined as the maximum number of
> connections per worker, nginx pre-allocated enough memory to handle all
> possible worker_connections, even before they are actually needed.
>
> For example, setting worker_connections to 10485760 causes nginx to take
> 4.3GB of memory upon init.
>
> Is this how its supposed to be?

Yes.

> Is there a way to tell nginx to only allocate memory as needed?

No.

Connection structures are small and cannot be freed, so nginx
allocates them all on startup.  This way it avoids memory
fragmentation, and makes connnection management easier and faster.

Note well that you cannot really use connection structures in
nginx without corresponding kernel structures such as sockets and
appropriate buffers, so using arbitrary large worker_connections
value does not make sense.  If you are seeing that it takes
significant amount of memory, most likely you've configured it
incorrectly.

For example, some relevant numbers can be seen in the following
blog post by WhatsApp:

https://blog.whatsapp.com/1-million-is-so-2011

Numbers in suggest that about 64G of memory is needed to handle 2
millions of connections.  Scaling this to 10 millions as in your
example gives about 256G, so 4G for worker_connections shouldn't
be noticeable.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx