I need your help, I have a problem with my Nginx server : all the Nginx
processes are consuming a huge amount of memory and it sets off the OOM
I don't understand why Nginx uses this much memory, it seems odd, I expected
the CPU to be the first problem instead of memory.
My Nginx server is just a reverse proxy, to all kind of backend. SSL can be
offloaded or not, it depends on the backend.
I have a huge amount of vhosts with SSL on this Nginx (~15K) and the server
have a lot of trafic (for me at least !) .
The /stub_status tell me the following thing at this moment :
Active connections: 3867
server accepts handled requests
2350039 2350039 5489648
Reading: 0 Writing: 332 Waiting: 3605
[root@nginxsrv nginx]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@nginxsrv nginx]# nginx -V
nginx version: nginx/1.17.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.1.0k 28 May 2019
TLS SNI support enabled
configure arguments: --add-module=/usr/local/src/ngx_brotli
--with-threads --with-openssl=/usr/local/src/openssl --with-http_v2_module
--with-http_flv_module --with-ipv6 --with-http_mp4_module
The server have a lot of ram, 120G and the CPU is an Intel Xeon E5-2697 CPU.
The CPU is not used that much.
A typical vhost file is quite simple, it's just a proxy_pass to a backend,
with some hack to do a retry is we first get and error message (based on
proxy_intercept_errors). http2 is on. I don't use any caches.
Things I have tested or I suspect to be an issue :
- I reduced the worker_processes to 6. I had issue with the frequent
reloading of Nginx with an upper value. I suspect that it's more difficult
for Nginx to reload properly if a lot of processes are spawned (correct me
if i'm wrong)
- I have gzip on and gzip_proxied to any. I suspect the compression to be an
issue, I reduced the buffers value to gzip_buffers 32 4k; and
gzip_comp_level to 1
- I reduced the ssl_session_cache from 10m to 1m
- I reduced the keep_alive timeout too (30)
- Reloading Nginx often seems to be a problem too, the memory usage seems
higher during the reloading task. The reloading is quite slow too, probably
because of the number of vhosts file to load (if someone know how to speed
up this, it interest me !)
- I have a lot of limit_conn configuration
- ssl_stapling is off, I had issue with it with the DNS resolution.
Do you have idea on what can cause a high memory usage on Nginx ?
> A typical vhost file is quite simple, it's just a proxy_pass to a backend,
> with some hack to do a retry is we first get and error message (based on
> proxy_intercept_errors). http2 is on. I don't use any caches.
Are you running PHP FPM? If so, check all your process manager
settings. I've seen when misconfigured that can eat up a ton of ram
I've always left the worker_processes to auto, as the docs say it
should equal the number of cores of the system. Though it sounds like
6 is adequate if you are not experiencing high CPU load. Though I'm
not sure if more cores would allow for faster reading / processing of
your configuration files, maybe bump up to 12 since that is the number
of actual cores for that CPU.
Why do you suspect the gzip module? If anything I would think it would
eat CPU cycles, not memory... Unless you are trying to compress some
*really* large content. I know there is a minimum size setting, I
don't know if there is a maximum one too... I see you have the brotli
compression module installed too... Have you set brotli_comp_level
(among the other settings) ???
The point of the SSL session cache is to reduce CPU usage by having to
constantly re-negotiate a SSL session... Think about how long a person
will read a page on your site(s)... You would really want at minimum
like 5 minutes, otherwise the cache really isn't saving you any
processing and only eating up memory. Depending on your log level, the
error log would let you know if your cache gets full and needs to be
increased (or the TTL lowered) as you would see something like, "could
not allocate new session in SSL session shared cache...."
Are you using a shared cache across *all* your virtual hosts for the
ssl_session_cache & limit_conn, or does each vhost have their own? I
could see how that would eat up memory real quick with the number of
vhosts you have.
Whenever possible you should try to use settings at the http level so
that it's only creating that instance once. If you are using the same
thing under every server directive, then each server instance has its
own copy and thus using more memory.
I'm running PHP-FPM for the API I created that dynamicaly create Nginx
vhosts (that's why I need to reload Nginx quite often).
It's not really use, like < 1000 requests / day.
99.999% of the trafic is just is associated with the 15K domain and Nginx
just do a proxy_pass to a backend server.
Nginx listen to 100+ IPs (if that's important).
For the GZIP module, it's just a guest, for me it's logical that GZIP will
use a lot of CPU but I thought that it could use some memory too for the
buffering or some sort of cache.
I'm aware that it's unlikely the problem.
I used to have the ssl_session_cache on each vhosts but I moved it to the
The memory consuption haven't change.
But, I have some limit_conn in each vhosts (included) so I think you're onto
something with this !
Bellow the config I can share, with some comment :
### Begin of nginx.conf ###
worker_processes 6; # Doing some test on this, previous value was 8, also
error_log /var/log/nginx/err.crit error;
You should set your SSL settings at the 'http' level, including
default certificates. Then in each 'server' directive you can
re-define the certificates when necessary.
You can also set all the common proxy_set_header vars at the 'http'
level instead of duplicating for each server. Maybe also look into
using a common upstream zone instead of always defining the proxy.
Finally, just because I would check your PHP-FPM settings (not in
nginx)... Make sure that at the very least:
pm = ondemand
pm.max_request = 1000
I've seen at times where not setting a max_request leads to eating up
ram in Apache, could possibly be an issue with Nginx.
Nothing else really stands out. I would just try to decrease buffers
where you can, especially those that are created for each server
instance. With all the vhosts you have that can really add up quickly.