Too many deleted open files in proxy_temp_path

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Too many deleted open files in proxy_temp_path

zakirenish
Hi everyone,

We are using nginx as reverse proxy to cache static content for a moodle LMS
site. The caching part is working fine but we started facing the cache path
disk full issues. When we checked using du, it was reporting very low used
space.Then we checked lsof output we found too many deleted file entries. We
realized, that file gets deleted but it holds the FD and disk space is never
released until we restart the nginx server. Most of the files were from
proxy_temp_path location. We tried this with tmpfs and normal disk but the
end result was same.

# nginx -v
nginx version: nginx/1.10.3 (Ubuntu)

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

=================== df output =================
# df -h /mnt/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       252G  8.7G  231G   4% /mnt

# du -sh /mnt/nginx/
4.0K    /mnt/nginx/

================== nginx.conf ==================
user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 1024;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 0;
        types_hash_max_size 2048;
        client_max_body_size 2048M;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json
application/javascript text/xml application/xml application/xml+rss
text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

==================== SSL site config =====================
log_format rt_cache '$remote_addr - $upstream_cache_status [$time_local]  '
                    '"$request" $status "$sent_http_content_type"
$sent_http_content_encoding $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';
proxy_cache_path /var/cache/tmpfs levels=1:2 keys_zone=my_cache:1000m
max_size=18g inactive=1d use_temp_path=off;;
proxy_temp_path    /mnt/nginx;

server {
        listen 443;
        server_name xxx.yyy.com;
        ssl_certificate /etc/apache2/ssl/xxx.yyy.com-nginx.crt;
        ssl_certificate_key /etc/apache2/ssl/xxx.yyy.com.key;

        ssl on;
        ssl_session_cache builtin:1000 shared:SSL:10m;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers
HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/xxx.yyy.com-access.log rt_cache;
        #access_log off;
        proxy_buffer_size                  512k;
        proxy_buffers                      4 512k;
        proxy_headers_hash_max_size        1024;

        location /nginx-status {
                stub_status on;
                allow all;
        }

        location ~*
\.(?:ico|jpg|css|png|js|swf|woff|eot|svg|ttf|html|gif|jpeg)$ {
                aio threads;
                proxy_cache my_cache;
                add_header X-Proxy-Cache $upstream_cache_status;
                expires 7d;
                proxy_ignore_headers Cache-Control;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass http://localhost;
                proxy_set_header Connection "";
                proxy_connect_timeout 600s;
                proxy_send_timeout 600s;
                proxy_read_timeout 600s;
                send_timeout 600s;
                proxy_http_version 1.1;
        }

        location / {
                #proxy_buffering off;
                proxy_ignore_headers Set-Cookie;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass http://localhost;
                proxy_set_header Connection "";
                proxy_connect_timeout 600s;
                proxy_send_timeout 600s;
                proxy_read_timeout 600s;
                send_timeout 600s;
                proxy_http_version 1.1;
        }
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287605,287605#msg-287605

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Too many deleted open files in proxy_temp_path

J.R.
> # nginx -v
> nginx version: nginx/1.10.3 (Ubuntu)

The last update for that version was over 3 years ago...

Try updating to 1.17.9...
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Too many deleted open files in proxy_temp_path

zakirenish
Oh.. yes. i thought ubuntu gives latest package. Okay i'll try that and
update here.

Thank You for pointing.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287605,287607#msg-287607

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Too many deleted open files in proxy_temp_path

Maxim Dounin
In reply to this post by zakirenish
Hello!

On Fri, Apr 10, 2020 at 11:26:17AM -0400, patademahesh wrote:

> We are using nginx as reverse proxy to cache static content for a moodle LMS
> site. The caching part is working fine but we started facing the cache path
> disk full issues. When we checked using du, it was reporting very low used
> space.Then we checked lsof output we found too many deleted file entries. We
> realized, that file gets deleted but it holds the FD and disk space is never
> released until we restart the nginx server. Most of the files were from
> proxy_temp_path location. We tried this with tmpfs and normal disk but the
> end result was same.
>
> # nginx -v
> nginx version: nginx/1.10.3 (Ubuntu)

Note that when writing temporary files when proxying (without
caching, but there is no caching configured in "location /"), it
is quite normal that temporary files are unlinked (deleted) right
after creation and cannot be seen by "du".  This way temporary
files are automatically removed by the system as long as a file is
closed - or if nginx is killed or crashes.

If you think that temporary files are not removed from disk even
if corresponding client connections are closed - this might
indicate a socket leak.  Usually, socket leaks can be seen by
other symptoms as well - such as connections in the CLOSED state
as shown by "netstat -an", or "open socket ... left in connection ..."
alerts during graceful shutdown of worker processes.  Given
that you are using an ancient nginx version, socket leaks might be
the case - there are at least some fixed since 1.10.3, mostly in
HTTP/2.

On the other hand, if deleted files is the only symptom you are
seeing, it might simply indicate that corresponding client
connection is still open and the client is slowly downloading the
response in question.  If that's true, you may want to either use
a filesystem with more space for temporary files, or tune
proxy buffering, see here:

http://nginx.org/r/proxy_max_temp_file_size

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Too many deleted open files in proxy_temp_path

zakirenish
In reply to this post by zakirenish
[[ Written a few days ago, but email bounced ]]
[[ Had to re-register to re-post. ]]
[[ sorry if this is repeating somebody else ]]

Hi all,

First time poster here, doing it on a lazy morning.

Have you tried to halt and start the service?

Linux filesystem doesn't release deleted files when the process still has
process and pointer references to them (i.e. most likely the reason for this
inquiry). Probably a bug.

Stop/start the service and all child threads could fix your filesystem
issue.

If you already have more than one Linux servers, each with  distinct IP
addresses, and your DNS with A record to each of them (à la round-robin),
with no HTTP client "stickyness" affinity to one node (e.g. cookie or way to
tell that one particular Browser must only go to node0). You won't create
downtime.

Hope this helps. Stay safe.

Renoir--

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287605,287623#msg-287623

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx