massive deleted open files in proxy cache

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

massive deleted open files in proxy cache

Marco Lenhardt
Hi!

We are useing Ubuntu 16.04 with nginx version 1.10.0-0ubuntu0.16.04.4.

nginx.conf:

user  nginx;
worker_processes  auto;
worker_rlimit_nofile 20480; # ulimit open files per worker process

events {
         # Performance
         worker_connections  2048; # openfilelimits beachten
         multi_accept on;
         use epoll;
}

http {
         open_file_cache max=10000 inactive=1d;
         open_file_cache_valid 1d;
         open_file_cache_min_uses 1;
         open_file_cache_errors off;

         proxy_cache_path /var/cache/nginx/proxy_cache
levels=1:2 keys_zone=html_cache:30m max_size=8192m inactive=4h
use_temp_path=off;
         proxy_cache_path /var/cache/nginx/wordpress_cache
levels=1:2 keys_zone=wordpress_cache:1m max_size=256m inactive=24h
use_temp_path=off;

         proxy_buffer_size          128k;
         proxy_buffers              4 256k;
         proxy_busy_buffers_size    256k;
}


# df -h |grep cache
tmpfs      300M   74M  227M  25% /var/cache/nginx/wordpress_cache
tmpfs      9.0G  4.1G  5.0G  45% /var/cache/nginx/proxy_cache

# df -i |grep cache
tmpfs    20632978    5457 20627521    1% /var/cache/nginx/wordpress_cache
tmpfs    20632978   74613 20558365    1% /var/cache/nginx/proxy_cache

# grep cache /etc/fstab
tmpfs /var/cache/nginx/proxy_cache/ tmpfs
rw,uid=109,gid=117,size=9G,mode=0755 0 0
tmpfs /var/cache/nginx/wordpress_cache/ tmpfs
rw,uid=109,gid=117,size=300M,mode=0755 0 0


# free -m
               total        used        free      shared  buff/cache
available
Mem:         161195      112884        1321        4626       46988
  42519
Swap:          3903         211        3692


Problem:
========
We got massive open file handles from nginx user located inside
proxy_cache_path with status deleted:

# lsof -E -M -T > lsof.`date +"%Y%d%m-%H%M%S"`.out

nginx  3613  nginx   48r   REG   0,42  148664   29697
/var/cache/nginx/proxy_cache/temp/5/23/04ca8002edd2daa3c538ada5b202d6eb
(deleted)
nginx  3613  nginx   50r   REG   0,42  161618   19416
/var/cache/nginx/proxy_cache/temp/1/40/d8f0a3563d18af4fbf43242e19283b15
(deleted)

# grep nginx lsof.20172002-085328.out |wc -l
69003

# grep nginx lsof.20172002-085328.out |grep deleted |wc -l
36312

# grep nginx lsof.20172002-085328.out |grep deleted |grep
"/var/cache/nginx/proxy_cache/temp" |wc -l
32004

The most of the 36k deleted files are located inside the temp cache folder.

My question is, why we got so much deleted files inside the cache. Why
is nginx not freeing these files?
Is there a problem with the proxy_cache_path option use_temp_path=off ?

I am in worry that the cache file system will be filled up with deleted
files, and will reach open files limits.

Or do we have a nginx misconfiguration somewhere?

Additionally we were often visited from the oom-killer, (in face of 20G
free memory). If I restart nginx before we reach 80k open ngnix files,
the oom-killer will not visit us!

Has anybody simmilar findings regarding deleted open files?

br,
Marco
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: massive deleted open files in proxy cache

satay
Have you been able to solve the issue? We're having the same problem after
upgrading 1.12.2 to 1.14

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,272519,280189#msg-280189

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: massive deleted open files in proxy cache

Peter Booth
Sounds weird.

1. It doesn’t make sense for your cache to be on a tmpfs share. Better to use s physical disk allow Linux ‘s page csche to do its job

2. How big are the files in the larger cache?  Min/median/max?

Sent from my iPhone

> On Jun 20, 2018, at 7:38 AM, rihad <[hidden email]> wrote:
>
> Have you been able to solve the issue? We're having the same problem after
> upgrading 1.12.2 to 1.14
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,272519,280189#msg-280189
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx