Problem with proxy_cache_path and limit

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with proxy_cache_path and limit

sonpg
nginx -v
nginx version: nginx/1.13.7

# Server globals
user                    www-data;
worker_processes        auto;
worker_rlimit_nofile    65535;
error_log               /var/log/nginx/error.log crit;
pid                     /var/run/nginx.pid;


# Worker config
events {
        worker_connections  1024;
        use                 epoll;
        multi_accept        on;
}


http {
   
        proxy_cache_path /home/admin/cachemaster levels=1:2 keys_zone=my_cache:100m
max_size=1g inactive=60m use_temp_path=off;
   
   }

server {
  listen       107.170.204.190:443;
  server_name  sf1.example www.sf1.example;
  ssl         on;
    ssl_certificate      /home/admin/conf/web/ssl.sf1.example.pem;
    ssl_certificate_key  /home/admin/conf/web/ssl.sf1.example.key;
    error_log  /var/log/apache2/domains/sf1.example.error.log error;

  location / {
    proxy_cache my_cache;
    proxy_set_header  X-Real-IP  $remote_addr;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 3;
        proxy_cache_valid 200 301 7d;
    proxy_pass <a href="https://example:443;">https://example:443;
    proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
   
  }
}

but the folder /home/admin/cachemaster fills up beyond 1gb, what am i
missing

Thanks

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277816#msg-277816

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

Zhang Chao
Hello!

> nginx -v 
> nginx version: nginx/1.13.7 

> # Server globals 
> user www-data; 
> worker_processes auto; 
> worker_rlimit_nofile 65535; 
> error_log /var/log/nginx/error.log crit; 
pid /var/run/nginx.pid; 
>
>
# Worker config 
events { 
worker_connections 1024; 
use epoll; 
multi_accept on; 



http { 

proxy_cache_path /home/admin/cachemaster levels=1:2 keys_zone=my_cache:100m 
max_size=1g inactive=60m use_temp_path=off; 

}

server { 
listen 107.170.204.190:443; 
server_name sf1.example www.sf1.example
ssl on; 
ssl_certificate /home/admin/conf/web/ssl.sf1.example.pem; 
ssl_certificate_key /home/admin/conf/web/ssl.sf1.example.key; 
error_log /var/log/apache2/domains/sf1.example.error.log error; 

location / { 
proxy_cache my_cache; 
proxy_set_header X-Real-IP $remote_addr; 
proxy_cache_revalidate on; 
proxy_cache_min_uses 3; 
proxy_cache_valid 200 301 7d; 
proxy_pass https://example:443
proxy_ignore_headers X-Accel-Expires Expires Cache-Control; 

} 


but the folder /home/admin/cachemaster fills up beyond 1gb, what am i 
missing 

Thanks 


I think is 1GB rather than 1Gb? since the minimum unit is bytes. how long has this be going on? or just a sudden burst?


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

sonpg
Hi Mate

i tried both

 nginx -t
nginx: [emerg] invalid max_size value "max_size=1GB" in
/etc/nginx/nginx.conf:109
nginx: configuration file /etc/nginx/nginx.conf test failed
root@nyc1:~# nginx -t
nginx: [emerg] invalid max_size value "max_size=1Gb" in
/etc/nginx/nginx.conf:109
nginx: configuration file /etc/nginx/nginx.conf test failed
root@nyc1:~#

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277818#msg-277818

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

sonpg
In reply to this post by sonpg
nothing, even tried 1024m

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277819#msg-277819

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

Maxim Dounin
In reply to this post by sonpg
Hello!

On Mon, Dec 18, 2017 at 09:14:43PM -0500, erick3k wrote:

[...]

> proxy_cache_path /home/admin/cachemaster levels=1:2 keys_zone=my_cache:100m
> max_size=1g inactive=60m use_temp_path=off;

[...]

> but the folder /home/admin/cachemaster fills up beyond 1gb, what am i
> missing

Could you please clarify what exactly do you observe?

Note that "max_size=" is not a strict limit, but rather a
threshold when nginx will start cleaning old cache items.  It
doesn't guarantee that the size of the folder will not exceed the
limit specified - rather, nginx will take actions when the size
exceeds the limit.

Note well that with "use_temp_path=off" temporary files are placed
into the cache directory, but they will not be counted against
max_size.

That is, with the specified configuration it is expected that the
folder can contain more than 1gb under load.  Especially if there
are large cacheable responses in flight, which will occupy space
as temporary files.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

sonpg
Interesting,

what i observe is the folder cachemaster going over 1gb, way over until it
fills the hard drive.

so what is the correct way to limit the cache size used by nginx?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277831#msg-277831

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

sonpg
In reply to this post by Maxim Dounin
removing use_temp_path=off max_size is literally ignored, still fills the
hard drive just in default nginx temp path

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277832#msg-277832

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

Maxim Dounin
In reply to this post by sonpg
Hello!

On Tue, Dec 19, 2017 at 12:48:41PM -0500, erick3k wrote:

> what i observe is the folder cachemaster going over 1gb, way over until it
> fills the hard drive.

Please define "until it fills the hard drive".  It is ok if you
have only 1gb or 2gb free, but it is certainly not ok if it fills
a free 200gb drive.

Additional question is about temporary files.  As previously
explained, temporary files are not counted against the cache
max_size limit, yet will occupy space in the same folder due to
"use_temp_path=off".  Consider using "use_temp_path=on" (the
default) with a proxy_temp_path pointing to a different folder to
test if the space is occupied by the cache or temporary files.

You may also want to take a look into the error log, to see if
there are any crit/alert/emerg errors.  Also you may want to
check what the nginx cache manager process is doing.

> so what is the correct way to limit the cache size used by nginx?

The "proxy_cache_path ... max_size=..." is the correct way.  
Though it should be understood that it doesn't limit the size of
the folder, but rather instruct nginx to remove old cache items if
the size of the cache is above the limit.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

sonpg
and that is done by the cache manager correct? when is this function
triggered ? i suspect the cache manager is not working, how can i tell if is
loaded?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277836#msg-277836

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

Maxim Dounin
Hello!

On Tue, Dec 19, 2017 at 01:31:10PM -0500, erick3k wrote:

> and that is done by the cache manager correct? when is this function
> triggered ? i suspect the cache manager is not working, how can i tell if is
> loaded?

Try something like "ps -alx", cache manager process title will be
"nginx: cache manager process".

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

sonpg
Yep is there, so am still back to 0, nginx is still filling up the hard disk
please help

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277816,277840#msg-277840

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Problem with proxy_cache_path and limit

Maxim Dounin
Hello!

On Tue, Dec 19, 2017 at 04:13:10PM -0500, erick3k wrote:

> Yep is there, so am still back to 0, nginx is still filling up the hard disk
> please help

I believe I've already asked you to define "filling up".  And you
were already asked to do some diagnostic steps, including:

- check if space is occupied by temporary files or the cache
  itself;

- check error logs;

- check what cache manager process is doing.

If you really want others to be able to help, you may want to do
what you were asked to and share the results.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx