Limiting total disk space used by cache + temp files

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Limiting total disk space used by cache + temp files

rnburn
Hi,

I'm trying to limit the overall disk usage for the cache + any temp files.

I have specified:
                        proxy_cache_path  /tmp/cache max_size=50m
use_temp_path=off;
                        proxy_max_temp_file_size 1m;

Yet, what I observe is that the temp file will grow to the full size of the
file being retrieved from the upstream, ignoring the value of
proxy_max_temp_file_size, and ignoring the max_size for the cache. Only
after the file is brought over completely does the cache manager kick in and
delete something older from the cache to bring it back in line with the 50mb
limit.

For example, if I had in the cache file1 (20 MB) and file2 (20 MB), and I
receive a request for file3 which is 20 MB, the total size of the /tmp/cache
directory will grow to 60 MB. Only after file3 is received in its entirety
will file1 be deleted bringing the size back down to 40 MB.

I don't mind setting use_temp_path=on, but in that case as well the temp
file grows beyond the proxy_max_temp_file_size.

Any advice?

Michael

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274065,274065#msg-274065

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Limiting total disk space used by cache + temp files

Maxim Dounin
Hello!

On Thu, May 04, 2017 at 02:50:53AM -0400, Michael Corn wrote:

> I'm trying to limit the overall disk usage for the cache + any temp files.
>
> I have specified:
>                         proxy_cache_path  /tmp/cache max_size=50m
> use_temp_path=off;
>                         proxy_max_temp_file_size 1m;
>
> Yet, what I observe is that the temp file will grow to the full size of the
> file being retrieved from the upstream, ignoring the value of
> proxy_max_temp_file_size, and ignoring the max_size for the cache. Only
> after the file is brought over completely does the cache manager kick in and
> delete something older from the cache to bring it back in line with the 50mb
> limit.
>
> For example, if I had in the cache file1 (20 MB) and file2 (20 MB), and I
> receive a request for file3 which is 20 MB, the total size of the /tmp/cache
> directory will grow to 60 MB. Only after file3 is received in its entirety
> will file1 be deleted bringing the size back down to 40 MB.
>
> I don't mind setting use_temp_path=on, but in that case as well the temp
> file grows beyond the proxy_max_temp_file_size.
>
> Any advice?

What you are trying to do is not going to work.

First of all, proxy_max_temp_file_size is ignored when using
cache.  Quoting http://nginx.org/r/proxy_max_temp_file_size:

: This restriction does not apply to responses that will be cached
: or stored on disk.

As for the cache max_size parameter, it is maintained by the cache
manager process, and it does so after the files are already in
the cache.  Quoting http://nginx.org/r/proxy_cache_path:

: The special "cache manager" process monitors the maximum cache
: size set by the max_size parameter. When this size is exceeded, it
: removes the least recently used data.

Moreover, cache manager may be busy with other tasks or just
sleeping, so there is inevitable difference between max_size
configured and possible maximum size of the cache.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx