Multiple Cache Manager Processes or Threads

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple Cache Manager Processes or Threads

ashuai
Hello,

I have an issue with the cache manager and the way I use it.
When I configure 2 different caches zones, one very huge and one very fast,
the cache manager can't delete files quickly enough and lead to a partition
full.

For example:
proxy_cache_path /mnt/hdd/cache levels=1:2:2 keys_zone=cache_hdd:40g
max_size=40000g inactive=5d;
proxy_cache_path /mnt/ram/cache levels=1:2   keys_zone=cache_ram:300m
max_size=300g inactive=1h;

On the beginning, ram cache is correctly purge around 40GB (+/- Input
bandwidth*10sec) , but when the hdd cache begins to fill up, ram cache
growing over 50GB. I think the cache manager is stuck by the slowness of the
filesystem / hardware.

I can fix this by using 2 nginx on the same machine, one configured as ram
cache, the other as hdd cache; but I wonder if it would be possible to
create a cache manager process for each proxy_cache_path directive.

Thank in advance.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277597,277597#msg-277597

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Cache Manager Processes or Threads

ashuai
Sorry, I gave wrong values:

On the beginning, ram cache is correctly purge around 300GB (+/- Input
bandwidth*10sec) , but when the hdd cache begins to fill up, ram cache
growing over 320GB.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277597,277598#msg-277598

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Cache Manager Processes or Threads

Maxim Dounin
In reply to this post by ashuai
Hello!

On Thu, Nov 30, 2017 at 12:20:19PM -0500, traquila wrote:

> I have an issue with the cache manager and the way I use it.
> When I configure 2 different caches zones, one very huge and one very fast,
> the cache manager can't delete files quickly enough and lead to a partition
> full.
>
> For example:
> proxy_cache_path /mnt/hdd/cache levels=1:2:2 keys_zone=cache_hdd:40g
> max_size=40000g inactive=5d;
> proxy_cache_path /mnt/ram/cache levels=1:2   keys_zone=cache_ram:300m
> max_size=300g inactive=1h;
>
> On the beginning, ram cache is correctly purge around 40GB (+/- Input
> bandwidth*10sec) , but when the hdd cache begins to fill up, ram cache
> growing over 50GB. I think the cache manager is stuck by the slowness of the
> filesystem / hardware.
>
> I can fix this by using 2 nginx on the same machine, one configured as ram
> cache, the other as hdd cache; but I wonder if it would be possible to
> create a cache manager process for each proxy_cache_path directive.

Which nginx version you are using?

With nginx 1.11.5+, there are manager_files / manager_sleep /
manager_threshold parameters you may want to play with, see
http://nginx.org/r/proxy_cache_path.  These parameters allows
limiting cache manager's work on a particular cache to some finite
time, and therefore help to better maintain specified max_size of
other caches.

If you are using an older version, an upgrade to the recent
version might help even without further tuning, as older versions
do not limit cache manager's work on a particular cache at all.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Cache Manager Processes or Threads

ashuai
Thank you for your answer,
I am using an old version (1.8.1).
I will try to upgrade to 1.12 and check if it solve my problem.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277597,277616#msg-277616

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx