Auto refresh for expired content?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Auto refresh for expired content?

rnburn
Hi Folks,

i´m using Nginx as a proxy for my mobile app - which works pretty fine so
far!

My main cache has the following config:
proxy_cache_path /var/cache/nginx/spieldaten levels=1:2
keys_zone=spieldaten:100m max_size=150m inactive=5m use_temp_path=off;

proxy_cache_valid 200 302 5m;

If a request is not cached by Nginx, it could take about 5 sec til it comes
back from the defined proxy_pass - once it is cached its below 1 sec :)

Now my question, is it possible that Nginx can "automaticly" get a fresh
copy from the proxy_pass, when it recognized that the cached request is
expired - so that none User has to wait about 5 sec to get fresh content and
gets always fast data from the Nginx cache?

Thanks in advance!

Regrads,
Maik

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274264,274264#msg-274264

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

Francis Daly
On Tue, May 16, 2017 at 12:47:57PM -0400, mkuehn wrote:

Hi there,

untested, but:

> Now my question, is it possible that Nginx can "automaticly" get a fresh
> copy from the proxy_pass, when it recognized that the cached request is
> expired - so that none User has to wait about 5 sec to get fresh content and
> gets always fast data from the Nginx cache?

does http://nginx.org/r/proxy_cache_background_update sound like it can
do what you want?

You might be able to achieve most of what you want outside of nginx
by running a "curl" command to fetch the content every few minutes;
the hope being that the curl command will be the one that is waiting
for 5 seconds and causing the local cache copy to be updated, while the
"real" clients are fed from the cache.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

rnburn
Hi Francis,

thanks a lot for your reply!

I tried to use proxy_cache_background_update with the following config
part.

proxy_cache_path /var/cache/nginx/spieldaten levels=1:2
keys_zone=spieldaten:100m max_size=150m inactive=5d use_temp_path=off;
... ... ...
            proxy_cache test;
            proxy_cache_valid 200 302 1m;
            proxy_cache_valid 404 1m;
            proxy_cache_valid any 2m;
            proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
            proxy_cache_use_stale error timeout updating http_500 http_502
http_503 http_504;
            proxy_cache_lock on;
            proxy_cache_background_update on;

So my hope was with an inactive setup of 5d, even the requests which are not
often called, will be keep i the cache and proxy_cache_background_update
will do the magic - if a user request content, it will be fetch the "old"
one from the cache and nginx will fetch a fresh copy in the background.
But unfortunately the user gets always the old one from the cache, if
proxy_cache_background_update is on - so something must be wrong with my
config?!?

I even thought about a cron job, but the problem is, i have thousands of
different requests - so my hope was, that nginx can do the magic, wenn a
content in the cache expired, there will be a "trigger" that nginx can fetch
a new one in the background...

Maybe you have an idea, why proxy_cache_background_update not working?

Thanks!
Maik

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274264,274276#msg-274276

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

Roman Arutyunyan
Hi,

On Wed, May 17, 2017 at 03:29:03AM -0400, mkuehn wrote:

> Hi Francis,
>
> thanks a lot for your reply!
>
> I tried to use proxy_cache_background_update with the following config
> part.
>
> proxy_cache_path /var/cache/nginx/spieldaten levels=1:2
> keys_zone=spieldaten:100m max_size=150m inactive=5d use_temp_path=off;
> ... ... ...
>             proxy_cache test;
>             proxy_cache_valid 200 302 1m;
>             proxy_cache_valid 404 1m;
>             proxy_cache_valid any 2m;
>             proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
>             proxy_cache_use_stale error timeout updating http_500 http_502
> http_503 http_504;
>             proxy_cache_lock on;
>             proxy_cache_background_update on;
>
> So my hope was with an inactive setup of 5d, even the requests which are not
> often called, will be keep i the cache and proxy_cache_background_update
> will do the magic - if a user request content, it will be fetch the "old"
> one from the cache and nginx will fetch a fresh copy in the background.
> But unfortunately the user gets always the old one from the cache, if
> proxy_cache_background_update is on - so something must be wrong with my
> config?!?

Can you check if the update request comes to your backend when user gets the
old cached response?

[..]

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

rnburn
Hi Roman,

thanks for your reply - with proxy_cache_background_update is OFF, the
correct testfile "test.js"is requested:

"GET /test.js HTTP/1.0" 200 199503 "-" "Mozilla/5.0 (Windows NT 6.1; Win64;
x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96
Safari/537.36"


With proxy_cache_background_update is ON, the only "/" is requested:

"GET / HTTP/1.0" 200 32287 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"

Seems there is something wrong :/

Best,
Maik

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274264,274282#msg-274282

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

Francis Daly
In reply to this post by rnburn
On Wed, May 17, 2017 at 03:29:03AM -0400, mkuehn wrote:

Hi there,

> proxy_cache_path /var/cache/nginx/spieldaten levels=1:2
> keys_zone=spieldaten:100m max_size=150m inactive=5d use_temp_path=off;
> ... ... ...
>             proxy_cache test;

I'm pretty sure that nginx's internationalisation/localisation efforts
don't allow it to recognise that "test" and "spieldaten" should be
considered equivalent :-)

That aside: the auto-refresh works for me.

Using your proxy_cache settings in a test nginx.conf:

==
http {
  server {
    listen 8880;
    return 200 "Now it is $time_local\n";
    access_log logs/upstream.log;
  }

  proxy_cache_path /tmp/testcache levels=1 keys_zone=test:10m max_size=10m inactive=5d use_temp_path=off;

  server {
    listen 8080;
    proxy_cache test;
    proxy_cache_valid 200 302 1m;
    proxy_cache_valid 404 1m;
    proxy_cache_valid any 2m;
    proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    proxy_cache_background_update on;

    access_log logs/main.log;
    location / {
      proxy_pass <a href="http://127.0.0.1:8880;">http://127.0.0.1:8880;
    }
  }
}
==

I can do

$ while :; do date; curl http://127.0.0.1:8080/; sleep 7; done

and the first request after the 1-minute expiry gives me the old content;
and the next request gives me the new content that was fetched 7 seconds
previously.

(And I can compare the different log files, to see how much my caching
saved the upstream port 8880 server from processing.)

Which seems to be exactly what you want.

Does that test work for you? If it does not, there is a problem to be
resolved. If it does, then you can start to compare your own system with
what is above, and spot the difference.

> Maybe you have an idea, why proxy_cache_background_update not working?

As above, it works for me. So - what is different in your system?

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

Roman Arutyunyan
In reply to this post by rnburn
On Wed, May 17, 2017 at 09:07:30AM -0400, mkuehn wrote:

> Hi Roman,
>
> thanks for your reply - with proxy_cache_background_update is OFF, the
> correct testfile "test.js"is requested:
>
> "GET /test.js HTTP/1.0" 200 199503 "-" "Mozilla/5.0 (Windows NT 6.1; Win64;
> x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96
> Safari/537.36"
>
>
> With proxy_cache_background_update is ON, the only "/" is requested:
>
> "GET / HTTP/1.0" 200 32287 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"
>
> Seems there is something wrong :/

What nginx version do you have?  Please update to the lastest version.
We made some fixes in the behavior of background update since it was released.

If it does not help, please provide more complete config (the part you sent
earlier does not even have proxy_pass).

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

rnburn
Hi Roman,
hi Francis,

here comes my production config - i made a few tests the last hours, it
always come to the same point - if proxy_cache_background_update is on,
nginx devilvers only the cached content and is not able to fetch a fresh
copy (dont know why) - i tried local files, i tried my production node.js
backend, i even tried an external server as proxy_path - always the same
result... :/
I hope you find something wrong in my configs and then everything is fine ;)
:)
I use centos with nginx/1.12.0

Best regards,
Maik



user nginx;
worker_processes  auto;

pid        /run/nginx.pid;

worker_rlimit_nofile         50000;

events {
    worker_connections  50000;
    use epoll;
    multi_accept on;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request"
'
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
log_format upstreamlog '[$time_local] $remote_addr - $remote_user -
$server_name  to: $upstream_addr: $request_uri args: $args
upstream_response_time $upstream_response_time msec $msec request_time
$request_time';
    sendfile       on;
    tcp_nopush     on;

    keepalive_timeout   65;
    tcp_nodelay         on;
    types_hash_max_size 2048;

    proxy_cache_path /var/cache/nginx/spieldaten levels=1:2
keys_zone=spieldaten:100m max_size=150m inactive=5d use_temp_path=off;
   

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  xy.com;
        root         /home/nodejs/pcnode;

        access_log             off;
        error_log              off;

        gzip on;
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 8 4k;
        gzip_http_version 1.1;
      gzip_http_version 1.1;
        gzip_min_length 256;
        gzip_types text/plain text/css application/json
application/x-javascript text/xml application/xml application/xml+rss
text/javascript;

        try_files $uri @node;

        location @node {
            #access_log  /var/log/nginx/access.log  main;
            access_log off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass <a href="http://localhost:8088;">http://localhost:8088;
            proxy_redirect off;

            proxy_connect_timeout   30s;
            proxy_send_timeout      30s;
            proxy_read_timeout      30s;

            proxy_cache spieldaten;
            proxy_cache_valid 200 302 5m;
            proxy_cache_valid 404 1m;
            proxy_cache_valid any 2m;
            proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
            proxy_cache_use_stale error timeout updating http_500 http_502
http_503 http_504;
            proxy_cache_lock on;
            proxy_cache_background_update on;

        }
     }
 }

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274264,274298#msg-274298

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Auto refresh for expired content?

rnburn
In reply to this post by Roman Arutyunyan
Hi Roman,

i would like to kindly ask you, if you already had the time to take a look
at my config, which i posted on May 18, 2017 10:02AM?

Thanks in advanced!
Maik

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274264,274493#msg-274493

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx