No revalidation when using stale-while-revalidate

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

No revalidation when using stale-while-revalidate

Adam Volek
Hi,

We're running into some strange behaviour with the
stale-while-revalidate extension of the cache-control header when using
nginx as a reverse proxy. When there is a stale response in the cache
with nonzero stale-while-revalidate time, nginx attempts revalidation
but seems to ignore the upstream answer if it has specific status code,
such as 404 or 500, and server a stale response to the client. Other
response codes such as 200 or 410 don't trigger this behaviour. Is this
intended, and if so, is there a way to configure nginx to treat 404 as
any other response?

I understand that this behaviour might be desirable in some situations
(especially for the responses with 5xx status codes), but in our case,
if the upstream return a 404 response, we would want the cache to start
returning it as well as soon as the revalidation is finished.

Below is a minimum nginx.conf we used to replicate the issue.

Kind regards,
Adam Volek


events {}

http {
     proxy_cache_path /tmp/ngx_cache keys_zone=one:10m;
     proxy_cache one;

     server {
         listen 8080 default_server;

         location / {
             proxy_pass http://example.com/;
         }
     }
}
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: No revalidation when using stale-while-revalidate

Maxim Dounin
Hello!

On Thu, Jul 23, 2020 at 08:06:36PM +0200, Adam Volek wrote:

> We're running into some strange behaviour with the
> stale-while-revalidate extension of the cache-control header
> when using nginx as a reverse proxy. When
> there is a stale response in the cache with nonzero
> stale-while-revalidate time, nginx attempts revalidation but
> seems to ignore the upstream answer if it
> has specific status code, such as 404 or 500, and server a stale
> response to the client. Other response codes such as 200 or 410
> don't trigger this
> behaviour. Is this intended, and if so, is there a way to
> configure nginx to treat 404 as any other response?

As long as the response returned isn't cacheable (either
as specified in the response Cache-Control / Expires
headers, or per proxy_cache_valid), nginx won't put
the response into cache and will continue serving previously
cached response till stale-while-revalidate timeout expires.

Most likely "specific status code" in your tests in fact means
responses returned by your upstream server without Cache-Control
headers, and hence not cached by nginx.

> I understand that this behaviour might be desirable in some
> situations (especially for the responses with 5xx status codes),
> but in our case, if the
> upstream return a 404 response, we would want the cache to start
> returning it as well as soon as the revalidation is finished.

As long as the above analysis is correct, the solution is simple:
make sure responses you want nginx to cache are cacheable, that
is, make sure appropriate headers are present or configure
proxy_cache_valid (http://nginx.org/r/proxy_cache_valid).

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: No revalidation when using stale-while-revalidate

Adam Volek
Hi,

On 24. 07. 20 4:33, Maxim Dounin wrote:
 > As long as the response returned isn't cacheable (either
 > as specified in the response Cache-Control / Expires
 > headers, or per proxy_cache_valid), nginx won't put
 > the response into cache and will continue serving previously
 > cached response till stale-while-revalidate timeout expires.
 >
 > Most likely "specific status code" in your tests in fact means
 > responses returned by your upstream server without Cache-Control
 > headers, and hence not cached by nginx.

This is not the case as far as I can tell. In our tests, the upstream
server was set up to send these two responses, the 204 first, and then
then the 404:

HTTP/1.1 204 No Content
Date: Fri, 24 Jul 2020 11:32:33 GMT
Connection: keep-alive
cache-control: max-age=5, stale-while-revalidate=10

HTTP/1.1 404 Not Found
Date: Fri, 24 Jul 2020 11:32:35 GMT
Content-Type: text/plain
Connection: close
cache-control: max-age=5, stale-while-revalidate=10

In this scenario, nginx returns fresh 204 for five seconds and then it
returns stale 204 for ten seconds even though it's attempting
revalidation according to access log at the upstream server. If we send
the following 410 response instead of 404 however, nginx behaves as we
would expect: it returns the fresh 204 for five seconds, then it
revalidates it almost instantly and starts returning the fresh 410:

HTTP/1.1 410 Gone
Date: Fri, 24 Jul 2020 11:41:56 GMT
Content-Type: text/plain
Connection: close
cache-control: max-age=5, stale-while-revalidate=10

Adam Volek

On 24. 07. 20 4:33, Maxim Dounin wrote:

> Hello!
>
> On Thu, Jul 23, 2020 at 08:06:36PM +0200, Adam Volek wrote:
>
>> We're running into some strange behaviour with the
>> stale-while-revalidate extension of the cache-control header
>> when using nginx as a reverse proxy. When
>> there is a stale response in the cache with nonzero
>> stale-while-revalidate time, nginx attempts revalidation but
>> seems to ignore the upstream answer if it
>> has specific status code, such as 404 or 500, and server a stale
>> response to the client. Other response codes such as 200 or 410
>> don't trigger this
>> behaviour. Is this intended, and if so, is there a way to
>> configure nginx to treat 404 as any other response?
>
> As long as the response returned isn't cacheable (either
> as specified in the response Cache-Control / Expires
> headers, or per proxy_cache_valid), nginx won't put
> the response into cache and will continue serving previously
> cached response till stale-while-revalidate timeout expires.
>
> Most likely "specific status code" in your tests in fact means
> responses returned by your upstream server without Cache-Control
> headers, and hence not cached by nginx.
>
>> I understand that this behaviour might be desirable in some
>> situations (especially for the responses with 5xx status codes),
>> but in our case, if the
>> upstream return a 404 response, we would want the cache to start
>> returning it as well as soon as the revalidation is finished.
>
> As long as the above analysis is correct, the solution is simple:
> make sure responses you want nginx to cache are cacheable, that
> is, make sure appropriate headers are present or configure
> proxy_cache_valid (http://nginx.org/r/proxy_cache_valid).
>
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: No revalidation when using stale-while-revalidate

Maxim Dounin
Hello!

On Fri, Jul 24, 2020 at 03:21:31PM +0200, Adam Volek wrote:

> On 24. 07. 20 4:33, Maxim Dounin wrote:
> > As long as the response returned isn't cacheable (either
> > as specified in the response Cache-Control / Expires
> > headers, or per proxy_cache_valid), nginx won't put
> > the response into cache and will continue serving previously
> > cached response till stale-while-revalidate timeout expires.
> >
> > Most likely "specific status code" in your tests in fact means
> > responses returned by your upstream server without Cache-Control
> > headers, and hence not cached by nginx.
>
> This is not the case as far as I can tell. In our tests, the upstream server was set up to send these two responses, the 204 first, and then then the 404:
>
> HTTP/1.1 204 No Content
> Date: Fri, 24 Jul 2020 11:32:33 GMT
> Connection: keep-alive
> cache-control: max-age=5, stale-while-revalidate=10
>
> HTTP/1.1 404 Not Found
> Date: Fri, 24 Jul 2020 11:32:35 GMT
> Content-Type: text/plain
> Connection: close
> cache-control: max-age=5, stale-while-revalidate=10
>
> In this scenario, nginx returns fresh 204 for five seconds and then it returns stale 204 for ten seconds even though it's attempting revalidation according
> to access log at the upstream server. If we send the following 410 response instead of 404 however, nginx behaves as we would expect: it returns the fresh
> 204 for five seconds, then it revalidates it almost instantly and starts returning the fresh 410:
>
> HTTP/1.1 410 Gone
> Date: Fri, 24 Jul 2020 11:41:56 GMT
> Content-Type: text/plain
> Connection: close
> cache-control: max-age=5, stale-while-revalidate=10

You are right, this seems to be an incorrect behaviour of
stale-while-revalidate / stale-if-error handling.

Internally, stale-if-error (and stale-while-revalidate) currently
behave as if "proxy_cache_use_stale" was set with all possible
flags (http://nginx.org/r/proxy_cache_use_stale) when handling
upstream server responses.  Notably this includes http_403,
http_404, and http_429 flags, and this causes the effect you
observe.

This probably should be fixed.

Just in case, the following configuration can be used to reproduce
the issue within nginx itself:

    proxy_cache_path cache keys_zone=one:1m;

    server {
        listen 8080;

        location / {
            proxy_pass <a href="http://127.0.0.1:8081;">http://127.0.0.1:8081;
            proxy_cache one;
            add_header X-Cache-Status $upstream_cache_status always;
        }
    }

    server {
        listen 8081;

        location / {
            add_header cache-control "max-age=5, stale-while-revalidate=10"
                       always;

            if ($connection = "3") {
                return 204;
            }

            return 404;
        }
    }

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: No revalidation when using stale-while-revalidate

Roman Arutyunyan
Hi,

On Mon, Jul 27, 2020 at 04:42:00AM +0300, Maxim Dounin wrote:

> Hello!
>
> On Fri, Jul 24, 2020 at 03:21:31PM +0200, Adam Volek wrote:
>
> > On 24. 07. 20 4:33, Maxim Dounin wrote:
> > > As long as the response returned isn't cacheable (either
> > > as specified in the response Cache-Control / Expires
> > > headers, or per proxy_cache_valid), nginx won't put
> > > the response into cache and will continue serving previously
> > > cached response till stale-while-revalidate timeout expires.
> > >
> > > Most likely "specific status code" in your tests in fact means
> > > responses returned by your upstream server without Cache-Control
> > > headers, and hence not cached by nginx.
> >
> > This is not the case as far as I can tell. In our tests, the upstream server was set up to send these two responses, the 204 first, and then then the 404:
> >
> > HTTP/1.1 204 No Content
> > Date: Fri, 24 Jul 2020 11:32:33 GMT
> > Connection: keep-alive
> > cache-control: max-age=5, stale-while-revalidate=10
> >
> > HTTP/1.1 404 Not Found
> > Date: Fri, 24 Jul 2020 11:32:35 GMT
> > Content-Type: text/plain
> > Connection: close
> > cache-control: max-age=5, stale-while-revalidate=10
> >
> > In this scenario, nginx returns fresh 204 for five seconds and then it returns stale 204 for ten seconds even though it's attempting revalidation according
> > to access log at the upstream server. If we send the following 410 response instead of 404 however, nginx behaves as we would expect: it returns the fresh
> > 204 for five seconds, then it revalidates it almost instantly and starts returning the fresh 410:
> >
> > HTTP/1.1 410 Gone
> > Date: Fri, 24 Jul 2020 11:41:56 GMT
> > Content-Type: text/plain
> > Connection: close
> > cache-control: max-age=5, stale-while-revalidate=10
>
> You are right, this seems to be an incorrect behaviour of
> stale-while-revalidate / stale-if-error handling.
>
> Internally, stale-if-error (and stale-while-revalidate) currently
> behave as if "proxy_cache_use_stale" was set with all possible
> flags (http://nginx.org/r/proxy_cache_use_stale) when handling
> upstream server responses.  Notably this includes http_403,
> http_404, and http_429 flags, and this causes the effect you
> observe.
>
> This probably should be fixed.
>
> Just in case, the following configuration can be used to reproduce
> the issue within nginx itself:
>
>     proxy_cache_path cache keys_zone=one:1m;
>
>     server {
>         listen 8080;
>
>         location / {
>             proxy_pass <a href="http://127.0.0.1:8081;">http://127.0.0.1:8081;
>             proxy_cache one;
>             add_header X-Cache-Status $upstream_cache_status always;
>         }
>     }
>
>     server {
>         listen 8081;
>
>         location / {
>             add_header cache-control "max-age=5, stale-while-revalidate=10"
>                        always;
>
>             if ($connection = "3") {
>                 return 204;
>             }
>
>             return 404;
>         }
>     }

The fix was committed:

https://hg.nginx.org/nginx/rev/7015f26aef90

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx