Proxying setup delivering wrong cache entry in some edge cases

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Proxying setup delivering wrong cache entry in some edge cases

wld75
We've been using Nginx as a caching proxy for quite a while in different
scenarios now. Since a few weeks and especially in the last couple of days
we continue to encounter a strange behaviour in one of our scenarios leading
to wrong content being delivered.

In that case we use Nginx as a caching proxy for a bunch of subdomains on a
kind of multitenancy application. We established the setup 4 months ago and
never had any problems until recently.

For example, a request to https://test.example.org/bla/fasel would deliver
the content for https://foo.example.org/bla/fasel. So basically it delivers
content for the wrong subdomain. Those occasions are very, very rare and
totally random in regards to the subdomain from which the content gets
delivered.

We currently use openresty 1.13.6.1.

Our config is quite large, so I will put it into a gist if that's OK:
https://gist.github.com/benzimmer/a4ee7b43ae4ade24a570301dfd0c12c2

This seems to be working fine for the most part, but every now and then we
see the described behaviour without being able to consistently reproduce
it.

If anyone has any clue why this might be happening, we'd be very grateful.
If you need any additional information, please feel free to ask away!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281606,281606#msg-281606

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

RE: Proxying setup delivering wrong cache entry in some edge cases

Reinis Rozitis
> For example, a request to https://test.example.org/bla/fasel would deliver
> the content for https://foo.example.org/bla/fasel. So basically it delivers
> content for the wrong subdomain. Those occasions are very, very rare and
> totally random in regards to the subdomain from which the content gets
> delivered.

Your configuration has:
proxy_cache_key $ae$scheme://$http_host$request_uri;
and
proxy_cache_use_stale error timeout invalid_header http_502;

I would start with either disabling the proxy_cache_use_stale and/or inspecting the $http_host.

If I'm not wrong $http_host doesn't get the same treatment as $host which also comes from user request headers but if not present or empty gets set as $server_name matching the request.
So if a client doesn't send the 'Host:'  header there might be cache entries basically just with the $request_uri which are served in some specific cases.

Nginx always stores and returns whatever the backend sent as a response.
Since you change the Host header:

    proxy_set_header Host $upstream_endpoint;
    proxy_set_header X-Forwarded-Host $http_host;

If possible you could add some debug headers on the backends - to see if request actually landing on the nginx proxy is correctly passed on the backend (like you could again be missing the X-Forwarded-Host header).

Also you can do a simple MD5 on the problematic request (like md5 https://foo.example.org/bla/fasel -> 4DFDF87BB2FD82629ACB91BB1B1B2A1C (obviously for the gzipped content you have to add 'gzip' at the beginning) and then check if the cache file in /opt/example-org-cache/c/a1/4dfdf87bb2fd82629acb91bb1b1b2a1c actually exists and what's the content of it.

rr

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx