Proxying setup delivering wrong cache entry in some edge cases
We've been using Nginx as a caching proxy for quite a while in different
scenarios now. Since a few weeks and especially in the last couple of days
we continue to encounter a strange behaviour in one of our scenarios leading
to wrong content being delivered.
In that case we use Nginx as a caching proxy for a bunch of subdomains on a
kind of multitenancy application. We established the setup 4 months ago and
never had any problems until recently.
Your configuration has:
proxy_cache_use_stale error timeout invalid_header http_502;
I would start with either disabling the proxy_cache_use_stale and/or inspecting the $http_host.
If I'm not wrong $http_host doesn't get the same treatment as $host which also comes from user request headers but if not present or empty gets set as $server_name matching the request.
So if a client doesn't send the 'Host:' header there might be cache entries basically just with the $request_uri which are served in some specific cases.
Nginx always stores and returns whatever the backend sent as a response.
Since you change the Host header:
If possible you could add some debug headers on the backends - to see if request actually landing on the nginx proxy is correctly passed on the backend (like you could again be missing the X-Forwarded-Host header).
Also you can do a simple MD5 on the problematic request (like md5 https://foo.example.org/bla/fasel -> 4DFDF87BB2FD82629ACB91BB1B1B2A1C (obviously for the gzipped content you have to add 'gzip' at the beginning) and then check if the cache file in /opt/example-org-cache/c/a1/4dfdf87bb2fd82629acb91bb1b1b2a1c actually exists and what's the content of it.