Proxying setup delivering wrong cache entry in some edge cases
We've been using Nginx as a caching proxy for quite a while in different
scenarios now. Since a few weeks and especially in the last couple of days
we continue to encounter a strange behaviour in one of our scenarios leading
to wrong content being delivered.
In that case we use Nginx as a caching proxy for a bunch of subdomains on a
kind of multitenancy application. We established the setup 4 months ago and
never had any problems until recently.
Your configuration has:
proxy_cache_use_stale error timeout invalid_header http_502;
I would start with either disabling the proxy_cache_use_stale and/or inspecting the $http_host.
If I'm not wrong $http_host doesn't get the same treatment as $host which also comes from user request headers but if not present or empty gets set as $server_name matching the request.
So if a client doesn't send the 'Host:' header there might be cache entries basically just with the $request_uri which are served in some specific cases.
Nginx always stores and returns whatever the backend sent as a response.
Since you change the Host header:
If possible you could add some debug headers on the backends - to see if request actually landing on the nginx proxy is correctly passed on the backend (like you could again be missing the X-Forwarded-Host header).
Also you can do a simple MD5 on the problematic request (like md5 https://foo.example.org/bla/fasel -> 4DFDF87BB2FD82629ACB91BB1B1B2A1C (obviously for the gzipped content you have to add 'gzip' at the beginning) and then check if the cache file in /opt/example-org-cache/c/a1/4dfdf87bb2fd82629acb91bb1b1b2a1c actually exists and what's the content of it.
Re: RE: Proxying setup delivering wrong cache entry in some edge cases
Thanks for your answer and apologies for the long delay.
How would the $http_host ever be empty? If I make a request without it I
receive a 400 Bad Request as the HTTP spec defines it. Does Nginx still
forward the request to the upstream server and populate a cache entry?
Additionally, if I make requests to our backend without a proper
X-Forwarded-For header I will always receive a 404 and not data for a wrong
Unfortunately we're still not able to reproduce the problem on our end, but
still receive complaints from users encountering the problem.
We removed all caching from the problematic endpoint, but the problem still
seems to persist. Are there any known conditions, where Nginx would pass on
a wrong host to the upstream server for any kind of reason?