Why Nginx send traffic to the next upstream on 504 error

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Why Nginx send traffic to the next upstream on 504 error

zakirenish
Hello,

We have an Nginx where we configured
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream
in Nginx main config file at http level.

We don't want to send traffic to the next upstream on 504:
----
proxy_next_upstream    error timeout http_502 http_503 non_idempotent;
----

At vhost level we don't redefine this directive. But periodically we see the
following in Nginx logs:
----
141.101.69.85 - 192.168.1.10 - 59.489 - [08/Aug/2020:11:10:41.098 +0000] -
POST - /api/ - HTTP/1.1 - 499 -  - 0 - 2054 - "Java/1.8.0_242" - api-worker
- "192.168.1.11:8080, 192.168.1.12:8080" - "504, -" - "0.000, 0.000" -
"31.000, 28.489"
----

api-worker - "192.168.1.11:8080, 192.168.1.12:8080" - "504, -"

From the logs we see that Nginx received 504 status from the first upstream
and then for some reason send traffic to the next one, despite the fact that
it should not do it on 504 http status.

We did a short test using http://httpstat.us/504
----
upstream test-504 {
  server 104.31.86.226:80;
  server 104.31.87.226:80;
}

server {
  listen      443 ssl http2;
  server_name domain.tld

  # Test 504
  location /test-504 {

    # proxy_next_upstream error timeout http_502 http_503 http_504
non_idempotent;

    proxy_pass http://test-504/504;

    proxy_set_header Host httpstat.us;
  }
}
----

If we comment 'proxy_next_upstream' it uses one defined in the main config
at http level and don't send traffic to the next upstream. If we uncomment
it, we see that Nginx send traffic to the next upstream. All works as
expected and described in the documentation.

But the question remained for our production: Why Nginx send traffic to the
next upstream on 504 error?
1. It is some misconfiguration on our side, maybe timeouts on any other
directives should be enabled/disabled?
2. It is some kind if misunderstanding how 'proxy_next_upstream' directive
works?
3. It is some kind of bug?

Thank you!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289020,289020#msg-289020

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Why Nginx send traffic to the next upstream on 504 error

Maxim Dounin
Hello!

On Sat, Aug 08, 2020 at 12:55:47PM -0400, stmx38 wrote:

> We have an Nginx where we configured
> http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream
> in Nginx main config file at http level.
>
> We don't want to send traffic to the next upstream on 504:
> ----
> proxy_next_upstream    error timeout http_502 http_503 non_idempotent;
> ----
>
> At vhost level we don't redefine this directive. But periodically we see the
> following in Nginx logs:
> ----
> 141.101.69.85 - 192.168.1.10 - 59.489 - [08/Aug/2020:11:10:41.098 +0000] -
> POST - /api/ - HTTP/1.1 - 499 -  - 0 - 2054 - "Java/1.8.0_242" - api-worker
> - "192.168.1.11:8080, 192.168.1.12:8080" - "504, -" - "0.000, 0.000" -
> "31.000, 28.489"
> ----
>
> api-worker - "192.168.1.11:8080, 192.168.1.12:8080" - "504, -"
>
> From the logs we see that Nginx received 504 status from the first upstream
> and then for some reason send traffic to the next one, despite the fact that
> it should not do it on 504 http status.

The "504" status in $uptream_status is used to indicate timeouts,
much like 502 to indicate errors.  These are status codes nginx
itself generates due to observed error conditions.  And your
"proxy_next_upstream" includes "timeout".

[...]

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Why Nginx send traffic to the next upstream on 504 error

zakirenish
Maxim, thank you for reply!

Some additional question here:
1. If we will remove 'timeout' will Nginx send traffic to the next upstream
on 502 error?
2. More general question related the the Q.1. When Nginx interpret reply as
502 - on timeout, if yes - on which one? We have a lot of timeouts defined
in the main config:
----
  client_header_timeout 30s;
  client_body_timeout   30s;
  send_timeout          65s;

  proxy_connect_timeout 5s;
  proxy_send_timeout    65s;
  proxy_read_timeout    65s;
----

Thanks!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289020,289025#msg-289025

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Why Nginx send traffic to the next upstream on 504 error

Maxim Dounin
Hello!

On Sun, Aug 09, 2020 at 10:45:14AM -0400, stmx38 wrote:

> Maxim, thank you for reply!
>
> Some additional question here:
> 1. If we will remove 'timeout' will Nginx send traffic to the next upstream
> on 502 error?

The "timeout" parameter corresponds to 504 returned to clients and
in $upstream_status, the "error" one corresponds to 502 returned to
clients and in $upstream_status.  Accordingly, removing "timeout"
won't affect anything related to 502.  It will, however, stop nginx
from trying next upstream servers on timeouts, also known as 504.

> 2. More general question related the the Q.1. When Nginx interpret reply as
> 502 - on timeout, if yes - on which one? We have a lot of timeouts defined
> in the main config:
> ----
>   client_header_timeout 30s;
>   client_body_timeout   30s;
>   send_timeout          65s;
>
>   proxy_connect_timeout 5s;
>   proxy_send_timeout    65s;
>   proxy_read_timeout    65s;
> ----

Quoting the docs (http://nginx.org/r/proxy_next_upstream):

    timeout
    a timeout has occurred while establishing a connection with the
    server, passing a request to it, or reading the response header;

That is, this implies timeouts when working with upstream server.  
Relevant configuration directives are proxy_connect_timeout,
proxy_send_timeout, and proxy_read_timeout.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx