upstream 429 and non-idempotent request

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

upstream 429 and non-idempotent request

Frank Liu
In case of upstream returning 429, I'd like to have nginx retry next upstream server. Since nginx by default won't retry non-idempotent requests, how do I force nginx to retry when receiving 429? I imagine this should be the default behavior anyway, or does nginx not care about returning code and will never retry non-idempotent?

Thanks!
Frank


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: upstream 429 and non-idempotent request

Maxim Dounin
Hello!

On Thu, Jun 08, 2017 at 01:10:25AM -0700, Frank Liu wrote:

> In case of upstream returning 429, I'd like to have nginx retry next
> upstream server. Since nginx by default won't retry non-idempotent
> requests, how do I force nginx to retry when receiving 429? I imagine this
> should be the default behavior anyway, or does nginx not care about
> returning code and will never retry non-idempotent?

Non-idemportent requests are not retried as long as the request is
already sent, regardless of a particular error.  If you want nginx
to retry non-idempotent requests, you can do so with
"proxy_next_upstream non-idempotent;", see
http://nginx.org/r/proxy_next_upstream.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: upstream 429 and non-idempotent request

Frank Liu
I fully understand the rationale of not retrying non-idempotent requests if they are already sent, but in case of 429 (maybe other cases as well), I don't see an issue of retrying even if request is sent. It would be better if we can selectively do something like "proxy_next_upstream non-idempotent-http_429;" or whatever http code that we know safe.

On Thu, Jun 8, 2017 at 8:20 AM, Maxim Dounin <[hidden email]> wrote:
Hello!

On Thu, Jun 08, 2017 at 01:10:25AM -0700, Frank Liu wrote:

> In case of upstream returning 429, I'd like to have nginx retry next
> upstream server. Since nginx by default won't retry non-idempotent
> requests, how do I force nginx to retry when receiving 429? I imagine this
> should be the default behavior anyway, or does nginx not care about
> returning code and will never retry non-idempotent?

Non-idemportent requests are not retried as long as the request is
already sent, regardless of a particular error.  If you want nginx
to retry non-idempotent requests, you can do so with
"proxy_next_upstream non-idempotent;", see
http://nginx.org/r/proxy_next_upstream.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: upstream 429 and non-idempotent request

Maxim Dounin
Hello!

On Thu, Jun 08, 2017 at 09:55:05AM -0700, Frank Liu wrote:

> I fully understand the rationale of not retrying non-idempotent requests if
> they are already sent, but in case of 429 (maybe other cases as well), I
> don't see an issue of retrying even if request is sent. It would be better
> if we can selectively do something like "proxy_next_upstream
> non-idempotent-http_429;" or whatever http code that we know safe.

The problem is that we don't really know if the code is safe or
not.

In general, non-idempotent requests are not retried as we
don't know if the request was processed or not.  The situation is
obvious when we got a network error after sending a request: as
the error might happen after the request was already processed, it
is wise to refrain from retrying it.

But what happen when we got a valid http error instead?  For
example, 502 in most cases means that there was a network error
somewhere else.  So, this basically means that 502 cannot be
retried as well.  And the other 5xx error codes are more or less
identical: we never know what really happened, and can't retry.

With 4xx errors it seems safe to assume that the request was not
processed, and hence retrying is possible.  But, for example, if
error_page is used on the backend server, 404 might be returned
if a network error happens and a corresponding error page cannot
be found.  Similarly, 429 might be returned if limit_req rejects a
request to the error page.

We've considered adding a logic to always retry non-idempotent
requests in case of 4xx errors when non-idempotence handing was
introduced.  But decided to keep things simple and safe, and never
retry non-idempotent requests.  On the other hand, introducing
configuration options to fine tune if non-idempotent requests
should be retried for each proxy_next_upstream case seems to be
overkill.

In general, current implementation assumes the following two
options:

- non-idempotent requests are not retried;

- there is a duplicate request protection in the application, so
  non-dempotent requests can be retried with "proxy_next_upstream
  non_idempotent;".

This seems to be enough for most, if not all, use cases.  If
something more complex is really needed, it can be configured
using error_page and additional error processing logic.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: upstream 429 and non-idempotent request

Frank Liu
Hi,

I fully understand the concern and complexity of different cases. Making any default assumption will have risks. That's why I suggested providing config options since users themselves know their use case and whether it is safe to retry.

Thanks!
Frank




On Tue, Jun 13, 2017 at 9:30 AM, Maxim Dounin <[hidden email]> wrote:
Hello!

On Thu, Jun 08, 2017 at 09:55:05AM -0700, Frank Liu wrote:

> I fully understand the rationale of not retrying non-idempotent requests if
> they are already sent, but in case of 429 (maybe other cases as well), I
> don't see an issue of retrying even if request is sent. It would be better
> if we can selectively do something like "proxy_next_upstream
> non-idempotent-http_429;" or whatever http code that we know safe.

The problem is that we don't really know if the code is safe or
not.

In general, non-idempotent requests are not retried as we
don't know if the request was processed or not.  The situation is
obvious when we got a network error after sending a request: as
the error might happen after the request was already processed, it
is wise to refrain from retrying it.

But what happen when we got a valid http error instead?  For
example, 502 in most cases means that there was a network error
somewhere else.  So, this basically means that 502 cannot be
retried as well.  And the other 5xx error codes are more or less
identical: we never know what really happened, and can't retry.

With 4xx errors it seems safe to assume that the request was not
processed, and hence retrying is possible.  But, for example, if
error_page is used on the backend server, 404 might be returned
if a network error happens and a corresponding error page cannot
be found.  Similarly, 429 might be returned if limit_req rejects a
request to the error page.

We've considered adding a logic to always retry non-idempotent
requests in case of 4xx errors when non-idempotence handing was
introduced.  But decided to keep things simple and safe, and never
retry non-idempotent requests.  On the other hand, introducing
configuration options to fine tune if non-idempotent requests
should be retried for each proxy_next_upstream case seems to be
overkill.

In general, current implementation assumes the following two
options:

- non-idempotent requests are not retried;

- there is a duplicate request protection in the application, so
  non-dempotent requests can be retried with "proxy_next_upstream
  non_idempotent;".

This seems to be enough for most, if not all, use cases.  If
something more complex is really needed, it can be configured
using error_page and additional error processing logic.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx