Nginx not retrying failed UDP messaged

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Nginx not retrying failed UDP messaged

vergil
Hi,

We are using NGINX as Load Balancer for load balancing RADIUS UDP traffic.

It seems NGINX is not retrying to send messages to another upstream server
if one of the upstream servers is down.

We are using NGINX 1.16.1. But the same behaviour exists in all NGINX
version upto NGINX 1.19.

In older NGINX version (1.13.10/1.14.1) this retry is working fine without
any issues.

Following is our NGINX configuration -

    server {
        listen 1813 udp reuseport;

        proxy_pass udp_radius;
        proxy_connect_timeout 60s;
        proxy_timeout 5s;
        proxy_responses 1;
        proxy_requests 1;

        proxy_buffer_size 64k;

        #access_log /var/log/nginx/radius.log upstreamlog buffer=64k
flush=1m;
        access_log off;
    }

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288649,288649#msg-288649

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx not retrying failed UDP message

Vladimir Homutov
On Thu, Jul 09, 2020 at 02:42:52PM -0400, harsh wrote:

> Hi,
>
> We are using NGINX as Load Balancer for load balancing RADIUS UDP traffic.
>
> It seems NGINX is not retrying to send messages to another upstream server
> if one of the upstream servers is down.
>
> We are using NGINX 1.16.1. But the same behaviour exists in all NGINX
> version upto NGINX 1.19.
>
> In older NGINX version (1.13.10/1.14.1) this retry is working fine without
> any issues.
>
> Following is our NGINX configuration -
>
>     server {
>         listen 1813 udp reuseport;
>
>         proxy_pass udp_radius;
>         proxy_connect_timeout 60s;
>         proxy_timeout 5s;
>         proxy_responses 1;
>         proxy_requests 1;
>
>         proxy_buffer_size 64k;
>
>         #access_log /var/log/nginx/radius.log upstreamlog buffer=64k
> flush=1m;
>         access_log off;
>     }

Can you please show full configuration? Is 'udp_radius' a hostname
or an upstream{} block ?


Regarding 'retrying to send message': since UDP is non-reliable, nginx
will retry with another server only if it will get error immediately
during send() call (not very probable). nginx sends packet, no error
immediately; later it can get either icmp-caused error on read or write,
or timeout can fire, and the upstream will be marked as failed. So, no
new packets will be sent to it. Until upstream is not marked dead,
new packets can be sent to it.

Probably, you see effects of this change:

Changes with nginx 1.15.0                                        05 Jun 2018

    *) Feature: now the stream module can handle multiple incoming UDP
       datagrams from a client within a single session.

You may want to look at debug log to see what is exactly happening -
there is information about upstream servers marked alive or not, and
clients requests and responses.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Nginx not retrying failed UDP message

vergil
Thanks Vladimir for the explanation.

'udp_radius' is an upstream{} block with 3 servers. What you mentioned is
exactly what we are seeing in our tests. Since we are dealing with Radius
Accounting request we don't want to lose failed messages.

Is there any way to force nginx to retry failed messages at all?

Thanks,
Harsh

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288662,288727#msg-288727

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx