Occasional successful upstreamed requests that don't get picked up

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Occasional successful upstreamed requests that don't get picked up

rnburn
Hello, I believe I have a tuning issue with NGINX - hoping someone can point
me in the right direction!

About 1 in 60,000 requests being proxied through Kong/NGINX are timing out.
These requests are getting to the upstreamed host and are successfully
logged in the load balancer in front of this upstreamed host.  So either
there's a network issue between that load balancer and NGINX or NGINX is
simply not able/willing to process the response.

Assuming this is an NGINX tuning issue, these are my settings (note these
hosts have 32 cores).  Traffic is not all that high, less than 10 req/sec
per instance and requests are usually satisfied in less than a second:

worker_processes auto;
worker_priority -10;
daemon off;
pid pids/nginx.pid;
error_log logs/error.log debug;

worker_rlimit_nofile 20000;

events {
    worker_connections 10000;
    multi_accept on;
}

Most all other config settings are "default" values.  There's nothing in the
Kong logs that indicate these dropped responses are being processed by Kong.
 There's no indication there aren't enough workers.  These timeouts do not
happen in clusters, they are more like singletons.

Any advice on things I should look at or diagnosis possibilities?  Thanks
very much, Ryan

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274267,274267#msg-274267

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Occasional successful upstreamed requests that don't get picked up

Peter Booth
Ryan,

What is the topology of the system that you are describing?

You mention kong/nginx, an upstream host, a load balancer, clients  ...

Are the load balancers hardware or software devices?
Is kong nginx simply forwarding to a load-balancer VIP that fronts multiple upstream systems?
Are there any firewalls or intrusion detection systems also in the mix?
Are your clients remote? internet based?
Are your nginx instances running on physical hosts, virtual machines, cloud based VMs?
Are your nginx instances and your upstream system s in the same datacenter or some distance apart?

Peter

On May 16, 2017, at 12:57 PM, rcutter <[hidden email]> wrote:

Hello, I believe I have a tuning issue with NGINX - hoping someone can point
me in the right direction!

About 1 in 60,000 requests being proxied through Kong/NGINX are timing out.
These requests are getting to the upstreamed host and are successfully
logged in the load balancer in front of this upstreamed host. So either
there's a network issue between that load balancer and NGINX or NGINX is
simply not able/willing to process the response.

Assuming this is an NGINX tuning issue, these are my settings (note these
hosts have 32 cores). Traffic is not all that high, less than 10 req/sec
per instance and requests are usually satisfied in less than a second:

worker_processes auto;
worker_priority -10;
daemon off;
pid pids/nginx.pid;
error_log logs/error.log debug;

worker_rlimit_nofile 20000;

events {
worker_connections 10000;
multi_accept on;
}

Most all other config settings are "default" values. There's nothing in the
Kong logs that indicate these dropped responses are being processed by Kong.
There's no indication there aren't enough workers. These timeouts do not
happen in clusters, they are more like singletons.

Any advice on things I should look at or diagnosis possibilities? Thanks
very much, Ryan

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274267,274267#msg-274267

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Occasional successful upstreamed requests that don't get picked up

rnburn
In reply to this post by rnburn
Just to follow up on this post in case people find it in the future - we
haven't found any evidence that there is an issue with NGINX or Kong.  If
folks experience this kind of issue, would recommend examining the network
(to include tcpdumps, etc) to figure out what's really happening with
dropped requests.

Thanks for the reply, pbooth!

-Ryan

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274267,274670#msg-274670

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx