why reuseport don't increases throughoutput?

[blason]

Hello

It shows the new feature reusport from v1.9.1 can increase the QPS by 2-3
times than accept_mutex on and off in this article
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/.  But the
result is disappointed when we have the test in our production with
V1.11.2.2. It don't even have the improvement but reduced, by 10%, dropped
from 42K QPS(with accept_mutex off) to 38K QPS(with reuseport enabled). and
it indeed reduce the latency. The two test cases have anything identicial
except that the later have reuseport enabled.  I wonder if I have missed
some special configuration.

Thanks.
Xiaofeng

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274281#msg-274281

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[CJ Ess]

I'd be interested in knowing more also - I know that the Linux 2.6 kernel is still really popular and didn't have the SO_REUSEPORT socket option (though it was in the include files and wouldn't cause an error if you referenced it), might that be what your running into?

On Wed, May 17, 2017 at 7:58 AM, fengx <[hidden email]> wrote:

Hello

It shows the new feature reusport from v1.9.1 can increase the QPS by 2-3
times than accept_mutex on and off in this article
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/.  But the
result is disappointed when we have the test in our production with
V1.11.2.2. It don't even have the improvement but reduced, by 10%, dropped
from 42K QPS(with accept_mutex off) to 38K QPS(with reuseport enabled). and
it indeed reduce the latency. The two test cases have anything identicial
except that the later have reuseport enabled.  I wonder if I have missed
some special configuration.

Thanks.
Xiaofeng

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274281#msg-274281

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[Zhang Chao]

Hi!

Just as the article says,

it can also mean that when a worker is stalled by a blocking operation, the block affects not only connections that the worker has already accepted, but also connection requests that the kernel has assigned to the worker since it became blocked.

Is there such a possibility that some blocking operations in your production result in the dropping of QPS?

On 17 May 2017 at 19:58:37, fengx ([hidden email]) wrote:

Hello

It shows the new feature reusport from v1.9.1 can increase the QPS by 2-3
times than accept_mutex on and off in this article
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/. But the
result is disappointed when we have the test in our production with
V1.11.2.2. It don't even have the improvement but reduced, by 10%, dropped
from 42K QPS(with accept_mutex off) to 38K QPS(with reuseport enabled). and
it indeed reduce the latency. The two test cases have anything identicial
except that the later have reuseport enabled. I wonder if I have missed
some special configuration.

Thanks.
Xiaofeng

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274281#msg-274281

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[Zhang Chao]

Hi!

Just as the article says,

it can also mean that when a worker is stalled by a blocking operation, the block affects not only connections that the worker has already accepted, but also connection requests that the kernel has assigned to the worker since it became blocked.

Is there such a possibility that some blocking operations in your production result in the dropping of QPS?

On 17 May 2017 at 19:58:37, fengx ([hidden email]) wrote:

Hello

It shows the new feature reusport from v1.9.1 can increase the QPS by 2-3
times than accept_mutex on and off in this article
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/. But the
result is disappointed when we have the test in our production with
V1.11.2.2. It don't even have the improvement but reduced, by 10%, dropped
from 42K QPS(with accept_mutex off) to 38K QPS(with reuseport enabled). and
it indeed reduce the latency. The two test cases have anything identicial
except that the later have reuseport enabled. I wonder if I have missed
some special configuration.

Thanks.
Xiaofeng

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274281#msg-274281

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[blason]

We have CentOS 7 with kernel 3.10.0 in the production. As known,
SO_REUSEPORT is introduced from kernel 3.9.  I also had the test on my own
laptop, Ubuntu 14, kernel 4.4 and got the similar result.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274398#msg-274398

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[blason]

There should been no blocking operation. At least, we have the same codebase
and same sample data in the two test cases.

In fact, our application is based on OpenResty with local redis instances.
On the 32 cores server, we have 22 nginx workers and 8 local redis
instances(shards). The lua codes should have rather simple business logic.
It accepts http requests, loads data from local redis via unix domain
sockets, do the simple transformation, and then write to the response.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274399#msg-274399

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[CJ Ess]

How about logging? If your using the default settings then nginx is logging directly to disk, and disk writes will block the worker. Do you see the same degradation with logging disabled or via syslog?

On Mon, May 22, 2017 at 10:59 PM, fengx <[hidden email]> wrote:

There should been no blocking operation. At least, we have the same codebase
and same sample data in the two test cases.

In fact, our application is based on OpenResty with local redis instances.
On the 32 cores server, we have 22 nginx workers and 8 local redis
instances(shards). The lua codes should have rather simple business logic.
It accepts http requests, loads data from local redis via unix domain
sockets, do the simple transformation, and then write to the response.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274399#msg-274399

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[Zhang Chao]

Well, i think may be you need an off-cpu flame graph :)

On 23 May 2017 at 10:59:29, fengx ([hidden email]) wrote:

There should been no blocking operation. At least, we have the same codebase
and same sample data in the two test cases.

In fact, our application is based on OpenResty with local redis instances.
On the 32 cores server, we have 22 nginx workers and 8 local redis
instances(shards). The lua codes should have rather simple business logic.
It accepts http requests, loads data from local redis via unix domain
sockets, do the simple transformation, and then write to the response.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274399#msg-274399

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[blason]

Hello, CJ Ess
Both of cases, access log is disabled and error log is enabled with level
ERROR. However, there only are a very few of errors in both cases, so I
think it does not matter with the logging. Anyway, I will have another test
with error log disabled later.

Hello, tokers
Yes, I will capture both of off-cpu and on-cpu flame graphs and share here
later.

Thanks.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274402#msg-274402

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

[CJ Ess]

Sounds like you've got all the common stuff well covered. I'm curious what the flame graphs show, I'd like to implement the port reuse feature for my employer if it works as described in the article you referenced. 

On Tue, May 23, 2017 at 3:39 AM, fengx <[hidden email]> wrote:

Hello, CJ Ess
Both of cases, access log is disabled and error log is enabled with level
ERROR. However, there only are a very few of errors in both cases, so I
think it does not matter with the logging. Anyway, I will have another test
with error log disabled later.

Hello, tokers
Yes, I will capture both of off-cpu and on-cpu flame graphs and share here
later.

Thanks.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274281,274402#msg-274402

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx