2 of 16 cores are constantly maxing out - how to balance the load?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

2 of 16 cores are constantly maxing out - how to balance the load?

Raffael Vogler
Hello!

I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The server is handling more than 10 million requests per hour.

https://imgur.com/a/iRZ7V

As you can see on the htop screenshot cores 6 and 7 are maxed out and that's the case constantly - even after restarting nginx those two cores stay at that level.

I wonder why is that so and how to balance the load more evenly?

Also I'm curious to know whether this might indicate a performance relevant issue or if it is most likely harmless and just looks odd.

> cat /etc/nginx/nginx.conf | grep -v '^\s*#'

user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
        worker_connections 768;
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        gzip_disable "msie6";
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}
Thanks
Raffael

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Lucas Rolff-2

If it’s the same two cores, it might be another process that uses the same two cores and thus happens to max out.

One very likely possibility would be interrupts from e.g. networking. You can check /proc/interrupts to see where interrupts from the network happens.

 

From: nginx <[hidden email]> on behalf of Raffael Vogler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, 11 January 2018 at 11.14
To: "[hidden email]" <[hidden email]>
Subject: 2 of 16 cores are constantly maxing out - how to balance the load?

 

Hello!

I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The server is handling more than 10 million requests per hour.

https://imgur.com/a/iRZ7V

As you can see on the htop screenshot cores 6 and 7 are maxed out and that's the case constantly - even after restarting nginx those two cores stay at that level.

I wonder why is that so and how to balance the load more evenly?

Also I'm curious to know whether this might indicate a performance relevant issue or if it is most likely harmless and just looks odd.

> cat /etc/nginx/nginx.conf | grep -v '^\s*#'
 
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
        worker_connections 768;
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        gzip_disable "msie6";
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}
Thanks
Raffael

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Raffael Vogler
Hey Lucas, your assumption seems to be correct. According to /proc/interrupts the following stats are significantly higher for those two cores (CPU5, CPU6 - 0-based indexing):

- CPU5: xen-percpu-ipi       callfuncsingle5
- CPU6: xen-percpu-ipi       callfuncsingle6
- CPU5: xen-pirq-msi-x     eth0-TxRx-0
- CPU6: xen-pirq-msi-x     eth0-TxRx-1
- CPU5,6: TLB shootdowns
- CPU5,6: Hypervisor callback interrupts

Is this something that can and should be optimized or is it simply a matter of fact due to the high load on the (fixed) available network card capacity?


On Thu, Jan 11, 2018 at 11:16 AM, Lucas Rolff <[hidden email]> wrote:

If it’s the same two cores, it might be another process that uses the same two cores and thus happens to max out.

One very likely possibility would be interrupts from e.g. networking. You can check /proc/interrupts to see where interrupts from the network happens.

 

From: nginx <[hidden email]> on behalf of Raffael Vogler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, 11 January 2018 at 11.14
To: "[hidden email]" <[hidden email]>
Subject: 2 of 16 cores are constantly maxing out - how to balance the load?

 

Hello!

I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The server is handling more than 10 million requests per hour.

https://imgur.com/a/iRZ7V

As you can see on the htop screenshot cores 6 and 7 are maxed out and that's the case constantly - even after restarting nginx those two cores stay at that level.

I wonder why is that so and how to balance the load more evenly?

Also I'm curious to know whether this might indicate a performance relevant issue or if it is most likely harmless and just looks odd.

> cat /etc/nginx/nginx.conf | grep -v '^\s*#'
 
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
        worker_connections 768;
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        gzip_disable "msie6";
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}
Thanks
Raffael

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Raffael Vogler, Chief Technology Officer

Yieldlove GmbH
Neuer Pferdemarkt 1, 20359 Hamburg  

www.yieldlove.com

[hidden email]
XING - LinkedIn
Skype: joyofdata 

Registernummer: HRB 127559; Registergericht: Amtsgericht Hamburg; USt-ID: DE815426709; Geschäftsführung:Benjamin Gries, Timo Hagenow, Ivan Tomic

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Raffael Vogler
Or would it make sense (if possible at all) to assign two or three more cores to networking interrupts?

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Lucas Rolff-2

In high traffic environments it generally make sense to “dedicate” a core to each RX and TX queue you have on the NIC – this way you lower the chances of a single core being overloaded from handling network and thus degrading performance.

 

And then at same time within nginx, map the individual processes to other cores.

 

So, let’s say say you have 8 cores and 1 RX and 1 TX queue:

Core 0: RX queue

Core 1: TX queue

Core 2 to 7: nginx processes

 

You’d then set nginx to 6 workers (if you’re not running other stuff on the box).

 

Now, in your case with php-fpm in the mix as well, controlling that can be hard ( not sure if you can pin php-fpm processes to cores ) – but for nginx and RX/TX queues, it’s for sure possible.

 

From: nginx <[hidden email]> on behalf of Raffael Vogler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, 11 January 2018 at 11.55
To: "[hidden email]" <[hidden email]>
Subject: Re: 2 of 16 cores are constantly maxing out - how to balance the load?

 

Or would it make sense (if possible at all) to assign two or three more cores to networking interrupts?


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Vlad K.
On 2018-01-11 11:59, Lucas Rolff wrote:
>
> Now, in your case with php-fpm in the mix as well, controlling that
> can be hard ( not sure if you can pin php-fpm processes to cores ) –
> but for nginx and RX/TX queues, it’s for sure possible.


Should be doable with cgroups / cpusets? CPUAffinity directive in the
service unit file ( see systemd.exec(5) )?



--
Vlad K.
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Peter Booth
Perhaps you should use pidstat to validate which processes are running on the two busy cores?

> On Jan 11, 2018, at 6:25 AM, Vlad K. <[hidden email]> wrote:
>
> On 2018-01-11 11:59, Lucas Rolff wrote:
>> Now, in your case with php-fpm in the mix as well, controlling that
>> can be hard ( not sure if you can pin php-fpm processes to cores ) –
>> but for nginx and RX/TX queues, it’s for sure possible.
>
>
> Should be doable with cgroups / cpusets? CPUAffinity directive in the service unit file ( see systemd.exec(5) )?
>
>
>
> --
> Vlad K.
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Raffael Vogler
> Perhaps you should use pidstat to validate which processes are running on the two busy cores?

Did that an can confirm that CPU 5 and 6 are not exclusively used by networking - but also by nginx and php-fpm.

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 2 of 16 cores are constantly maxing out - how to balance the load?

Raffael Vogler
In reply to this post by Lucas Rolff-2
> So, let’s say say you have 8 cores and 1 RX and 1 TX queue:
> Core 0: RX queue
> Core 1: TX queue
> Core 2 to 7: nginx processes

what tool or configuration file would I have to use to dedicate cores to processes?

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx