Quantcast

Reload of NGinX doesnt kill some of the older worker processes

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Reload of NGinX doesnt kill some of the older worker processes

AjaySawant
I am facing an issue where once I issued a reload to the NGinX binary, few
of the older worker processes are not dying. They still remain orphaned.

This is the configuration before issuing a reload :

[poduser@ucfc2z3a-1582-lb8-nginx1 logs]$ ps -ef | grep nginx
poduser  12540 22030  0 06:39 ?        00:00:00 nginx: worker process      
   
poduser  12541 22030  0 06:39 ?        00:00:00 nginx: worker process      
   
poduser  12762 11601  0 06:41 pts/0    00:00:00 grep nginx
poduser  22030     1  1 May12 ?        00:49:01 nginx: master process
/u01/app/Oracle_Nginx/sbin/nginx
poduser  23528 22030  0 May12 ?        00:00:22 nginx: worker process      
   
poduser  24950 22030  0 May12 ?        00:00:22 nginx: worker process

Configuration after issuing a relaod

[poduser@ucfc2z3a-1582-lb8-nginx1 logs]$ ps -ef | grep nginx
poduser  13280 22030  2 06:45 ?        00:00:00 nginx: worker process      
   
poduser  13281 22030  2 06:45 ?        00:00:00 nginx: worker process      
   
poduser  13323 11601  0 06:45 pts/0    00:00:00 grep nginx
poduser  22030     1  1 May12 ?        00:49:02 nginx: master process
/u01/app/Oracle_Nginx/sbin/nginx
poduser  23528 22030  0 May12 ?        00:00:22 nginx: worker process      
   
poduser  24950 22030  0 May12 ?        00:00:22 nginx: worker process    

If you notice, there are two worker processes orphaned with PID's 23528 and
24950. Could someone please explain the cause for this, as to why few of the
worker processes are orphaned?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274213,274213#msg-274213

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reload of NGinX doesnt kill some of the older worker processes

Maxim Dounin
Hello!

On Mon, May 15, 2017 at 06:54:47AM -0400, shivramg94 wrote:

> I am facing an issue where once I issued a reload to the NGinX binary, few
> of the older worker processes are not dying. They still remain orphaned.
>
> This is the configuration before issuing a reload :
>
> [poduser@ucfc2z3a-1582-lb8-nginx1 logs]$ ps -ef | grep nginx
> poduser  12540 22030  0 06:39 ?        00:00:00 nginx: worker process      
>    
> poduser  12541 22030  0 06:39 ?        00:00:00 nginx: worker process      
>    
> poduser  12762 11601  0 06:41 pts/0    00:00:00 grep nginx
> poduser  22030     1  1 May12 ?        00:49:01 nginx: master process
> /u01/app/Oracle_Nginx/sbin/nginx
> poduser  23528 22030  0 May12 ?        00:00:22 nginx: worker process      
>    
> poduser  24950 22030  0 May12 ?        00:00:22 nginx: worker process
>
> Configuration after issuing a relaod
>
> [poduser@ucfc2z3a-1582-lb8-nginx1 logs]$ ps -ef | grep nginx
> poduser  13280 22030  2 06:45 ?        00:00:00 nginx: worker process      
>    
> poduser  13281 22030  2 06:45 ?        00:00:00 nginx: worker process      
>    
> poduser  13323 11601  0 06:45 pts/0    00:00:00 grep nginx
> poduser  22030     1  1 May12 ?        00:49:02 nginx: master process
> /u01/app/Oracle_Nginx/sbin/nginx
> poduser  23528 22030  0 May12 ?        00:00:22 nginx: worker process      
>    
> poduser  24950 22030  0 May12 ?        00:00:22 nginx: worker process    
>
> If you notice, there are two worker processes orphaned with PID's 23528 and
> 24950. Could someone please explain the cause for this, as to why few of the
> worker processes are orphaned?

From the "nginx: worker process" process titles it look like these
processes were not notified they are expected to shut down.  This
may happen if notification fail for some reason - for example,
due to inapropriate system limits.  Try checking error logs, it
may shed some light on what happended.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reload of NGinX doesnt kill some of the older worker processes

AjaySawant
Hi Maxim,

This is what I could find in the error logs

2017/05/15 11:32:18 [notice] 21499#0: signal process started
2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (88: Socket operation
on non-socket)
2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (88: Socket operation
on non-socket)
2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (88: Socket operation
on non-socket)
2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (88: Socket operation
on non-socket)
2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (32: Broken pipe)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274213,274229#msg-274229

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reload of NGinX doesnt kill some of the older worker processes

AjaySawant
At times, the error logs say

2017/05/15 11:37:01 [notice] 22229#0: signal process started
2017/05/15 11:37:02 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
2017/05/15 11:37:02 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
2017/05/15 11:37:04 [alert] 22030#0: sendmsg() failed (9: Bad file
descriptor)
2017/05/15 11:37:04 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
2017/05/15 11:37:04 [alert] 22030#0: sendmsg() failed (9: Bad file
descriptor)
2017/05/15 11:37:04 [alert] 22030#0: sendmsg() failed (32: Broken pipe)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274213,274230#msg-274230

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reload of NGinX doesnt kill some of the older worker processes

Maxim Dounin
In reply to this post by AjaySawant
Hello!

On Mon, May 15, 2017 at 11:37:54AM -0400, shivramg94 wrote:

> Hi Maxim,
>
> This is what I could find in the error logs
>
> 2017/05/15 11:32:18 [notice] 21499#0: signal process started
> 2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (88: Socket operation
> on non-socket)
> 2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
> 2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (88: Socket operation
> on non-socket)
> 2017/05/15 11:32:19 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
> 2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (88: Socket operation
> on non-socket)
> 2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (32: Broken pipe)
> 2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (88: Socket operation
> on non-socket)
> 2017/05/15 11:32:20 [alert] 22030#0: sendmsg() failed (32: Broken pipe)

So, clearly comminication between master process and these workers
is now broken.  You have to look for earlier errors to find out
what exactly caused this - likely it hit resource limit on number
of open files or something like this.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reload of NGinX doesnt kill some of the older worker processes

AjaySawant
Earlier, it says the pid file doesn't exist even though the master and
worker processes were running.


2017/05/12 15:35:41 [notice] 19042#0: signal process started
2017/05/12 15:35:41 [error] 19042#0: open() "/u01/data/logs/nginx.pid"
failed (2: No such file or directory)

Can the above issue ( where the nginx.pid file went missing) and the
communication break up between the master and the worker processes be
correlated?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274213,274233#msg-274233

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reload of NGinX doesnt kill some of the older worker processes

Maxim Dounin
Hello!

On Mon, May 15, 2017 at 12:41:37PM -0400, shivramg94 wrote:

> Earlier, it says the pid file doesn't exist even though the master and
> worker processes were running.
>
>
> 2017/05/12 15:35:41 [notice] 19042#0: signal process started
> 2017/05/12 15:35:41 [error] 19042#0: open() "/u01/data/logs/nginx.pid"
> failed (2: No such file or directory)
>
> Can the above issue ( where the nginx.pid file went missing) and the
> communication break up between the master and the worker processes be
> correlated?

This message is from signal process, 19042, and indicate only that
"nginx -s" was used incorrectly (likely with wrong configuration).  
And not, this is not something that can break master / worker
communication.  Look for 'alert' and 'crit' messages.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Loading...