Quantcast

killed child process

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

killed child process

Alex Samad
Hi

so I have lots of clients on long lived tcp connections , getting rp into 2 back end app servers

I had a line in my error log, saying one of the upstream was failed caused it timeout - 


then I got this

2017/05/18 13:30:42 [notice] 2662#2662: exiting
2017/05/18 13:30:42 [notice] 2662#2662: exit
2017/05/18 13:30:42 [notice] 2661#2661: signal 17 (SIGCHLD) received
2017/05/18 13:30:42 [notice] 2661#2661: worker process 2662 exited with code 0
2017/05/18 13:30:42 [notice] 2661#2661: signal 29 (SIGIO) received


I am not sure what initiated this ? there are no cron jobs running at that time 

and it looks like a lot of my long lived tcp session where TCP-FIN'ed.


I also checked my audit logs to see what command / process run at that time ... nothing that signaled or initiated a nginx reload ...

Alex





_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: killed child process

Maxim Dounin
Hello!

On Fri, May 19, 2017 at 11:28:05AM +1000, Alex Samad wrote:

> Hi
>
> so I have lots of clients on long lived tcp connections , getting rp into 2
> back end app servers
>
> I had a line in my error log, saying one of the upstream was failed caused
> it timeout -
>
>
> then I got this
>
> 2017/05/18 13:30:42 [notice] 2662#2662: exiting
> 2017/05/18 13:30:42 [notice] 2662#2662: exit
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 17 (SIGCHLD) received
> 2017/05/18 13:30:42 [notice] 2661#2661: worker process 2662 exited with
> code 0
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 29 (SIGIO) received
>
>
> I am not sure what initiated this ? there are no cron jobs running at that
> time
>
> and it looks like a lot of my long lived tcp session where TCP-FIN'ed.
>
>
> I also checked my audit logs to see what command / process run at that time
> ... nothing that signaled or initiated a nginx reload ...

Try searching error log for prior messages from process 2662 (note
that each error message have process ID in it, "... 2662#..."), it
should help to identify why it exited.

Most likley it was an old worker previously instructed to shutdown
gracefully due to a configuration reload, and it finished serving
all the existing clients and exited.  If it's the case, you should
see something like "gracefully shutting down" from the worker
somewhere earlier in logs, and "reconfiguring" from master just
before it.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: killed child process

Alex Samad
Yes this exactly, I ended up been schooled by support :)

Seems like my understanding  of graceful shutdown / reload ..

for the list and the archives

No keep alive for http1.0, has to be http1.1


client -> nginx keep alive session - these are shutdown once the current request is completed 
nginx -> backend server keep alive session - these are shutdown once the current request is completed

client -> nginx - websockets ... these are kept alive until the client closes it
nginx -> backend - websocket ... these are kept alive until the client closes it


client -> nginx -> backend ... ssl passthrough ?  I presume these are kept alive until either the client or the backend closes it.


it would be nice to allow a reload/refresh but keep keep-alive session open until the client closes it

Alex







On 19 May 2017 at 23:10, Maxim Dounin <[hidden email]> wrote:
Hello!

On Fri, May 19, 2017 at 11:28:05AM +1000, Alex Samad wrote:

> Hi
>
> so I have lots of clients on long lived tcp connections , getting rp into 2
> back end app servers
>
> I had a line in my error log, saying one of the upstream was failed caused
> it timeout -
>
>
> then I got this
>
> 2017/05/18 13:30:42 [notice] 2662#2662: exiting
> 2017/05/18 13:30:42 [notice] 2662#2662: exit
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 17 (SIGCHLD) received
> 2017/05/18 13:30:42 [notice] 2661#2661: worker process 2662 exited with
> code 0
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 29 (SIGIO) received
>
>
> I am not sure what initiated this ? there are no cron jobs running at that
> time
>
> and it looks like a lot of my long lived tcp session where TCP-FIN'ed.
>
>
> I also checked my audit logs to see what command / process run at that time
> ... nothing that signaled or initiated a nginx reload ...

Try searching error log for prior messages from process 2662 (note
that each error message have process ID in it, "... 2662#..."), it
should help to identify why it exited.

Most likley it was an old worker previously instructed to shutdown
gracefully due to a configuration reload, and it finished serving
all the existing clients and exited.  If it's the case, you should
see something like "gracefully shutting down" from the worker
somewhere earlier in logs, and "reconfiguring" from master just
before it.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: killed child process

nginx mailing list
... and you would end up with connections serving different content (as per different configuration) on the long run, leading potentially to an increased number of problems.
How would you shut them down, if not gracefully?

If you want to keep long-lived connections open, do not make changes server-side, such as asking it to reopen configuration (and thus to apply it to every worker).
---
B. R.

On Fri, May 19, 2017 at 11:40 PM, Alex Samad <[hidden email]> wrote:
Yes this exactly, I ended up been schooled by support :)

Seems like my understanding  of graceful shutdown / reload ..

for the list and the archives

No keep alive for http1.0, has to be http1.1


client -> nginx keep alive session - these are shutdown once the current request is completed 
nginx -> backend server keep alive session - these are shutdown once the current request is completed

client -> nginx - websockets ... these are kept alive until the client closes it
nginx -> backend - websocket ... these are kept alive until the client closes it


client -> nginx -> backend ... ssl passthrough ?  I presume these are kept alive until either the client or the backend closes it.


it would be nice to allow a reload/refresh but keep keep-alive session open until the client closes it

Alex







On 19 May 2017 at 23:10, Maxim Dounin <[hidden email]> wrote:
Hello!

On Fri, May 19, 2017 at 11:28:05AM +1000, Alex Samad wrote:

> Hi
>
> so I have lots of clients on long lived tcp connections , getting rp into 2
> back end app servers
>
> I had a line in my error log, saying one of the upstream was failed caused
> it timeout -
>
>
> then I got this
>
> 2017/05/18 13:30:42 [notice] 2662#2662: exiting
> 2017/05/18 13:30:42 [notice] 2662#2662: exit
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 17 (SIGCHLD) received
> 2017/05/18 13:30:42 [notice] 2661#2661: worker process 2662 exited with
> code 0
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 29 (SIGIO) received
>
>
> I am not sure what initiated this ? there are no cron jobs running at that
> time
>
> and it looks like a lot of my long lived tcp session where TCP-FIN'ed.
>
>
> I also checked my audit logs to see what command / process run at that time
> ... nothing that signaled or initiated a nginx reload ...

Try searching error log for prior messages from process 2662 (note
that each error message have process ID in it, "... 2662#..."), it
should help to identify why it exited.

Most likley it was an old worker previously instructed to shutdown
gracefully due to a configuration reload, and it finished serving
all the existing clients and exited.  If it's the case, you should
see something like "gracefully shutting down" from the worker
somewhere earlier in logs, and "reconfiguring" from master just
before it.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: killed child process

Alex Samad
Well at least in my case, I can ask the application to make an orderly reconnect.  Where if nginx does it it just closes the connection.

The option to do it seems like better than having no option.

Alex

On 20 May 2017 at 21:11, B.R. via nginx <[hidden email]> wrote:
... and you would end up with connections serving different content (as per different configuration) on the long run, leading potentially to an increased number of problems.
How would you shut them down, if not gracefully?

If you want to keep long-lived connections open, do not make changes server-side, such as asking it to reopen configuration (and thus to apply it to every worker).
---
B. R.

On Fri, May 19, 2017 at 11:40 PM, Alex Samad <[hidden email]> wrote:
Yes this exactly, I ended up been schooled by support :)

Seems like my understanding  of graceful shutdown / reload ..

for the list and the archives

No keep alive for http1.0, has to be http1.1


client -> nginx keep alive session - these are shutdown once the current request is completed 
nginx -> backend server keep alive session - these are shutdown once the current request is completed

client -> nginx - websockets ... these are kept alive until the client closes it
nginx -> backend - websocket ... these are kept alive until the client closes it


client -> nginx -> backend ... ssl passthrough ?  I presume these are kept alive until either the client or the backend closes it.


it would be nice to allow a reload/refresh but keep keep-alive session open until the client closes it

Alex







On 19 May 2017 at 23:10, Maxim Dounin <[hidden email]> wrote:
Hello!

On Fri, May 19, 2017 at 11:28:05AM +1000, Alex Samad wrote:

> Hi
>
> so I have lots of clients on long lived tcp connections , getting rp into 2
> back end app servers
>
> I had a line in my error log, saying one of the upstream was failed caused
> it timeout -
>
>
> then I got this
>
> 2017/05/18 13:30:42 [notice] 2662#2662: exiting
> 2017/05/18 13:30:42 [notice] 2662#2662: exit
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 17 (SIGCHLD) received
> 2017/05/18 13:30:42 [notice] 2661#2661: worker process 2662 exited with
> code 0
> 2017/05/18 13:30:42 [notice] 2661#2661: signal 29 (SIGIO) received
>
>
> I am not sure what initiated this ? there are no cron jobs running at that
> time
>
> and it looks like a lot of my long lived tcp session where TCP-FIN'ed.
>
>
> I also checked my audit logs to see what command / process run at that time
> ... nothing that signaled or initiated a nginx reload ...

Try searching error log for prior messages from process 2662 (note
that each error message have process ID in it, "... 2662#..."), it
should help to identify why it exited.

Most likley it was an old worker previously instructed to shutdown
gracefully due to a configuration reload, and it finished serving
all the existing clients and exited.  If it's the case, you should
see something like "gracefully shutting down" from the worker
somewhere earlier in logs, and "reconfiguring" from master just
before it.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Loading...