Keepalived Connections Reset after reloading the configuration (HUP Signal)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Keepalived Connections Reset after reloading the configuration (HUP Signal)

vedranf
Hello,

   I'm dealing with a problem. When reloading the nginx configuration,
all keepalived connections receive the  TCP reset flag after I send a
HUP signal to the master process. If I comment the line responsible for
enabling the keepalive feature in the configuration, the problem
disappear (nginx version is 0.9.7).


Thanks in advance,
            Jocelyn Mocquant

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,197927,197927#msg-197927


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Keepalived Connections Reset after reloading the configuration (HUP Signal)

Maxim Dounin
Hello!

On Fri, May 13, 2011 at 07:24:12PM -0400, Jocelyn Mocquant wrote:

> Hello,
>
>    I'm dealing with a problem. When reloading the nginx configuration,
> all keepalived connections receive the  TCP reset flag after I send a
> HUP signal to the master process. If I comment the line responsible for
> enabling the keepalive feature in the configuration, the problem
> disappear (nginx version is 0.9.7).

On configuration reloading nginx starts new worker processes with
new configuration and old worker processes are terminated as soon
as they finish processing of requests.  All keepalive connections
in old workers are closed accordingly.  HTTP/1.1 clients are
required to handle keepalive connection close, so this shoudln't
be a problem.

It's not clear why you see RST instead of normal close with
FIN/FIN+ACK/ACK (are you?), but this shouldn't the problem by
itself anyway.

Maxim Dounin

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Keepalived Connections Reset after reloading the configuration (HUP Signal)

Jocelyn
Thanks Maxim for your answer,

If I remove the keep alive directive, all happens as expected. Apparently, sending USR2 and WINCH solve the problem.

--
Regards
       Jocelyn


On Mon, May 16, 2011 at 4:33 AM, Maxim Dounin <[hidden email]> wrote:
Hello!

On Fri, May 13, 2011 at 07:24:12PM -0400, Jocelyn Mocquant wrote:

> Hello,
>
>    I'm dealing with a problem. When reloading the nginx configuration,
> all keepalived connections receive the  TCP reset flag after I send a
> HUP signal to the master process. If I comment the line responsible for
> enabling the keepalive feature in the configuration, the problem
> disappear (nginx version is 0.9.7).

On configuration reloading nginx starts new worker processes with
new configuration and old worker processes are terminated as soon
as they finish processing of requests.  All keepalive connections
in old workers are closed accordingly.  HTTP/1.1 clients are
required to handle keepalive connection close, so this shoudln't
be a problem.

It's not clear why you see RST instead of normal close with
FIN/FIN+ACK/ACK (are you?), but this shouldn't the problem by
itself anyway.

Maxim Dounin

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Keepalived Connections Reset after reloading the configuration (HUP Signal)

zakirenish
Hi,

We are seeing some fallout from this behaviour on keep-alive connections
when proxying traffic from remote POPs back to an Origin DC that, due to
latency, brings about a race condition in the socket shutdown sequence. The
result being the fateful "upstream prematurely closed connection while
reading response header from upstream" in the Remote POP.

A walk through of what we are seeing:

1. Config reload happens on the Origin DC.
2. Socket shutdowns are sent to all open, but not transacting, keep-alive
connections.
3. Remote POP sends data on a cached connection at around the same time as
#2, because at this point it has not received the disconnect yet.
4. Remote POP then receives the disconnect and errors with "upstream
prematurely..".

Ideally we should be able to have the Origin honour the
`worker_shutdown_timeout` (or some other setting) for keep-alive
connections. That way we would be able to use the `keepalive_timeout`
setting for upstreams to ensure the upstream's cached connections always
time out before a worker is shutdown. Would that be possible or is there
another way to mitigate this scenario?

/David

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,197927,283564#msg-283564

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Keepalived Connections Reset after reloading the configuration (HUP Signal)

Maxim Dounin
Hello!

On Thu, Mar 28, 2019 at 08:49:48PM -0400, darthhexx wrote:

> Hi,
>
> We are seeing some fallout from this behaviour on keep-alive connections
> when proxying traffic from remote POPs back to an Origin DC that, due to
> latency, brings about a race condition in the socket shutdown sequence. The
> result being the fateful "upstream prematurely closed connection while
> reading response header from upstream" in the Remote POP.
>
> A walk through of what we are seeing:
>
> 1. Config reload happens on the Origin DC.
> 2. Socket shutdowns are sent to all open, but not transacting, keep-alive
> connections.
> 3. Remote POP sends data on a cached connection at around the same time as
> #2, because at this point it has not received the disconnect yet.
> 4. Remote POP then receives the disconnect and errors with "upstream
> prematurely..".
>
> Ideally we should be able to have the Origin honour the
> `worker_shutdown_timeout` (or some other setting) for keep-alive
> connections. That way we would be able to use the `keepalive_timeout`
> setting for upstreams to ensure the upstream's cached connections always
> time out before a worker is shutdown. Would that be possible or is there
> another way to mitigate this scenario?

As per HTTP RFC, clients are expected to be prepared to such close
events (https://tools.ietf.org/html/rfc2616#section-8.1.4).  In
nginx, if an error happens when nginx tries to use a cached
connection, it automatically tries again as long as it is
permitted by "proxy_next_upstream"
(http://nginx.org/r/proxy_next_upstream).

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Keepalived Connections Reset after reloading the configuration (HUP Signal)

zakirenish
In reply to this post by zakirenish
Having similar issue. Clients connections getting closed on reload.

Maxim, would you point me to a code location where I can make server wait
for `worker_shutdown_timeout` before terminating the conenction?
either through `ngx_set_shutdown_timer(cycle);` or sleep on
`ccf->shutdown_timeout`?

I am looking through `ngx_process_cycle.c` where the QUIT signal sent on the
channel. But having hard time figuring out what I should change.

I would really appreciate it!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,197927,289405#msg-289405

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx