SSLEngine closed already exception triggered by reload

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SSLEngine closed already exception triggered by reload

wld75
I noticed that if I setup a simple scenario where a client is making
concurrent requests on a server with nginx configured as a reverse proxy and
SSL traffic termination endpoint, if I trigger a reload with 'nginx -s
reload' mid requests,  often times the client will throw an
'javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)' exception.

I'm using Scala with the Play framework, which uses netty under the hood.

Is there any configuration that could avoid these exceptions being thrown?

I cannot reproduce it using for example using Play to serve HTTPS, so I can
possibly rule out a problem in the client and think it is a problem with
nginx?

Thank you.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281786,281786#msg-281786

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: SSLEngine closed already exception triggered by reload

wld75
One setting that I noticed mitigates the problem is to use `lingering_close
always;` however in our infrastructure this can lead to the build up of
worker processes (for the duration of the lingering_timeout).  What are the
advantages and drawbacks of using this setting?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281786,281801#msg-281801

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: SSLEngine closed already exception triggered by reload

Maxim Dounin
In reply to this post by wld75
Hello!

On Mon, Nov 05, 2018 at 09:14:33AM -0500, nginxuser2018 wrote:

> I noticed that if I setup a simple scenario where a client is making
> concurrent requests on a server with nginx configured as a reverse proxy and
> SSL traffic termination endpoint, if I trigger a reload with 'nginx -s
> reload' mid requests,  often times the client will throw an
> 'javax.net.ssl.SSLException: SSLEngine closed already
> at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)' exception.
>
> I'm using Scala with the Play framework, which uses netty under the hood.
>
> Is there any configuration that could avoid these exceptions being thrown?
>
> I cannot reproduce it using for example using Play to serve HTTPS, so I can
> possibly rule out a problem in the client and think it is a problem with
> nginx?

On Tue, Nov 06, 2018 at 08:49:07AM -0500, nginxuser2018 wrote:

> One setting that I noticed mitigates the problem is to use `lingering_close
> always;` however in our infrastructure this can lead to the build up of
> worker processes (for the duration of the lingering_timeout).  What are the
> advantages and drawbacks of using this setting?

Upon configuration reload, nginx will close connections after it
finishes processing already active requests in these connections.  
And given that "lingering_close always;" helps, I think there are
two possible cases here:

1. Closing the connection by nginx happens and the wrong time,
right before the next request is received on this connection, so
RST is sent on the connection before the client is able get the
response and the connection close.  If this is indeed the case,
using "lingering_close always;" might be the right thing to do -
or, alternatively, lingering close automatic logic might need to
be improved.

2. The client isn't smart enough to check that the connection is
closed before sending the next request, and/or isn't smart enough
to recover from asynchronous close events (a good description can
be found in RFC 2616, "8.1.4 Practical Considerations",
https://tools.ietf.org/html/rfc2616#section-8.1.4).  In this case,
a proper fix would be to improve the client.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: SSLEngine closed already exception triggered by reload

wld75
Hi Maxim,

We have tried different settings with 'lingering_close always;' and
'lingering_time', 'lingering_timeout' up to 240s with no success.

Would you be able to confirm whether it is an nginx problem in the lingering
close automatic logic as you mentioned if I provide an example to reproduce
it?

Thanks,
Dario



Maxim Dounin Wrote:
-------------------------------------------------------

> Hello!
>
> On Mon, Nov 05, 2018 at 09:14:33AM -0500, nginxuser2018 wrote:
>
> > I noticed that if I setup a simple scenario where a client is making
> > concurrent requests on a server with nginx configured as a reverse
> proxy and
> > SSL traffic termination endpoint, if I trigger a reload with 'nginx
> -s
> > reload' mid requests,  often times the client will throw an
> > 'javax.net.ssl.SSLException: SSLEngine closed already
> > at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)'
> exception.
> >
> > I'm using Scala with the Play framework, which uses netty under the
> hood.
> >
> > Is there any configuration that could avoid these exceptions being
> thrown?
> >
> > I cannot reproduce it using for example using Play to serve HTTPS,
> so I can
> > possibly rule out a problem in the client and think it is a problem
> with
> > nginx?
>
> On Tue, Nov 06, 2018 at 08:49:07AM -0500, nginxuser2018 wrote:
>
> > One setting that I noticed mitigates the problem is to use
> `lingering_close
> > always;` however in our infrastructure this can lead to the build up
> of
> > worker processes (for the duration of the lingering_timeout).  What
> are the
> > advantages and drawbacks of using this setting?
>
> Upon configuration reload, nginx will close connections after it
> finishes processing already active requests in these connections.  
> And given that "lingering_close always;" helps, I think there are
> two possible cases here:
>
> 1. Closing the connection by nginx happens and the wrong time,
> right before the next request is received on this connection, so
> RST is sent on the connection before the client is able get the
> response and the connection close.  If this is indeed the case,
> using "lingering_close always;" might be the right thing to do -
> or, alternatively, lingering close automatic logic might need to
> be improved.
>
> 2. The client isn't smart enough to check that the connection is
> closed before sending the next request, and/or isn't smart enough
> to recover from asynchronous close events (a good description can
> be found in RFC 2616, "8.1.4 Practical Considerations",
> https://tools.ietf.org/html/rfc2616#section-8.1.4).  In this case,
> a proper fix would be to improve the client.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281786,281894#msg-281894

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: SSLEngine closed already exception triggered by reload

Maxim Dounin
Hello!

On Fri, Nov 09, 2018 at 12:42:48PM -0500, nginxuser2018 wrote:

> We have tried different settings with 'lingering_close always;' and
> 'lingering_time', 'lingering_timeout' up to 240s with no success.
>
> Would you be able to confirm whether it is an nginx problem in the lingering
> close automatic logic as you mentioned if I provide an example to reproduce
> it?

If 'lingering_close always;' does not help, in contrast to what
you wrote in your second message, this is certainly client's
problem.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx