openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
Logging getting swamped with:

[crit] 1808#2740: *20747 SSL_read() failed (SSL: error:14095126:SSL
routines:ssl3_read_n:unexpected eof while reading) while keepalive

Related to: https://github.com/openssl/openssl/issues/10880
and this commit:
https://github.com/openssl/openssl/commit/db943f43a60d1b5b1277e4b5317e8f288e7a0a3a

Question: does this need to resolved in openssl or nginx ?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287377#msg-287377

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

J.R.
> [crit] 1808#2740: *20747 SSL_read() failed (SSL: error:14095126:SSL
> routines:ssl3_read_n:unexpected eof while reading) while keepalive

Just curious, but were you getting these errors while running 1.1.1d
or they just started after upgrade to 1.1.1e ?
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
After using 1.1.1e, see also the commit where an explicit entry has been
added.
nginx just reports back what openssl passes, if this was unexpected (none
critical) nginx needs to be patched, if not this openssl workaround (10880)
needs to be changed.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287380#msg-287380

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

RE: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Reinis Rozitis
> After using 1.1.1e, see also the commit where an explicit entry has been
> added.
> nginx just reports back what openssl passes, if this was unexpected (none
> critical) nginx needs to be patched, if not this openssl workaround (10880)
> needs to be changed.

Any comment on this from any nginx devs?
Been running 1.1.1c for some time and out of curiosity upgraded to 1.1.1e and indeed there are a lot of "(SSL: error:14095126:SSL routines:ssl3_read_n:unexpected eof while reading)".

Is it "safe" to temporary revert the patch to reduce the noise (as per the github thread - the EOF (other than the "data loss") most likely  has been there previously just not being returned as error) or are there more deeper problems with openssl/tls 1.3 etc?


Also since there are no plans to implement quic even in openssl 3.0 does it maybe make sense to compile nginx with BoringSSL?

rr



_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Sergey Kandaurov
In reply to this post by Olaf van der Spek

> On 18 Mar 2020, at 14:17, itpp2012 <[hidden email]> wrote:
>
> Logging getting swamped with:
>
> [crit] 1808#2740: *20747 SSL_read() failed (SSL: error:14095126:SSL
> routines:ssl3_read_n:unexpected eof while reading) while keepalive
>
> Related to: https://github.com/openssl/openssl/issues/10880
> and this commit:
> https://github.com/openssl/openssl/commit/db943f43a60d1b5b1277e4b5317e8f288e7a0a3a
>
> Question: does this need to resolved in openssl or nginx ?

So, they deliberately changed existing behaviour, known since
at least OpenSSL 0.9.7, in the stable branch which should not
be targeted (per their words) for introducing behaviour changes.
That is unfortunate and beyond explanation.

To simply shut up the crit, this would require such an ugly hack.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -2301,7 +2301,13 @@ ngx_ssl_handle_recv(ngx_connection_t *c,
     c->ssl->no_wait_shutdown = 1;
     c->ssl->no_send_shutdown = 1;
 
-    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
+    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0
+#ifdef SSL_R_UNEXPECTED_EOF_WHILE_READING
+        || (sslerr == SSL_ERROR_SSL && ERR_GET_REASON(ERR_peek_error())
+                                       == SSL_R_UNEXPECTED_EOF_WHILE_READING)
+#endif
+       )
+    {
         ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
                        "peer shutdown SSL cleanly");
         return NGX_DONE;


--
Sergey Kandaurov

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Maxim Dounin
Hello!

On Fri, Mar 20, 2020 at 10:41:32AM +0300, Sergey Kandaurov wrote:

>
> > On 18 Mar 2020, at 14:17, itpp2012 <[hidden email]> wrote:
> >
> > Logging getting swamped with:
> >
> > [crit] 1808#2740: *20747 SSL_read() failed (SSL: error:14095126:SSL
> > routines:ssl3_read_n:unexpected eof while reading) while keepalive
> >
> > Related to: https://github.com/openssl/openssl/issues/10880
> > and this commit:
> > https://github.com/openssl/openssl/commit/db943f43a60d1b5b1277e4b5317e8f288e7a0a3a
> >
> > Question: does this need to resolved in openssl or nginx ?
>
> So, they deliberately changed existing behaviour, known since
> at least OpenSSL 0.9.7, in the stable branch which should not
> be targeted (per their words) for introducing behaviour changes.
> That is unfortunate and beyond explanation.
>
> To simply shut up the crit, this would require such an ugly hack.
>
> diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
> --- a/src/event/ngx_event_openssl.c
> +++ b/src/event/ngx_event_openssl.c
> @@ -2301,7 +2301,13 @@ ngx_ssl_handle_recv(ngx_connection_t *c,
>      c->ssl->no_wait_shutdown = 1;
>      c->ssl->no_send_shutdown = 1;
>  
> -    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
> +    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0
> +#ifdef SSL_R_UNEXPECTED_EOF_WHILE_READING
> +        || (sslerr == SSL_ERROR_SSL && ERR_GET_REASON(ERR_peek_error())
> +                                       == SSL_R_UNEXPECTED_EOF_WHILE_READING)
> +#endif
> +       )
> +    {
>          ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
>                         "peer shutdown SSL cleanly");
>          return NGX_DONE;

I think a separate condition in an #ifdef might be preferred here,
probably with better debug logging as well.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
Maxim Dounin Wrote:
-------------------------------------------------------
> On Fri, Mar 20, 2020 at 10:41:32AM +0300, Sergey Kandaurov wrote:
> > > On 18 Mar 2020, at 14:17, itpp2012 <[hidden email]>
wrote:
> > > Question: does this need to resolved in openssl or nginx ?

> > So, they deliberately changed existing behaviour, known since
> > at least OpenSSL 0.9.7, in the stable branch which should not
> > be targeted (per their words) for introducing behaviour changes.
> > That is unfortunate and beyond explanation.

> I think a separate condition in an #ifdef might be preferred here,
> probably with better debug logging as well.

I'd prefer an openssl fix but can we now assume nginx prefers a nginx fix ?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287404#msg-287404

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Maxim Dounin
Hello!

On Fri, Mar 20, 2020 at 09:54:08AM -0400, itpp2012 wrote:

> Maxim Dounin Wrote:
> -------------------------------------------------------
> > On Fri, Mar 20, 2020 at 10:41:32AM +0300, Sergey Kandaurov wrote:
> > > > On 18 Mar 2020, at 14:17, itpp2012 <[hidden email]>
> wrote:
> > > > Question: does this need to resolved in openssl or nginx ?
>
> > > So, they deliberately changed existing behaviour, known since
> > > at least OpenSSL 0.9.7, in the stable branch which should not
> > > be targeted (per their words) for introducing behaviour changes.
> > > That is unfortunate and beyond explanation.
>
> > I think a separate condition in an #ifdef might be preferred here,
> > probably with better debug logging as well.
>
> I'd prefer an openssl fix but can we now assume nginx prefers a nginx fix ?

Well, reverting OpenSSL behaviour back to one existed for years
would be awesome.  Unfortunately, this might never happen, as
OpenSSL's team usually don't care.

Also, even if this will happen, there will be at least some
versions of OpenSSL when things behave incorrectly.

As such, we certainly have to consider how to fix it on nginx
side.  Whether or not we'll commit the fix is a different
question.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
In reply to this post by Olaf van der Spek
Other not as often as the primary but all related to 1.1.1e:

All [crit]:
SSL_read() failed (SSL: error:14095126:SSL routines:ssl3_read_n:unexpected
eof while reading) while processing HTTP/2 connection
SSL_read() failed (SSL: error:14095126:SSL routines:ssl3_read_n:unexpected
eof while reading) while keepalive
SSL_read() failed (SSL: error:14095126:SSL routines:ssl3_read_n:unexpected
eof while reading) while waiting for request
SSL_do_handshake() failed (SSL: error:14095126:SSL
routines:ssl3_read_n:unexpected eof while reading) while SSL handshaking

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287418#msg-287418

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
How about this as this catches all 3 while conditions:

+++ src/event/ngx_event_openssl.c
@@ -2318,

    c->ssl->no_wait_shutdown = 1;
    c->ssl->no_send_shutdown = 1;

    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
                       "peer shutdown SSL cleanly");
        return NGX_DONE;
    }

+    /* https://forum.nginx.org/read.php?2,287377  */
+    /* https://github.com/openssl/openssl/issues/11381  */
+#ifdef SSL_R_UNEXPECTED_EOF_WHILE_READING
+    if (sslerr == SSL_ERROR_SSL && ERR_GET_REASON(ERR_peek_error())
+                   == SSL_R_UNEXPECTED_EOF_WHILE_READING) {
+        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
+                       "ssl3_read_n:unexpected eof while reading");
+        return NGX_DONE;
+    }
+#endif
+
    ngx_ssl_connection_error(c, sslerr, err, "SSL_read() failed");

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287420#msg-287420

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Sergey Kandaurov

> On 22 Mar 2020, at 21:39, itpp2012 <[hidden email]> wrote:
>
> How about this as this catches all 3 while conditions:
>
> +++ src/event/ngx_event_openssl.c
> @@ -2318,
>
>    c->ssl->no_wait_shutdown = 1;
>    c->ssl->no_send_shutdown = 1;
>
>    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
>        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
>                       "peer shutdown SSL cleanly");
>        return NGX_DONE;
>    }
>
> +    /* https://forum.nginx.org/read.php?2,287377  */
> +    /* https://github.com/openssl/openssl/issues/11381  */
> +#ifdef SSL_R_UNEXPECTED_EOF_WHILE_READING
> +    if (sslerr == SSL_ERROR_SSL && ERR_GET_REASON(ERR_peek_error())
> +                   == SSL_R_UNEXPECTED_EOF_WHILE_READING) {
> +        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> +                       "ssl3_read_n:unexpected eof while reading");
> +        return NGX_DONE;
> +    }
> +#endif
> +
>    ngx_ssl_connection_error(c, sslerr, err, "SSL_read() failed");

How would this catch the reported error in SSL_do_handshake() ?
I'd replicate this check in ngx_ssl_handshake().
And probably for SSL_read_early_data, SSL_shutdown, SSL_peak,
(ok, we don't use SSL_peak), but this is a moot point.

--
Sergey Kandaurov

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
It doesn't and there are a few more for which this doesn't work either, it
needs a lot more work and testing.
I had a new concept patch but today decided to roll back to 1.1.1d and back
port 1.1.1e (de) patches only.
Only NGX_ERROR mitigates a truncation attack, not NGX_DONE (which is open
for debate).

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287426#msg-287426

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Maxim Dounin
In reply to this post by Sergey Kandaurov
Hello!

On Mon, Mar 23, 2020 at 02:04:36PM +0300, Sergey Kandaurov wrote:

>
> > On 22 Mar 2020, at 21:39, itpp2012 <[hidden email]> wrote:
> >
> > How about this as this catches all 3 while conditions:
> >
> > +++ src/event/ngx_event_openssl.c
> > @@ -2318,
> >
> >    c->ssl->no_wait_shutdown = 1;
> >    c->ssl->no_send_shutdown = 1;
> >
> >    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
> >        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> >                       "peer shutdown SSL cleanly");
> >        return NGX_DONE;
> >    }
> >
> > +    /* https://forum.nginx.org/read.php?2,287377  */
> > +    /* https://github.com/openssl/openssl/issues/11381  */
> > +#ifdef SSL_R_UNEXPECTED_EOF_WHILE_READING
> > +    if (sslerr == SSL_ERROR_SSL && ERR_GET_REASON(ERR_peek_error())
> > +                   == SSL_R_UNEXPECTED_EOF_WHILE_READING) {
> > +        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > +                       "ssl3_read_n:unexpected eof while reading");
> > +        return NGX_DONE;
> > +    }
> > +#endif
> > +
> >    ngx_ssl_connection_error(c, sslerr, err, "SSL_read() failed");
>
> How would this catch the reported error in SSL_do_handshake() ?
> I'd replicate this check in ngx_ssl_handshake().
> And probably for SSL_read_early_data, SSL_shutdown, SSL_peak,
> (ok, we don't use SSL_peak), but this is a moot point.

Given the session resumption issue[1], I tend to think the best
solution for now is to recommend to avoid using OpenSSL 1.1.1e.

[1] https://github.com/openssl/openssl/issues/11378

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Thomas Stephen Lee
On Mon, Mar 23, 2020 at 6:05 PM Maxim Dounin <[hidden email]> wrote:
Hello!

On Mon, Mar 23, 2020 at 02:04:36PM +0300, Sergey Kandaurov wrote:

>
> > On 22 Mar 2020, at 21:39, itpp2012 <[hidden email]> wrote:
> >
> > How about this as this catches all 3 while conditions:
> >
> > +++ src/event/ngx_event_openssl.c
> > @@ -2318,
> >
> >    c->ssl->no_wait_shutdown = 1;
> >    c->ssl->no_send_shutdown = 1;
> >
> >    if (sslerr == SSL_ERROR_ZERO_RETURN || ERR_peek_error() == 0) {
> >        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> >                       "peer shutdown SSL cleanly");
> >        return NGX_DONE;
> >    }
> >
> > +    /* https://forum.nginx.org/read.php?2,287377  */
> > +    /* https://github.com/openssl/openssl/issues/11381  */
> > +#ifdef SSL_R_UNEXPECTED_EOF_WHILE_READING
> > +    if (sslerr == SSL_ERROR_SSL && ERR_GET_REASON(ERR_peek_error())
> > +                   == SSL_R_UNEXPECTED_EOF_WHILE_READING) {
> > +        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0,
> > +                       "ssl3_read_n:unexpected eof while reading");
> > +        return NGX_DONE;
> > +    }
> > +#endif
> > +
> >    ngx_ssl_connection_error(c, sslerr, err, "SSL_read() failed");
>
> How would this catch the reported error in SSL_do_handshake() ?
> I'd replicate this check in ngx_ssl_handshake().
> And probably for SSL_read_early_data, SSL_shutdown, SSL_peak,
> (ok, we don't use SSL_peak), but this is a moot point.

Given the session resumption issue[1], I tend to think the best
solution for now is to recommend to avoid using OpenSSL 1.1.1e.

[1] https://github.com/openssl/openssl/issues/11378

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

Hi,

does

OpenSSL 1.1.1f.

fix the issue ?

thanks.

---
Lee

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: openssl 1.1.1e 14095126:SSL routines:ssl3_read_n

Olaf van der Spek
Thomas Stephen Lee Wrote:
-------------------------------------------------------
> Hi,
>
> does
>
> OpenSSL 1.1.1f.
>
> fix the issue ?

Yes.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287377,287532#msg-287532

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx