proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

PGNet Dev
 I'm running

        nginx -V
                nginx version: nginx/1.19.0 (pgnd Build)
                built with OpenSSL 1.1.1g  21 Apr 2020
                TLS SNI support enabled
                ...

It serves as front-end SSL termination, site host, and reverse-proxy to backend apps.

I'm trying to get a backend app to proxy_ssl_verify the proxy connection to it.

I have two self-signed certs:

One for "TLS Web Client Authentication, E-mail Protection"

        openssl x509 -in test.example.com.client.crt -text | egrep "Subject.*CN|DNS|TLS"
                Subject: C = US, ST = NY, L = New_York, O = example2.com, OU = myCA, CN = test.example.com, emailAddress = [hidden email]
                        TLS Web Client Authentication, E-mail Protection
                        DNS:test.example.com, DNS:www.test.example.com, DNS:localhost

and the other, for "TLS Web Server Authentication"

        openssl x509 -in test.example.com.server.crt -text | egrep "Subject.*CN|DNS|TLS"
                Subject: C = US, ST = NY, L = New_York, O = example2.com, OU = myCA, CN = test.example.com, emailAddress = [hidden email]
                        TLS Web Server Authentication
                        DNS:test.example.com, DNS:www.test.example.com, DNS:localhost

The certs 'match' CN & SAN, differing in "X509v3 Extended Key Usage".

Both are verified "OK" with my local CA cert

        openssl verify -CAfile myCA.crt.pem test.example.com.server.crt
                test.example.com.server.crt: OK

        openssl verify -CAfile /myCA.crt.pem test.example.com.client.crt
                test.example.com.client.crt: OK

My main nginx config includes,

        upstream test.example.com {
                server test.example.com:11111;
        }
        server {

                listen 10.10.10.1:443 ssl http2;
                server_name example.com;
                ...

                ssl_verify_client on;
                ssl_client_certificate  "/etc/ssl/nginx/myCA.crt";
                ssl_verify_depth 2;
                ssl_certificate         "/etc/ssl/nginx/example.com.server.crt";
                ssl_certificate_key     "/etc/ssl/nginx/example.com.server.key";
                ssl_trusted_certificate "/etc/ssl/nginx/myCA.crt";

                location /app1 {
                        proxy_pass                    https://test.example.com;
                        proxy_ssl_certificate         "/etc/ssl/nginx/test.example.com.client.crt";
                        proxy_ssl_certificate_key     "/etc/ssl/nginx/test.example.com.client.key";
                        proxy_ssl_trusted_certificate "/etc/ssl/nginx/myCA.crt";
                        proxy_ssl_verify       on;
                        proxy_ssl_verify_depth 2;
                        include includes/reverse-proxy.inc;
                }
        }

and the upstream config,

        server {
                listen 127.0.0.1:11111 ssl http2;
                server_name test.example.com;

                root /data/webapps/demo_app/;
                index index.php;
                expires -1;

                ssl_certificate        "/etc/ssl/nginx/test.example.com.server.crt";
                ssl_certificate_key    "/etc/ssl/nginx/test.example.com.server.key";

                ssl_client_certificate "/etc/ssl/nginx/myCA.crt";
                ssl_verify_client optional;
                ssl_verify_depth 2;

                location ~ \.php {
                        try_files $uri =404;
                        fastcgi_pass   phpfpm;
                        fastcgi_index  index.php;
                        fastcgi_param  PATH_INFO $fastcgi_script_name;
                        include        fastcgi_params;
                }

        }

access to

        https://example.com/app1

responds,

        502 Bad Gateway

logs, show an SSL handshake fail

        ...
        2020/05/29 19:00:06 [debug] 29419#29419: *7 SSL: TLSv1.3, cipher: "TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any Au=any Enc=CHACHA20/POLY1305(256) Mac=AEAD"
        2020/05/29 19:00:06 [debug] 29419#29419: *7 http upstream ssl handshake: "/app1/?"
        2020/05/29 19:00:06 [debug] 29419#29419: *7 X509_check_host(): no match
        2020/05/29 19:00:06 [error] 29419#29419: *7 upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream, client: 10.10.10.73, server: example.com, request: "GET /app1/ HTTP/2.0", upstream: "https://127.0.0.1:11111/app1/", host: "example.com"
        2020/05/29 19:00:06 [debug] 29419#29419: *7 http next upstream, 2
        ...

If I toggle

- ssl_verify_client on;
+ ssl_verify_client off;

then I'm able to connect to the backend site, as expected.

What exactly is NOT matching in the handshake?  CN & SAN do ...

&/or, is there a config problem above?

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Maxim Dounin
Hello!

On Fri, May 29, 2020 at 07:09:45PM -0700, PGNet Dev wrote:

>  I'm running
>
> nginx -V
> nginx version: nginx/1.19.0 (pgnd Build)
> built with OpenSSL 1.1.1g  21 Apr 2020
> TLS SNI support enabled
> ...
>
> It serves as front-end SSL termination, site host, and reverse-proxy to backend apps.
>
> I'm trying to get a backend app to proxy_ssl_verify the proxy connection to it.
>
> I have two self-signed certs:
>
> One for "TLS Web Client Authentication, E-mail Protection"
>
> openssl x509 -in test.example.com.client.crt -text | egrep "Subject.*CN|DNS|TLS"
>        Subject: C = US, ST = NY, L = New_York, O = example2.com, OU = myCA, CN = test.example.com, emailAddress = [hidden email]
>                TLS Web Client Authentication, E-mail Protection
>                DNS:test.example.com, DNS:www.test.example.com, DNS:localhost
>
> and the other, for "TLS Web Server Authentication"
>
> openssl x509 -in test.example.com.server.crt -text | egrep "Subject.*CN|DNS|TLS"
>        Subject: C = US, ST = NY, L = New_York, O = example2.com, OU = myCA, CN = test.example.com, emailAddress = [hidden email]
>                TLS Web Server Authentication
>                DNS:test.example.com, DNS:www.test.example.com, DNS:localhost
>
> The certs 'match' CN & SAN, differing in "X509v3 Extended Key Usage".
>
> Both are verified "OK" with my local CA cert
>
> openssl verify -CAfile myCA.crt.pem test.example.com.server.crt
> test.example.com.server.crt: OK
>
> openssl verify -CAfile /myCA.crt.pem test.example.com.client.crt
> test.example.com.client.crt: OK
>
> My main nginx config includes,
>
> upstream test.example.com {
> server test.example.com:11111;
> }
> server {
>
> listen 10.10.10.1:443 ssl http2;
> server_name example.com;
> ...
>
> ssl_verify_client on;
> ssl_client_certificate  "/etc/ssl/nginx/myCA.crt";
> ssl_verify_depth 2;
> ssl_certificate         "/etc/ssl/nginx/example.com.server.crt";
> ssl_certificate_key     "/etc/ssl/nginx/example.com.server.key";
> ssl_trusted_certificate "/etc/ssl/nginx/myCA.crt";
>
> location /app1 {
> proxy_pass                    https://test.example.com;
> proxy_ssl_certificate         "/etc/ssl/nginx/test.example.com.client.crt";
> proxy_ssl_certificate_key     "/etc/ssl/nginx/test.example.com.client.key";
> proxy_ssl_trusted_certificate "/etc/ssl/nginx/myCA.crt";
> proxy_ssl_verify       on;
> proxy_ssl_verify_depth 2;
> include includes/reverse-proxy.inc;
> }
> }
>
> and the upstream config,
>
> server {
> listen 127.0.0.1:11111 ssl http2;
> server_name test.example.com;
>
> root /data/webapps/demo_app/;
> index index.php;
> expires -1;
>
> ssl_certificate        "/etc/ssl/nginx/test.example.com.server.crt";
> ssl_certificate_key    "/etc/ssl/nginx/test.example.com.server.key";
>
> ssl_client_certificate "/etc/ssl/nginx/myCA.crt";
> ssl_verify_client optional;
> ssl_verify_depth 2;
>
> location ~ \.php {
> try_files $uri =404;
> fastcgi_pass   phpfpm;
> fastcgi_index  index.php;
> fastcgi_param  PATH_INFO $fastcgi_script_name;
> include        fastcgi_params;
> }
>
> }
>
> access to
>
> https://example.com/app1
>
> responds,
>
> 502 Bad Gateway
>
> logs, show an SSL handshake fail
>
> ...
> 2020/05/29 19:00:06 [debug] 29419#29419: *7 SSL: TLSv1.3, cipher: "TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any Au=any Enc=CHACHA20/POLY1305(256) Mac=AEAD"
> 2020/05/29 19:00:06 [debug] 29419#29419: *7 http upstream ssl handshake: "/app1/?"
> 2020/05/29 19:00:06 [debug] 29419#29419: *7 X509_check_host(): no match
> 2020/05/29 19:00:06 [error] 29419#29419: *7 upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream, client: 10.10.10.73, server: example.com, request: "GET /app1/ HTTP/2.0", upstream: "https://127.0.0.1:11111/app1/", host: "example.com"
> 2020/05/29 19:00:06 [debug] 29419#29419: *7 http next upstream, 2
> ...
>
> If I toggle
>
> - ssl_verify_client on;
> + ssl_verify_client off;
>
> then I'm able to connect to the backend site, as expected.
>
> What exactly is NOT matching in the handshake?  CN & SAN do ...
>
> &/or, is there a config problem above?

Most likely the problem is that the certificate returned depends
on the name provided via Server Name Indication (SNI).  That is,
that the server block in the upstream server configuration is not
the default one, and the default one returns a different
certificate.

Usage of Server Name Indication for upstream SSL connections isn't
enabled by default, and this isn't switched on in your
configuration.  Try

    proxy_ssl_server_name on;

to see if it helps.  See http://nginx.org/r/proxy_ssl_server_name 
for details.

You may also try the following patch to provide somewhat better
debug logging when checking upstream server SSL certificates:

# HG changeset patch
# User Maxim Dounin <[hidden email]>
# Date 1591025575 -10800
#      Mon Jun 01 18:32:55 2020 +0300
# Node ID eaa39944438dbb10507760890bddc45c19a5ad6f
# Parent  8cadaf7e7231865f2f81c03cb785c045dda6bf8b
SSL: added verify callback to ngx_ssl_trusted_certificate().

This ensures that certificate verification is properly logged to debug
log during upstream server certificate verification.  This should help
with debugging various certificate issues.

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -920,6 +920,8 @@ ngx_int_t
 ngx_ssl_trusted_certificate(ngx_conf_t *cf, ngx_ssl_t *ssl, ngx_str_t *cert,
     ngx_int_t depth)
 {
+    SSL_CTX_set_verify(ssl->ctx, SSL_VERIFY_PEER, ngx_ssl_verify_callback);
+
     SSL_CTX_set_verify_depth(ssl->ctx, depth);
 
     if (cert->len == 0) {



--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

PGNet Dev
On 6/1/20 8:42 AM, Maxim Dounin wrote:
>
>      proxy_ssl_server_name on;
>
> to see if it helps.  See http://nginx.org/r/proxy_ssl_server_name
> for details.

enabling it _has_ an effect.

now,

access to



        https://example.com/app1



responds,



- 502 Bad Gateway
+ 421 Misdirected Request

>
> You may also try the following patch to provide somewhat better
> debug logging when checking upstream server SSL certificates:

I'll get this in place & see what i learn ...
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

PGNet Dev
with patch applied, and 'proxy_ssl_server_name on;'

this is where the problem appears

        2020/06/02 00:50:08 [debug] 20166#20166: *3 verify:1, error:0, depth:2, subject:"/O=example.com/OU=example.com_CA/L=New_York/ST=NY/C=US/emailAddress=[hidden email]/CN=example.com_CA", issuer:"/O=example.com/OU=example.com_CA/L=New_York/ST=NY/C=US/emailAddress=[hidden email]/CN=example.com_CA"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 verify:1, error:0, depth:1, subject:"/C=US/ST=NY/O=example.com/OU=example.com_CA/CN=example.com_CA_INTERMEDIATE/emailAddress=[hidden email]", issuer:"/O=example.com/OU=example.com_CA/L=New_York/ST=NY/C=US/emailAddress=[hidden email]/CN=example.com_CA"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 verify:1, error:0, depth:0, subject:"/C=US/ST=NY/L=New_York/O=example.com/OU=example.com_CA/CN=test.example.net/emailAddress=[hidden email]", issuer:"/C=US/ST=NY/O=example.com/OU=example.com_CA/CN=example.com_CA_INTERMEDIATE/emailAddress=[hidden email]"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 ssl new session: 0E2A0672:32:1105
        2020/06/02 00:50:08 [debug] 20166#20166: *3 ssl new session: 31C878D7:32:1104
        2020/06/02 00:50:08 [debug] 20166#20166: *3 SSL_do_handshake: 1
        2020/06/02 00:50:08 [debug] 20166#20166: *3 SSL: TLSv1.3, cipher: "TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any Au=any Enc=CHACHA20/POLY1305(256) Mac=AEAD"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 reusable connection: 1
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http wait request handler
        2020/06/02 00:50:08 [debug] 20166#20166: *3 malloc: 0000555967A0B2E0:1024
        2020/06/02 00:50:08 [debug] 20166#20166: *3 SSL_read: 772
        2020/06/02 00:50:08 [debug] 20166#20166: *3 SSL_read: -1
        2020/06/02 00:50:08 [debug] 20166#20166: *3 SSL_get_error: 2
        2020/06/02 00:50:08 [debug] 20166#20166: *3 reusable connection: 0
        2020/06/02 00:50:08 [debug] 20166#20166: *3 posix_memalign: 00005559678F6460:4096 @16
        2020/06/02 00:50:08 [debug] 20166#20166: *3 posix_memalign: 00005559675113A0:4096 @16
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http process request line
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http request line: "GET /app1 HTTP/1.1"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http uri: "/app1"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http args: ""
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http exten: ""
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http process request header line
        2020/06/02 00:50:08 [info] 20166#20166: *3 client attempted to request the server name different from the one that was negotiated while reading client request headers, client: 127.0.0.1, server: test.example.net, request: "GET /app1 HTTP/1.1", host: "example.net"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http finalize request: 421, "/app1?" a:1, c:1
        2020/06/02 00:50:08 [debug] 20166#20166: *3 event timer del: 50: 3334703
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http special response: 421, "/app1?"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 http set discard body
        2020/06/02 00:50:08 [debug] 20166#20166: *3 headers more header filter, uri "/app1"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 lua capture header filter, uri "/app1"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 xslt filter header
        2020/06/02 00:50:08 [debug] 20166#20166: *3 charset: "" > "utf-8"
        2020/06/02 00:50:08 [debug] 20166#20166: *3 HTTP/1.1 421 Misdirected Request

noting

        2020/06/02 00:50:08 [info] 20166#20166: *3 client attempted to request the server name different from the one that was negotiated while reading client request headers, client: 127.0.0.1, server: test.example.net, request: "GET /app1 HTTP/1.1", host: "example.net"

now, need to stare at this and try to figure out 'why?'
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Sergey Kandaurov

> On 2 Jun 2020, at 07:58, PGNet Dev <[hidden email]> wrote:
>
> 2020/06/02 00:50:08 [info] 20166#20166: *3 client attempted to request the server name different from the one that was negotiated while reading client request headers, client: 127.0.0.1, server: test.example.net, request: "GET /app1 HTTP/1.1", host: "example.net"
>
> now, need to stare at this and try to figure out 'why?'

That means client provided TLS "server_name" extension (SNI),
then requested a different origin in the Host header.

In your case, the mangled name "test.example.net" (via SNI)
didn't match another mangled name "example.net" (in Host).

For the formal specification, see the last paragraph in RFC 6066, section-3:

   If an application negotiates a server name using an application
   protocol and then upgrades to TLS, and if a server_name extension is
   sent, then the extension SHOULD contain the same name that was
   negotiated in the application protocol.  If the server_name is
   established in the TLS session handshake, the client SHOULD NOT
   attempt to request a different server name at the application layer.

421 is defined for such cases in HTTP.

--
Sergey Kandaurov

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Francis Daly
On Tue, Jun 02, 2020 at 12:51:55PM +0300, Sergey Kandaurov wrote:

Hi there,

> That means client provided TLS "server_name" extension (SNI),
> then requested a different origin in the Host header.

That suggests that if you choose to use "proxy_ssl_server_name on;",
then you almost certainly do not want to add your own "proxy_set_header
Host" value.

The nginx code probably should not try to check for (and reject) that
combination of directives-and-values; but might it be worth adding a
note to http://nginx.org/r/proxy_ssl_server_name to say that that other
directive is probably a bad idea, especially if you get a http 421 response
from your upstream?

Cheers,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

PGNet Dev
On 6/2/20 8:27 AM, Francis Daly wrote:
> That suggests that if you choose to use "proxy_ssl_server_name on;",
> then you almost certainly do not want to add your own "proxy_set_header
> Host" value.
>
> The nginx code probably should not try to check for (and reject) that
> combination of directives-and-values; but might it be worth adding a
> note to http://nginx.org/r/proxy_ssl_server_name to say that that other
> directive is probably a bad idea, especially if you get a http 421 response
> from your upstream?

trying to simplify/repeat, i've

vhost config,

        upstream test-upstream {
                server test.example.com:11111;
        }

        server {
                listen 10.10.10.1:443 ssl http2;
                server_name example.com;

                ...
                location /app1 {

                        proxy_ssl_verify       on;
                        proxy_ssl_verify_depth 2;
                        proxy_ssl_certificate         "/etc/ssl/nginx/test.client.crt";
                        proxy_ssl_certificate_key     "/etc/ssl/nginx/test.client.key";
                        proxy_ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";

                        proxy_pass https://test-upstream/;
                        proxy_ssl_server_name on;
                        proxy_ssl_name test.example.com;

                }
        }

and, upstream config

                server {
                        listen 127.0.0.1:11111 ssl http2;
                        server_name test.example.com;

                        root /srv/www/test;
                        index index.php;
                        expires -1;

                        ssl_certificate         "/etc/ssl/nginx/test.server.crt";
                        ssl_certificate_key     "/etc/ssl/nginx/test.server.key";
                        ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";

                        ssl_verify_client off;
                        ssl_verify_depth 2;
                        ssl_client_certificate  "/etc/ssl/nginx/ca_int.crt";

                        location ~ \.php {
                                try_files $uri =404;
                                fastcgi_pass   phpfpm;
                                fastcgi_index  index.php;
                                fastcgi_param  PATH_INFO $fastcgi_script_name;
                                include        includes/fastcgi/fastcgi_params;
                        }

                        error_log   /var/log/nginx/test.error.log  info;
                }

on access to

        https://example.com/app1

still get

        421 Misdirected Request

in log

        ==> /var/log/nginx/test.error.log <==
        2020/06/02 11:52:13 [info] 8713#8713: *18 client attempted to request the server name different from the one that was negotiated while reading client request headers, client: 127.0.0.1, server: test.example.com, request: "GET / HTTP/1.0", host: "test-upstream"

Is that

        host: "test-upstream"

to be expected?  it's an upstream name, not an actual host.

Still unable to wrap my head around where this mis-match is coming from ... I have a nagging suspicion I'm missing something *really* obvious :-/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Maxim Dounin
In reply to this post by Francis Daly
Hello!

On Tue, Jun 02, 2020 at 04:27:28PM +0100, Francis Daly wrote:

> On Tue, Jun 02, 2020 at 12:51:55PM +0300, Sergey Kandaurov wrote:
>
> Hi there,
>
> > That means client provided TLS "server_name" extension (SNI),
> > then requested a different origin in the Host header.
>
> That suggests that if you choose to use "proxy_ssl_server_name on;",
> then you almost certainly do not want to add your own "proxy_set_header
> Host" value.
>
> The nginx code probably should not try to check for (and reject) that
> combination of directives-and-values; but might it be worth adding a
> note to http://nginx.org/r/proxy_ssl_server_name to say that that other
> directive is probably a bad idea, especially if you get a http 421 response
> from your upstream?

Not exactly.

The 421 Misdirected Request error is only returned
when one tries to access a virtual server with SSL client
certificate verification enabled, and used a different server name
during the SSL handshake.  Normally one can use Host header which
is different from the SNI server name, and this is often happens
in real life (e.g., connection reuse in HTTP/2 implies requests to
multiple hostnames via one connection).

That's more about being careful when configuring things,
especially when configuring SSL.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Maxim Dounin
In reply to this post by PGNet Dev
Hello!

On Tue, Jun 02, 2020 at 12:10:45PM -0700, PGNet Dev wrote:

> On 6/2/20 8:27 AM, Francis Daly wrote:
> > That suggests that if you choose to use "proxy_ssl_server_name on;",
> > then you almost certainly do not want to add your own "proxy_set_header
> > Host" value.
> >
> > The nginx code probably should not try to check for (and reject) that
> > combination of directives-and-values; but might it be worth adding a
> > note to http://nginx.org/r/proxy_ssl_server_name to say that that other
> > directive is probably a bad idea, especially if you get a http 421 response
> > from your upstream?
>
> trying to simplify/repeat, i've
>
> vhost config,
>
> upstream test-upstream {
> server test.example.com:11111;
> }
>
> server {
> listen 10.10.10.1:443 ssl http2;
> server_name example.com;
>
> ...
> location /app1 {
>
> proxy_ssl_verify       on;
> proxy_ssl_verify_depth 2;
> proxy_ssl_certificate         "/etc/ssl/nginx/test.client.crt";
> proxy_ssl_certificate_key     "/etc/ssl/nginx/test.client.key";
> proxy_ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";
>
> proxy_pass https://test-upstream/;
> proxy_ssl_server_name on;
> proxy_ssl_name test.example.com;
>
> }
> }
>
> and, upstream config
>
> server {
> listen 127.0.0.1:11111 ssl http2;
> server_name test.example.com;
>
> root /srv/www/test;
> index index.php;
> expires -1;
>
> ssl_certificate         "/etc/ssl/nginx/test.server.crt";
> ssl_certificate_key     "/etc/ssl/nginx/test.server.key";
> ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";
>
> ssl_verify_client off;
> ssl_verify_depth 2;
> ssl_client_certificate  "/etc/ssl/nginx/ca_int.crt";
>
> location ~ \.php {
> try_files $uri =404;
> fastcgi_pass   phpfpm;
> fastcgi_index  index.php;
> fastcgi_param  PATH_INFO $fastcgi_script_name;
> include        includes/fastcgi/fastcgi_params;
> }
>
> error_log   /var/log/nginx/test.error.log  info;
> }
>
> on access to
>
> https://example.com/app1
>
> still get
>
> 421 Misdirected Request
>
> in log
>
> ==> /var/log/nginx/test.error.log <==
> 2020/06/02 11:52:13 [info] 8713#8713: *18 client attempted to request the server name different from the one that was negotiated while reading client request headers, client: 127.0.0.1, server: test.example.com, request: "GET / HTTP/1.0", host: "test-upstream"
>
> Is that
>
> host: "test-upstream"
>
> to be expected?  it's an upstream name, not an actual host.

Yes, it is expected.  Quoting http://nginx.org/r/proxy_set_header:

: By default, only two fields are redefined:
:
: proxy_set_header Host       $proxy_host;
: proxy_set_header Connection close;

That is, the name you've written in the proxy_pass directive is
the actual hostname, and it will be used in the Host header when
creating requests to upstream server.  And it is also used in the
proxy_ssl_name, so it will be used during SSL handshake for SNI
and certificate verification.

It's not just "an upstream name".  If you want it to be only an
upstream name, you'll have to redefine at least proxy_ssl_name and
"proxy_set_header Host".  (Well, not really, since $proxy_host is
also used at least in the proxy_cache_key, but this is probably
not that important.)

Alternatively, you may want to use the real name, and define an
upstream{} block with that name.  This way you won't need to
redefine anything.

> Still unable to wrap my head around where this mis-match is
> coming from ... I have a nagging suspicion I'm missing something
> *really* obvious :-/

The mis-match comes from trying to redefine the name in some parts
of the configuration but not others.  Hope the above explanation
helps.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

PGNet Dev
On 6/2/20 12:34 PM, Maxim Dounin wrote:
> The mis-match comes from trying to redefine the name in some parts
> of the configuration but not others.  Hope the above explanation
> helps.

I've reread your comment

        That is, the name you've written in the proxy_pass directive is
        the actual hostname, and it will be used in the Host header when
        creating requests to upstream server.  And it is also used in the
        proxy_ssl_name, so it will be used during SSL handshake for SNI
        and certificate verification.

        It's not just "an upstream name".  If you want it to be only an
        upstream name, you'll have to redefine at least proxy_ssl_name and
        "proxy_set_header Host".  (Well, not really, since $proxy_host is
        also used at least in the proxy_cache_key, but this is probably
        not that important.)

a bunch of times.  Still can't grasp it clearly.  Which is the source of the pebkac :-/

Otoh, simply _doing_

        Alternatively, you may want to use the real name, and define an
        upstream{} block with that name.  This way you won't need to
        redefine anything.

i.e., changing to EITHER

case (1):

        vhost config,

- upstream test-upstream {
+ upstream test.example.com {
                        server test.example.com:11111;
                }

                server {
                        listen 10.10.10.1:443 ssl http2;
                        server_name example.com;

                        ...
                        location /app1 {

                                proxy_ssl_verify       on;
                                proxy_ssl_verify_depth 2;
                                proxy_ssl_certificate         "/etc/ssl/nginx/test.client.crt";
                                proxy_ssl_certificate_key     "/etc/ssl/nginx/test.client.key";
                                proxy_ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";

- proxy_pass https://test-upstream/;
+ proxy_pass https://test.example.com/;
                                proxy_ssl_server_name on;
                                proxy_ssl_name test.example.com;

                        }
                }

        and, upstream config

                        server {
                                listen 127.0.0.1:11111 ssl http2;
                                server_name test.example.com;

                                root /srv/www/test;
                                index index.php;
                                expires -1;

                                ssl_certificate         "/etc/ssl/nginx/test.server.crt";
                                ssl_certificate_key     "/etc/ssl/nginx/test.server.key";
                                ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";

- ssl_verify_client off;
+ ssl_verify_client on;
                                ssl_verify_depth 2;
                                ssl_client_certificate  "/etc/ssl/nginx/ca_int.crt";

                                location ~ \.php {
                                        try_files $uri =404;
                                        fastcgi_pass   phpfpm;
                                        fastcgi_index  index.php;
                                        fastcgi_param  PATH_INFO $fastcgi_script_name;
                                        include        includes/fastcgi/fastcgi_params;
                                }

                                error_log   /var/log/nginx/test.error.log  info;
                        }

or

case (2):

        vhost config,

- upstream test-upstream {
+ upstream JUNK {
                        server test.example.com:11111;
                }

                server {
                        listen 10.10.10.1:443 ssl http2;
                        server_name example.com;

                        ...
                        location /app1 {

                                proxy_ssl_verify       on;
                                proxy_ssl_verify_depth 2;
                                proxy_ssl_certificate         "/etc/ssl/nginx/test.client.crt";
                                proxy_ssl_certificate_key     "/etc/ssl/nginx/test.client.key";
                                proxy_ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";

- proxy_pass https://test-upstream/;
+ proxy_pass https://test.example.com:11111/;
                                proxy_ssl_server_name on;
                                proxy_ssl_name test.example.com;

                        }
                }

        and, upstream config

                        server {
                                listen 127.0.0.1:11111 ssl http2;
                                server_name test.example.com;

                                root /srv/www/test;
                                index index.php;
                                expires -1;

                                ssl_certificate         "/etc/ssl/nginx/test.server.crt";
                                ssl_certificate_key     "/etc/ssl/nginx/test.server.key";
                                ssl_trusted_certificate "/etc/ssl/nginx/ca_int.crt";

- ssl_verify_client off;
+ ssl_verify_client on;
                                ssl_verify_depth 2;
                                ssl_client_certificate  "/etc/ssl/nginx/ca_int.crt";

                                location ~ \.php {
                                        try_files $uri =404;
                                        fastcgi_pass   phpfpm;
                                        fastcgi_index  index.php;
                                        fastcgi_param  PATH_INFO $fastcgi_script_name;
                                        include        includes/fastcgi/fastcgi_params;
                                }

                                error_log   /var/log/nginx/test.error.log  info;
                        }

now, in _either_ case, access to

        https://example.com/app1
        https://example.com/app1/

_does_ return my 'test' app correctly

i _do_ see in logs

in case (2), a single error instance,

                2020/06/02 12:51:11 [debug] 6140#6140: *3 reusable connection: 1
                2020/06/02 12:51:11 [debug] 6140#6140: *3 http wait request handler
                2020/06/02 12:51:11 [debug] 6140#6140: *3 malloc: 0000563CDA76DF10:1024
                2020/06/02 12:51:11 [debug] 6140#6140: *3 SSL_read: 345
                2020/06/02 12:51:11 [debug] 6140#6140: *3 SSL_read: -1
??? 2020/06/02 12:51:11 [debug] 6140#6140: *3 SSL_get_error: 2
                2020/06/02 12:51:11 [debug] 6140#6140: *3 reusable connection: 0
                2020/06/02 12:51:11 [debug] 6140#6140: *3 posix_memalign: 0000563CDA2963A0:4096 @16
                2020/06/02 12:51:11 [debug] 6140#6140: *3 posix_memalign: 0000563CDA650060:4096 @16

&

in case (1), a double error instance

                2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read_early_data: 2, 0
                2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_do_handshake: 1
                2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL: TLSv1.2, cipher: "ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD"
                2020/06/02 12:53:46 [debug] 6267#6267: *6 reusable connection: 1
                2020/06/02 12:53:46 [debug] 6267#6267: *6 http wait request handler
                2020/06/02 12:53:46 [debug] 6267#6267: *6 malloc: 0000563C0F2ADAB0:1024
                2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read: -1
??? 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_get_error: 2
                2020/06/02 12:53:46 [debug] 6267#6267: *6 free: 0000563C0F2ADAB0
                2020/06/02 12:53:46 [debug] 6267#6267: *6 http wait request handler
                2020/06/02 12:53:46 [debug] 6267#6267: *6 malloc: 0000563C0F2ADAB0:1024
                2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read: 339
                2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read: -1
??? 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_get_error: 2
                2020/06/02 12:53:46 [debug] 6267#6267: *6 reusable connection: 0
                2020/06/02 12:53:46 [debug] 6267#6267: *6 posix_memalign: 0000563C0F18FA60:4096 @16
                2020/06/02 12:53:46 [debug] 6267#6267: *6 posix_memalign: 0000563C0EDD4B10:4096 @16
                2020/06/02 12:53:46 [debug] 6267#6267: *6 http process request line


but that error doesn't seem to be fatal.

any idea what's causing those^^ errors?


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Maxim Dounin
Hello!

On Tue, Jun 02, 2020 at 01:01:18PM -0700, PGNet Dev wrote:

> On 6/2/20 12:34 PM, Maxim Dounin wrote:
> > The mis-match comes from trying to redefine the name in some parts
> > of the configuration but not others.  Hope the above explanation
> > helps.
>
> I've reread your comment
>
> That is, the name you've written in the proxy_pass directive is
> the actual hostname, and it will be used in the Host header when
> creating requests to upstream server.  And it is also used in the
> proxy_ssl_name, so it will be used during SSL handshake for SNI
> and certificate verification.
>
> It's not just "an upstream name".  If you want it to be only an
> upstream name, you'll have to redefine at least proxy_ssl_name and
> "proxy_set_header Host".  (Well, not really, since $proxy_host is
> also used at least in the proxy_cache_key, but this is probably
> not that important.)
>
> a bunch of times.  Still can't grasp it clearly.  Which is the source of the pebkac :-/

Read: if you want to use an internal upstream name in proxy_pass,
consider using _both_ "proxy_ssl_name" and "proxy_set_header
Host", for example:

    proxy_pass https://test-upstream;
    proxy_set_header Host test.example.com;
    proxy_ssl_name test.example.com;

There are few other places where the hostname from the proxy_pass
directive is used, but the probably aren't that important.

> Otoh, simply _doing_
>
> Alternatively, you may want to use the real name, and define an
> upstream{} block with that name.  This way you won't need to
> redefine anything.
>
> i.e., changing to EITHER

[...]

> now, in _either_ case, access to
>
> https://example.com/app1
> https://example.com/app1/
>
> _does_ return my 'test' app correctly

So everything is fine, as expected.

> i _do_ see in logs
>
> in case (2), a single error instance,
>
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 reusable connection: 1
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 http wait request handler
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 malloc: 0000563CDA76DF10:1024
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 SSL_read: 345
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 SSL_read: -1
> ??? 2020/06/02 12:51:11 [debug] 6140#6140: *3 SSL_get_error: 2
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 reusable connection: 0
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 posix_memalign: 0000563CDA2963A0:4096 @16
> 2020/06/02 12:51:11 [debug] 6140#6140: *3 posix_memalign: 0000563CDA650060:4096 @16
>
> &
>
> in case (1), a double error instance
>
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read_early_data: 2, 0
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_do_handshake: 1
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL: TLSv1.2, cipher: "ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD"
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 reusable connection: 1
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 http wait request handler
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 malloc: 0000563C0F2ADAB0:1024
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read: -1
> ??? 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_get_error: 2
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 free: 0000563C0F2ADAB0
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 http wait request handler
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 malloc: 0000563C0F2ADAB0:1024
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read: 339
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_read: -1
> ??? 2020/06/02 12:53:46 [debug] 6267#6267: *6 SSL_get_error: 2
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 reusable connection: 0
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 posix_memalign: 0000563C0F18FA60:4096 @16
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 posix_memalign: 0000563C0EDD4B10:4096 @16
> 2020/06/02 12:53:46 [debug] 6267#6267: *6 http process request line
>
>
> but that error doesn't seem to be fatal.
>
> any idea what's causing those^^ errors?

These aren't errors, these are debug messages.  The
SSL_get_error() return code 2 means SSL_ERROR_WANT_READ, that is,
SSL_read() consumed all the data from the socket and needs more
data to read further.  These messages are perfectly normal and
expected.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: proxy_ssl_verify error: 'upstream SSL certificate does not match "test.example.com" while SSL handshaking to upstream', for CN/SAN 'matched' client & server certs ?

Francis Daly
In reply to this post by Maxim Dounin
On Tue, Jun 02, 2020 at 10:22:06PM +0300, Maxim Dounin wrote:
> On Tue, Jun 02, 2020 at 04:27:28PM +0100, Francis Daly wrote:

Hi there,

Thanks for the extra information.

> > That suggests that if you choose to use "proxy_ssl_server_name on;",
> > then you almost certainly do not want to add your own "proxy_set_header
> > Host" value.

> Not exactly.
>
> The 421 Misdirected Request error is only returned
> when one tries to access a virtual server with SSL client
> certificate verification enabled, and used a different server name
> during the SSL handshake.

Is the "client certificate verification" part important there, in the
general case? The upstream server could be anything, so could be more
picky about matching SNI name and Host header, I guess.


So based on the other mails in the thread, I'll update my own notes to
say: if you use "proxy_ssl_server_name on;", then probably make sure
that "proxy_set_header Host" and "proxy_ssl_name" have the same value;
by default they do, but if you change only one there may be problems.

> Normally one can use Host header which
> is different from the SNI server name, and this is often happens
> in real life (e.g., connection reuse in HTTP/2 implies requests to
> multiple hostnames via one connection).

True; web searches do reveal some people reporting 421 error in normal
use cases, but they seem mainly down to badly configured servers.

> That's more about being careful when configuring things,
> especially when configuring SSL.

Agreed.

Since the cases where special consideration is needed are not trivial
to enumerate, adding a note for only one of them is not the most useful
thing to do.

Cheers,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx