I have a project that involves mutual / two way TLS and one of
the requirements is that the TLS handshake must fail ie. be
terminated before completion if the handshake is in anyway
unsuccessful, eg. no client certificate provided or client
certificate not trusted.
After having no success getting nginx (v1.16.1) & openssl
(v1.0.2k-fips) to fail the handshake I ended up looking at the
nginx source code, in particular src/event/ngx_event_openssl.c,
and from what I read here https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_verify.html
I think a small but necessary code change is required.
Some possible approaches when choosing to remain using nginx as
the server end of the mutual TLS connection
in static int ngx_ssl_verify_callback(int ok,
X509_STORE_CTX *x509_store) make it configurable whether
1 is always returned or the value of ok
in ngx_int_t ngx_ssl_client_certificate(ngx_conf_t *cf,
ngx_ssl_t *ssl, ngx_str_t *cert, ngx_int_t depth) make
it configurable whether SSL_CTX_set_verify(ssl->ctx,
SSL_VERIFY_PEER, ngx_ssl_verify_callback); is called or
SSL_CTX_set_verify(ssl->ctx, SSL_VERIFY_PEER, NULL);
Is a code change required or is there a way for the handshake
failure to be 'enabled' as opposed to ending up with a
successfully established TLS connection. Admittedly within nginx
there's all the detail that the TLS connection doesn't conform
to the configured requirements of the TLS connection but this
doesn't satisfy the requirements for the project.
I won't bother going in to the details of the project but will
just say it's a third party certification body that requires the
TLS handshake to be terminated before completion if the
handshake is in anyway unsuccessful.