Symptom:
We hit various SSL handshake, connection, verification errors when we config upstream SSL in nginx following the guide
*1 upstream SSL certificate does not match "loginapp" while SSL handshaking to upstream, client: 110.126.174.93, server: test.com, request: "GET /auth/ HTTP/1.1", upstream: "https://198.19.163.33:5555/", host: "platform.nw-dev.kubernetes.cba", referrer: "https://platform.nw-dev.kubernetes.cba/"
2021/12/10 01:06:27 [error] 7#7: *1 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 110.126.174.93, server: test.com, request: "GET /auth/ HTTP/1.1", upstream: "https://198.19.163.33:5555/", host: "test.com", ref
Diagnose:
It turns out the upstream in nginx config has the same DOMAIN requirement of HTTPS. For example:upstream myapp{
server myapp.svc.test.com:556;
}
location /myapp{
proxy_pass https://myapp;
proxy_ssl_trusted_certificate /etc/nginx/rootca/ClusterCA.pem;
proxy_ssl_verify on;
proxy_ssl_session_reuse on;
}
The TLS certificate from the myapp.svc.test.com would need to have COMMON or DNS alias myapp in it, otherwise, the SSL handshake will fail. In other words, the upstream "myapp" needs to match the the TLS certificate CN in the myapp.svc.test.com. The tricky part is that the CN in the TLS certficate needs to match upstream, not the "myapp.svc.test.com" ie below will work as long as TLS in random.example.com has "myapp1.svc.test.com" in it.
upstream myapp1.svc.test.com {
server random.example.com:556;
}
To add another layer of security, to prevent man-in-the middle attack which host fake HTTPS for hacking purpose, we can set certain CA we trust for the upsteam server
Set proxy_ssl_verify is on, the upsteam TLS certificate needs to be signed by the certain CA nginx can trust which is configured at proxy_ssl_trusted_certificate, otherwise, SSL verifies will fail.
To further harden security, we can add mTLS,example like below refer what is mTLS here
location /myapp{
proxy_pass https://backend.example.com;
proxy_ssl_certificate /etc/nginx/client.pem;
proxy_ssl_certificate_key /etc/nginx/client.key;
proxy_ssl_trusted_certificate /etc/nginx/trusted_ca_cert.crt;
proxy_ssl_verify on;
proxy_ssl_session_reuse on;
}
Refer more detailed info via nginx bug doc