Connection timeout on SSL with shared hosting

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Connection timeout on SSL with shared hosting

zakirenish
Hi All,
Newbie question. I posted this on Stack Overflow but haven't gotten any
replies yet.
https://stackoverflow.com/questions/63391424/why-do-i-get-connection-timeout-on-ssl-even-though-nginx-is-listening-and-firewa

Most/many visitors to my site https://example.org get a connection timeout.
Some visitors get through, possibly ones redirected from http://example.org
or those who've previously visited the site.

I'm trying to determine if this is a firewall issue or an nginx
configuration issue.

Firewall

I'm using UFW as a firewall, which has the following rules:

To                         Action      From
--                         ------      ----
SSH                        ALLOW       Anywhere                  
Nginx Full                 ALLOW       Anywhere                  
80/tcp                     ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere                  
SSH (v6)                   ALLOW       Anywhere (v6)            
Nginx Full (v6)            ALLOW       Anywhere (v6)            
80/tcp (v6)                ALLOW       Anywhere (v6)            
443/tcp (v6)               ALLOW       Anywhere (v6)

I could give some relevant rules from iptables if anyone needs that, but I'd
need some direction on what to look for.

For sudo netstat -anop | grep LISTEN | grep ':443' I get

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN  
   120907/nginx: worke  off (0.00/0/0)
tcp6       0      0 :::443                  :::*                    LISTEN  
   120907/nginx: worke  off (0.00/0/0)

Not sure what "worke off" means.

nginx

It's a virtual host with the server name myservername.com which serves up
two websites, example.org and example.com/directory. Example.org points to a
docker container running eXist-db. Example.com/directory is serving up a
directory on localhost:8080 proxied from another server where example.com
lives. Example.com/directory is running smoothly on https when I access it
in the browser -- I presume this is because it actually talks to the
example.com host over http.

Example.org and myservername.com both have certs from let's encrypt
generated by certbot.

When I try nmap from my local machine I get some results I can't explain.
Notice the discrepancy between ports 80 and ports 443 and between IPv4 and
IPv6

$ nmap -A -T4 -p443 example.org
443/tcp filtered https

$ nmap -A -T4 -p443 my.server.ip.address
443/tcp filtered https

$ nmap -A -T4 -p443 -6 my:server:ip::v6:address
443/tcp open  ssl/http nginx 1.10.3

$ nmap -A -T4 -p80 example.org
80/tcp open  http    nginx 1.10.3

$ nmap -A -T4 -p80 my.server.ip.address
80/tcp open  http    nginx 1.10.3

My nginx.conf is

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        client_max_body_size 50M;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json
application/javascript text/xml application/xml application/xml+rss
text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

and my nginx server blocks:

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name _ myservername.com;
        return 301 https://myservername.com$request_uri;
}

server {
        # SSL configuration
        #
        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;
       
        server_name _ myservername.com;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_pass <a href="http://localhost:8080;">http://localhost:8080;
       }

        ssl_certificate
/etc/letsencrypt/live/myservername.com/fullchain.pem;
        ssl_certificate_key
/etc/letsencrypt/live/myservername.com/privkey.pem;
}

server {
        listen 80;
        listen [::]:80;

        server_name example.com www.example.com;

        gzip off;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass <a href="http://localhost:8080;">http://localhost:8080;
        }
}

server {
       listen 80;
       listen [::]:80;

       server_name example.org www.example.org;
       return 301 https://example.org$request_uri;
}

server {

        # SSL configuration
        #
        listen 443 ssl;
        listen [::]:443 ssl;
       
        server_name example.org www.example.org;

        gzip off;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass
<a href="http://docker.container.ip.address:port/exist/apps/example/;">http://docker.container.ip.address:port/exist/apps/example/;
        }

        location /workshop2020/ {
                return 302 http://example.org/forum2020/;
        }


    location /exist/apps/example/ {
            rewrite ^/exist/apps/example/(.*)$ /$1;
    }


    ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem; #
managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem; #
managed by Certbot

}

Very grateful for any help!!
Nathan

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289099#msg-289099

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

Thomas Ward

You said this is "shared hosting" - when you say "shared hosting" do you mean this is *not* a dedicated machine but one machine out of many in a shared environment?

Have you tested briefly by disabling your firewall just to see if that fixes the issue?

What is the backend?  You're passing everything to 8080 which suggests the backend might be having issues too.


Thomas


On 8/13/20 3:04 PM, nathanpgibson wrote:
Hi All, 
Newbie question. I posted this on Stack Overflow but haven't gotten any
replies yet.
https://stackoverflow.com/questions/63391424/why-do-i-get-connection-timeout-on-ssl-even-though-nginx-is-listening-and-firewa

Most/many visitors to my site https://example.org get a connection timeout.
Some visitors get through, possibly ones redirected from http://example.org
or those who've previously visited the site.

I'm trying to determine if this is a firewall issue or an nginx
configuration issue.

Firewall

I'm using UFW as a firewall, which has the following rules:

To                         Action      From
--                         ------      ----
SSH                        ALLOW       Anywhere                  
Nginx Full                 ALLOW       Anywhere                  
80/tcp                     ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere                  
SSH (v6)                   ALLOW       Anywhere (v6)             
Nginx Full (v6)            ALLOW       Anywhere (v6)             
80/tcp (v6)                ALLOW       Anywhere (v6)             
443/tcp (v6)               ALLOW       Anywhere (v6) 

I could give some relevant rules from iptables if anyone needs that, but I'd
need some direction on what to look for.

For sudo netstat -anop | grep LISTEN | grep ':443' I get

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN  
   120907/nginx: worke  off (0.00/0/0)
tcp6       0      0 :::443                  :::*                    LISTEN  
   120907/nginx: worke  off (0.00/0/0)

Not sure what "worke off" means.

nginx

It's a virtual host with the server name myservername.com which serves up
two websites, example.org and example.com/directory. Example.org points to a
docker container running eXist-db. Example.com/directory is serving up a
directory on localhost:8080 proxied from another server where example.com
lives. Example.com/directory is running smoothly on https when I access it
in the browser -- I presume this is because it actually talks to the
example.com host over http.

Example.org and myservername.com both have certs from let's encrypt
generated by certbot.

When I try nmap from my local machine I get some results I can't explain.
Notice the discrepancy between ports 80 and ports 443 and between IPv4 and
IPv6

$ nmap -A -T4 -p443 example.org
443/tcp filtered https

$ nmap -A -T4 -p443 my.server.ip.address
443/tcp filtered https

$ nmap -A -T4 -p443 -6 my:server:ip::v6:address
443/tcp open  ssl/http nginx 1.10.3

$ nmap -A -T4 -p80 example.org
80/tcp open  http    nginx 1.10.3

$ nmap -A -T4 -p80 my.server.ip.address
80/tcp open  http    nginx 1.10.3

My nginx.conf is

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        client_max_body_size 50M;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json
application/javascript text/xml application/xml application/xml+rss
text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

and my nginx server blocks:

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name _ myservername.com;
        return 301 https://myservername.com$request_uri;
}

server {
        # SSL configuration
        #
        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;
        
        server_name _ myservername.com;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_pass http://localhost:8080;
       }

        ssl_certificate
/etc/letsencrypt/live/myservername.com/fullchain.pem;
        ssl_certificate_key
/etc/letsencrypt/live/myservername.com/privkey.pem;
}

server {
        listen 80;
        listen [::]:80;

        server_name example.com www.example.com;

        gzip off;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://localhost:8080;
        }
}

server {
       listen 80;
       listen [::]:80;

       server_name example.org www.example.org;
       return 301 https://example.org$request_uri;
}

server {

        # SSL configuration
        #
        listen 443 ssl;
        listen [::]:443 ssl;
        
        server_name example.org www.example.org;

        gzip off;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass
http://docker.container.ip.address:port/exist/apps/example/;
        }

        location /workshop2020/ {
                return 302 http://example.org/forum2020/;
        }


    location /exist/apps/example/ { 
            rewrite ^/exist/apps/example/(.*)$ /$1; 
    }


    ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem; #
managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem; #
managed by Certbot

}

Very grateful for any help!!
Nathan

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289099#msg-289099

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

zakirenish
Thanks for the reply, Thomas.

> You said this is "shared hosting" - when you say "shared hosting" do you
> mean this is *not* a dedicated machine but one machine out of many in a
> shared environment?

Sorry, I meant virtual hosting.

> Have you tested briefly by disabling your firewall just to see if that
> fixes the issue?

When I disable UFW I get the same nmap results. Somebody else configured the
server previously, so there could be something besides UFW interfering, but
I'm not sure where to check for that.

> What is the backend?  You're passing everything to 8080 which suggests
> the backend might be having issues too.

8080 is eXist-db running in a docker container (for example.org) and
standalone (for example.com). There are no issues connecting to these via
http.

Any thoughts what to try next?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289101#msg-289101

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

zakirenish
Just wondering if anyone has further thoughts on what to try here?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289172#msg-289172

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

Francis Daly
On Mon, Aug 24, 2020 at 07:35:24AM -0400, nathanpgibson wrote:

Hi there,

> Just wondering if anyone has further thoughts on what to try here?

You wrote:

"""
When I try nmap from my local machine I get some results I can't
explain. Notice the discrepancy between ports 80 and ports 443 and
between IPv4 and IPv6

$ nmap -A -T4 -p443 example.org
443/tcp filtered https

$ nmap -A -T4 -p443 my.server.ip.address
443/tcp filtered https

$ nmap -A -T4 -p443 -6 my:server:ip::v6:address
443/tcp open ssl/http nginx 1.10.3

$ nmap -A -T4 -p80 example.org
80/tcp open http nginx 1.10.3

$ nmap -A -T4 -p80 my.server.ip.address
80/tcp open http nginx 1.10.3
"""

For nmap, filtered means: Nmap cannot determine whether the port is
open because packet filtering prevents its probes from reaching the
port. The filtering could be from a dedicated firewall device, router
rules, or host-based firewall software.

(From https://nmap.org/book/man-port-scanning-basics.html)

That means that something in between your nmap testing client and your
nginx server is interfering with the IPv4 https/port 443 traffic. Find
and fix that something, and things will probably work better.


You also indicate that most visitors get a connection timeout message,
while some get through.

Do your nginx logs indicate that all of the ones that get through are
using IPv6, not IPv4? That might also point at IPv4 being blocked.

(Or: do your nginx logs indicate that all of the ones that get through
are coming from similar IP addresses? Perhaps there is wonky routing
involved? Although that would not explain the difference between ports
80 and 443 of the same IPv4 address.)

If you "tcpdump" on the nginx server for the port 443 traffic, do you
see anything? If tcpdump sees the traffic but nginx does not, there is
probably a local (on the same server as nginx) network control device
("firewall") involved. If tcpdump does not see the traffic, then there
is an external network control device involved.

If you, for example, "tcptraceroute" to your IPv4 address, port 443,
from a remote client, how far does the traffic get? That might hint at
where the first block is happening.

But right now, there is nothing obviously related to nginx in this
diagnosis.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

zakirenish
Thanks so much, Francis Daly! This is a huge help in isolating the problem.


Based on the nginx access log, IPv6 requests to port 443 are getting to
nginx but IPv4 requests to port 443 are not. But they are getting to
tcpdump. All I see there is a bunch of packets with the tcpflag [S]. I take
it this means the handshake is not completing.

It was easy to confirm this by turning off IPv6 in my browser, at which
point https stopped resolving for the site in the browser but I could see
the packets coming in on tcpdump.

So presumably some sort of firewall on the local server machine--probably
one I didn't configure or know about! I'll try to figure out how to find
that blockage.

In any case, apparently not an nginx issue, as you rightly perceived.

Thanks again for the help!
Nathan

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289184#msg-289184

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

zakirenish
Turned out there was an INPUT DROP rule in iptables (but not in ip6tables),
although I am using ufw as a firewall. Now https works and my nginx
redirects are functioning as expected!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289186#msg-289186

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

Francis Daly
On Tue, Aug 25, 2020 at 07:49:07AM -0400, nathanpgibson wrote:

Hi there,

> Turned out there was an INPUT DROP rule in iptables (but not in ip6tables),
> although I am using ufw as a firewall. Now https works and my nginx
> redirects are functioning as expected!

Great that you found and fixed the problem; and thanks for sharing the
answer with the list -- it will probably help the next person with
a similar head-scratching issue!

(I guess you either removed the INPUT DROP rule; or added an explicit
"allow 443" beside the "allow 80" rule that was already there. Whichever
it was, it was "make the local firewall allow the traffic get to nginx".)

Cheers,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Connection timeout on SSL with shared hosting

zakirenish
> (I guess you either removed the INPUT DROP rule; or added an explicit
> "allow 443" beside the "allow 80" rule that was already there.
> Whichever
> it was, it was "make the local firewall allow the traffic get to
> nginx".)

Right, the allow 443 actually existed but there was a rule above it that was
routing traffic such that it didn't even get to my allow rule. Using
iptables -nvL I was able to see the packet count and see that 0 packets were
getting to my allow rule.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289099,289268#msg-289268

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx