nginx error (connect() failed, 61 "Connection refused", but everything works)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

nginx error (connect() failed, 61 "Connection refused", but everything works)

Gerben Wierda
I have a set of minio (S3-compatoble block storage) servers running behind nginx on macOS High Sierra. While everything seems to work OK, I have noticed an unexplained error in the nginx logs.
For instance, when I use minio’s mc to ls a file:
   mc ls nginx/gerbentest/duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes
mc reports (as expected):
   [2017-12-21 20:36:41 CET]  36KiB duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes

So, everything works. So does duplicati (the backup solution that uses those mini backends). I can also connect direct to the servers and it works fine.

But nginx reports (apparently once per session):
   2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: “GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1”, upstream: “http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes”, host: “MYHOST:9000”

I’d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is:

server {
    listen              9000 ssl;
    server_name         MYHOST;
    ssl_certificate     minio_certificate_chained.crt;
    ssl_certificate_key minio_certificate.key;
    ssl_protocols       TLSv1.2;
    proxy_buffering     off;
    client_max_body_size 1000m;
    location / {
        proxy_set_header Host $http_host;
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9001;
        }
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9002;
        }
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9003;
        }
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9004;
        }
    }
}




_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx error (connect() failed, 61 "Connection refused", but everything works)

Gerben Wierda
Answering myself: if I change localhost to 127.0.0.1 the error message goes away.

Why, though?


On 15 Aug 2018, at 12:02, Gerben Wierda <[hidden email]> wrote:

I have a set of minio (S3-compatoble block storage) servers running behind nginx on macOS High Sierra. While everything seems to work OK, I have noticed an unexplained error in the nginx logs.
For instance, when I use minio’s mc to ls a file:
   mc ls nginx/gerbentest/duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes
mc reports (as expected):
   [2017-12-21 20:36:41 CET]  36KiB duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes

So, everything works. So does duplicati (the backup solution that uses those mini backends). I can also connect direct to the servers and it works fine.

But nginx reports (apparently once per session):
   2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: “GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1”, upstream: “http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes”, host: “MYHOST:9000”

I’d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is:

server {
    listen              9000 ssl;
    server_name         MYHOST;
    ssl_certificate     minio_certificate_chained.crt;
    ssl_certificate_key minio_certificate.key;
    ssl_protocols       TLSv1.2;
    proxy_buffering     off;
    client_max_body_size 1000m;
    location / {
        proxy_set_header Host $http_host;
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9001;
        }
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9002;
        }
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9003;
        }
        if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
            proxy_pass http://localhost:9004;
        }
    }
}



_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx error (connect() failed, 61 "Connection refused", but everything works)

Sergey Kandaurov
In reply to this post by Gerben Wierda

> On 15 Aug 2018, at 13:02, Gerben Wierda <[hidden email]> wrote:

[..]

> But nginx reports (apparently once per session):
>    2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: “GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1”, upstream: “http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes”, host: “MYHOST:9000”

Note [::1]:9003 in upstream, which is likely the address nobody listens.

> I’d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is:
>
> server {
>     listen              9000 ssl;
>     server_name         MYHOST;
>     ssl_certificate     minio_certificate_chained.crt;
>     ssl_certificate_key minio_certificate.key;
>     ssl_protocols       TLSv1.2;
>     proxy_buffering     off;
>     client_max_body_size 1000m;
>     location / {
>         proxy_set_header Host $http_host;
>         if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
>             proxy_pass <a href="http://localhost:9001;">http://localhost:9001;
>         }

If a domain name resolves to several addresses,
which is apparently the case for "localhost",
all of them will be used in a round-robin fashion.

See for details:
http://nginx.org/r/proxy_pass

--
Sergey Kandaurov

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx error (connect() failed, 61 "Connection refused", but everything works)

Gerben Wierda
On 15 Aug 2018, at 14:05, Sergey Kandaurov <[hidden email]> wrote:


On 15 Aug 2018, at 13:02, Gerben Wierda <[hidden email]> wrote:

[..]

But nginx reports (apparently once per session):
  2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: “GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1”, upstream: “http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes”, host: “MYHOST:9000”

Note [::1]:9003 in upstream, which is likely the address nobody listens.

Ha, yes, now I see, the address is the IPv6 address for localhost. Silly me, I though in the network stack 127.0.0.1 and [::1] worked more or less as aliases for the same port, but apparently not. So, while minio is listening on 127.0.0.1:9003 it is not listening on [::1]:9003

I’d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is:

server {
   listen              9000 ssl;
   server_name         MYHOST;
   ssl_certificate     minio_certificate_chained.crt;
   ssl_certificate_key minio_certificate.key;
   ssl_protocols       TLSv1.2;
   proxy_buffering     off;
   client_max_body_size 1000m;
   location / {
       proxy_set_header Host $http_host;
       if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") {
           proxy_pass http://localhost:9001;
       }

If a domain name resolves to several addresses,
which is apparently the case for "localhost”,
all of them will be used in a round-robin fashion.

See for details:
http://nginx.org/r/proxy_pass

If I use dig or nslookup to resolve localhost, I get just 127.0.0.1. But my /etc/hosts contains both IPv4 and IPv6 for localhost and nginx round-robins over both apparently. Using 127.0.0.1 instead of localhost is the correct solution, then.

G



--
Sergey Kandaurov

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx