Proxy_Protocol with local Proxy_Pass to local secondary nginx process

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Proxy_Protocol with local Proxy_Pass to local secondary nginx process

Joseph Wonesh
Hello,

I am trying to route requests such that those requiring websockets will route to a long-lived nginx process, and all others will go to the general reverse-proxy which handles all other traffic. These nginx processes exist in our AWS cloud behind an ELB that has been configured to use Proxy Protocol. Note that all of this works correctly with our current setup which uses only one nginx process that is configured to use proxy_protocol.

The change to this setup is as follows:

The first nginx server handling all ingress uses proxy_protocol and forwards requests to either the websocket or non-websocket nginx servers locally:

server {
listen 8080 proxy_protocol;
real_ip_header proxy_protocol;
charset utf-8;
client_max_body_size 20M;

#send to websocket process
location /client {
proxy_pass http://localhost:8084;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Proxy-Scheme $scheme;
proxy_set_header X-Proxy-Port $proxy_port;
proxy_set_header X-ELB-Proxy-Scheme "https";
proxy_set_header X-ELB-Proxy-Port "443";

# Always support web socket connection upgrades
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}

#send to non-websocket process
location / {
proxy_pass http://localhost:8082;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Proxy-Scheme $scheme;
proxy_set_header X-Proxy-Port $proxy_port;
proxy_set_header X-ELB-Proxy-Scheme "https";
proxy_set_header X-ELB-Proxy-Port "443";

# Always support web socket connection upgrades
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}

When any non-websocket request is sent to localhost:8082, I get an empty reply. If I remove proxy_protocol from the first server, I get a response as expected. Obviously, I need proxy_protocl to support the ingress from our ELB, so removing it is not an option. However, I would like to know what pieces I am missing to route traffic correctly -- and I would also like to know why proxying a request locally from a proxy_protocol enabled server to another nginx process (regardless of this second process using proxy_protocol or not) fails.

For reference, the basic configuration of this secondary nginx process is below:

upstream console {
   server localhost:3000 max_fails=3 fail_timeout=60 weight=1;
}

server {
listen 8082;
client_max_body_size 20M; location /console { proxy_pass http://console } . . . }


Thank you all for your time,
Joseph Wonesh


This message is private and confidential. If you have received message in error, please notify us and remove from your system. 
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Proxy_Protocol with local Proxy_Pass to local secondary nginx process

Roman Arutyunyan
Hello Joseph,

On Wed, Aug 22, 2018 at 10:24:23AM -0400, Joseph Wonesh wrote:

> Hello,
>
> I am trying to route requests such that those requiring websockets will
> route to a long-lived nginx process, and all others will go to the general
> reverse-proxy which handles all other traffic. These nginx processes exist
> in our AWS cloud behind an ELB that has been configured to use Proxy
> Protocol. Note that all of this works correctly with our current setup
> which uses only one nginx process that is configured to use proxy_protocol.
>
> The change to this setup is as follows:
>
> The first nginx server handling all ingress uses proxy_protocol and
> forwards requests to either the websocket or non-websocket nginx servers
> locally:
>
> server {
>     listen 8080 proxy_protocol;
>     real_ip_header proxy_protocol;
>     charset utf-8;
>     client_max_body_size 20M;
>
>     #send to websocket process
>     location /client {
>       proxy_pass <a href="http://localhost:8084;">http://localhost:8084;
>       proxy_set_header Host $host;
>       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>       proxy_set_header X-Real-IP $proxy_protocol_addr;
>       proxy_set_header X-NginX-Proxy true;
>       proxy_set_header X-Proxy-Scheme $scheme;
>       proxy_set_header X-Proxy-Port $proxy_port;
>       proxy_set_header X-ELB-Proxy-Scheme "https";
>       proxy_set_header X-ELB-Proxy-Port "443";
>
>       # Always support web socket connection upgrades
>       proxy_http_version 1.1;
>       proxy_set_header Upgrade $http_upgrade;
>       proxy_set_header Connection "upgrade";
>     }
>
>   #send to non-websocket process
>   location / {
>           proxy_pass <a href="http://localhost:8082;">http://localhost:8082;
>           proxy_set_header Host $host;
>           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>           proxy_set_header X-Real-IP $proxy_protocol_addr;
>           proxy_set_header X-NginX-Proxy true;
>           proxy_set_header X-Proxy-Scheme $scheme;
>           proxy_set_header X-Proxy-Port $proxy_port;
>           proxy_set_header X-ELB-Proxy-Scheme "https";
>           proxy_set_header X-ELB-Proxy-Port "443";
>
>           # Always support web socket connection upgrades
>           proxy_http_version 1.1;
>           proxy_set_header Upgrade $http_upgrade;
>           proxy_set_header Connection "upgrade";
>   }
>
>
> When any non-websocket request is sent to localhost:8082, I get an empty
> reply. If I remove proxy_protocol from the first server, I get a response
> as expected.

If removing the proxy_protocol parameter from a server's listen directive fixes
the issue with a specific connection, this means that this connection does not
support the PROXY protocol, whether it is curl or a proxied connection from AWS.
So please make sure the PROXY protocol header is actually sent over the
connection.  It's easy to check by tcpdumping at most 107 first bytes,
especially if PROXY protocol v1 is used.

> Obviously, I need proxy_protocl to support the ingress from
> our ELB, so removing it is not an option. However, I would like to know
> what pieces I am missing to route traffic correctly -- and I would also
> like to know why proxying a request locally from a proxy_protocol enabled
> server to another nginx process (regardless of this second process using
> proxy_protocol or not) fails.

If you configure nginx to expect PROXY protocol (listen ... proxy_protocol),
then the client should actually send it.  AWS balancers can send it, as well
as nginx stream module can (proxy_protocol on).  The nginx http module
cannot send PROXY protocol header (use X-Forwarded-For instead), so you
cannot proxy an HTTP connection to a PROXY-protocol-enabled server.

> For reference, the basic configuration of this secondary nginx process is
> below:
>
> upstream console {
>    server localhost:3000 max_fails=3 fail_timeout=60 weight=1;
> }
>
> server {
>     listen 8082;
>    client_max_body_size 20M;
>
>    location /console {
>      proxy_pass http://console
>    }
>
>    .
>    .
>    .
>
>
> }
>
>
>
> Thank you all for your time,
> Joseph Wonesh
>
> --
> This message is private and confidential. If you have received message in
> error, please notify us and remove from your system. 

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Proxy_Protocol with local Proxy_Pass to local secondary nginx process

Joseph Wonesh
Thank you for your reply, Roman.

It turns out the issue was adding the various proxy and forwarded headers in that entry. Changing it to the below block fixed my issue:

#send to non-websocket process
location / {
proxy_pass http://localhost:8082;
proxy_set_header Host $host;
}

Now everything is working as expected.

Cheers,
Joseph Wonesh

On Thu, Aug 23, 2018 at 6:46 AM, Roman Arutyunyan <[hidden email]> wrote:
Hello Joseph,

On Wed, Aug 22, 2018 at 10:24:23AM -0400, Joseph Wonesh wrote:
> Hello,
>
> I am trying to route requests such that those requiring websockets will
> route to a long-lived nginx process, and all others will go to the general
> reverse-proxy which handles all other traffic. These nginx processes exist
> in our AWS cloud behind an ELB that has been configured to use Proxy
> Protocol. Note that all of this works correctly with our current setup
> which uses only one nginx process that is configured to use proxy_protocol.
>
> The change to this setup is as follows:
>
> The first nginx server handling all ingress uses proxy_protocol and
> forwards requests to either the websocket or non-websocket nginx servers
> locally:
>
> server {
>     listen 8080 proxy_protocol;
>     real_ip_header proxy_protocol;
>     charset utf-8;
>     client_max_body_size 20M;
>
>     #send to websocket process
>     location /client {
>       proxy_pass http://localhost:8084;
>       proxy_set_header Host $host;
>       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>       proxy_set_header X-Real-IP $proxy_protocol_addr;
>       proxy_set_header X-NginX-Proxy true;
>       proxy_set_header X-Proxy-Scheme $scheme;
>       proxy_set_header X-Proxy-Port $proxy_port;
>       proxy_set_header X-ELB-Proxy-Scheme "https";
>       proxy_set_header X-ELB-Proxy-Port "443";
>
>       # Always support web socket connection upgrades
>       proxy_http_version 1.1;
>       proxy_set_header Upgrade $http_upgrade;
>       proxy_set_header Connection "upgrade";
>     }
>
>   #send to non-websocket process
>   location / {
>           proxy_pass http://localhost:8082;
>           proxy_set_header Host $host;
>           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>           proxy_set_header X-Real-IP $proxy_protocol_addr;
>           proxy_set_header X-NginX-Proxy true;
>           proxy_set_header X-Proxy-Scheme $scheme;
>           proxy_set_header X-Proxy-Port $proxy_port;
>           proxy_set_header X-ELB-Proxy-Scheme "https";
>           proxy_set_header X-ELB-Proxy-Port "443";
>
>           # Always support web socket connection upgrades
>           proxy_http_version 1.1;
>           proxy_set_header Upgrade $http_upgrade;
>           proxy_set_header Connection "upgrade";
>   }
>
>
> When any non-websocket request is sent to localhost:8082, I get an empty
> reply. If I remove proxy_protocol from the first server, I get a response
> as expected.

If removing the proxy_protocol parameter from a server's listen directive fixes
the issue with a specific connection, this means that this connection does not
support the PROXY protocol, whether it is curl or a proxied connection from AWS.
So please make sure the PROXY protocol header is actually sent over the
connection.  It's easy to check by tcpdumping at most 107 first bytes,
especially if PROXY protocol v1 is used.

> Obviously, I need proxy_protocl to support the ingress from
> our ELB, so removing it is not an option. However, I would like to know
> what pieces I am missing to route traffic correctly -- and I would also
> like to know why proxying a request locally from a proxy_protocol enabled
> server to another nginx process (regardless of this second process using
> proxy_protocol or not) fails.

If you configure nginx to expect PROXY protocol (listen ... proxy_protocol),
then the client should actually send it.  AWS balancers can send it, as well
as nginx stream module can (proxy_protocol on).  The nginx http module
cannot send PROXY protocol header (use X-Forwarded-For instead), so you
cannot proxy an HTTP connection to a PROXY-protocol-enabled server.

> For reference, the basic configuration of this secondary nginx process is
> below:
>
> upstream console {
>    server localhost:3000 max_fails=3 fail_timeout=60 weight=1;
> }
>
> server {
>     listen 8082;
>    client_max_body_size 20M;
>
>    location /console {
>      proxy_pass http://console
>    }
>
>    .
>    .
>    .
>
>
> }
>
>
>
> Thank you all for your time,
> Joseph Wonesh
>
> --
> This message is private and confidential. If you have received message in
> error, please notify us and remove from your system. 

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


This message is private and confidential. If you have received message in error, please notify us and remove from your system. 
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx