Unable to use a GET url-param

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Unable to use a GET url-param

Ajay Garg
Hi All.

When I do the following call ::

                            https://username:password@1.2.3.4?upstream_protocol=http

I get a 500 error (on the browser-client), with the following seen in /var/log/nginx/error.log (on nginx-server)

######################################################
2017/04/14 13:03:51 [error] 16039#16039: *1 invalid URL prefix in "://127.0.0.1:5000", client: 182.69.5.226, server: , request: "GET /cgi-bin/webproc HTTP/1.1", host: "1.2.3.4", referrer: "https://1.2.3.4/?upstream_protocol=http"
######################################################


Following is the server-block section in /etc/nginx/conf.d/default.conf

######################################################
server {

                listen 443 ssl;

                ssl_certificate /etc/nginx/ssl/nginx.crt;
                ssl_certificate_key /etc/nginx/ssl/nginx.key;

                location / {

                        auth_basic 'Restricted';
                        auth_basic_user_file /etc/nginx/ssl/.htpasswd;

                        proxy_pass $arg_upstream_protocol://127.0.0.1:$forwarded_port;
                }
        }
######################################################

It definitely looks that "upstream_protocol" parameter is not being picked up by $arg_upstream_protocol.

What am I missing?
Will be grateful for pointers.


Thanks and Regards,
Ajay

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Francis Daly
On Fri, Apr 14, 2017 at 06:42:22PM +0530, Ajay Garg wrote:

Hi there,

> When I do the following call ::
>
> https://username:password@1.2.3.4?upstream_protocol=http

> 2017/04/14 13:03:51 [error] 16039#16039: *1 invalid URL prefix in "://
> 127.0.0.1:5000", client: 182.69.5.226, server: , request: "GET
> /cgi-bin/webproc HTTP/1.1", host: "1.2.3.4", referrer: "
> https://1.2.3.4/?upstream_protocol=http"

> What am I missing?

The request in the log line is not the same as the first request provided.

What is the output of "curl -v" on the first request? If it is not
exactly what you expect, what does the nginx log say for that one request?

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Ajay Garg
Hi Francis.

I tried the curl method, and I happened to land on an interesting observation.

a)
If there is no forwarded-port in listening state (port 5000 in this case) for the upstream-server, the request suitably returns a 502 error. More importantly, the $arg_upstream_protocol does seem to be parsed properly ::

#####################################################
2017/04/15 09:08:56 [error] 16039#16039: *40 connect() failed (111: Connection refused) while connecting to upstream, client: 182.69.5.226, server: , request: "GET /?upstream_protocol=http HTTP/1.1", upstream: "http://127.0.0.1:5000/?upstream_protocol=http", host: "1.2.3.4"
#####################################################

b)
However, if the forwarded port is in listening state, I get the usual 500 error ::

#####################################################
2017/04/15 09:08:21 [error] 16039#16039: *37 invalid URL prefix in "://127.0.0.1:5000", client: 182.69.5.226, server: , request: "GET /cgi-bin/webproc HTTP/1.1", host: "1.2.3.4", referrer: "https://1.2.3.4/?upstream_protocol=http"
#####################################################


Note that /cgi-bin/webproc is the default location for the upstream-server.
Also, to re-iterate, following is the proxy-pass directive ::

                        proxy_pass $arg_upstream_protocol://127.0.0.1:$forwarded_port;


So, the GET-param is being parsed fine (as evident from case a), seems I need to do some url-rewritings while the requests move to and from between nginx and upstream-server, right?


On Sat, Apr 15, 2017 at 1:55 PM, Francis Daly <[hidden email]> wrote:
On Fri, Apr 14, 2017 at 06:42:22PM +0530, Ajay Garg wrote:

Hi there,

> When I do the following call ::
>
> https://username:password@1.2.3.4?upstream_protocol=http

> 2017/04/14 13:03:51 [error] 16039#16039: *1 invalid URL prefix in "://
> 127.0.0.1:5000", client: 182.69.5.226, server: , request: "GET
> /cgi-bin/webproc HTTP/1.1", host: "1.2.3.4", referrer: "
> https://1.2.3.4/?upstream_protocol=http"

> What am I missing?

The request in the log line is not the same as the first request provided.

What is the output of "curl -v" on the first request? If it is not
exactly what you expect, what does the nginx log say for that one request?

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Regards,
Ajay

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Francis Daly
On Sat, Apr 15, 2017 at 02:47:26PM +0530, Ajay Garg wrote:

Hi there,

> If there is no forwarded-port in listening state (port 5000 in this case)
> for the upstream-server, the request suitably returns a 502 error. More
> importantly, the $arg_upstream_protocol does seem to be parsed properly ::

Why do you have $arg_upstream_protocol? What is its purpose?

After you answer that, consider: why do you not also have
$arg_forwarded_port?

If the port to connect to, and the protocol to connect with, are
conceptually analogous, they should probably be handled in the same way.

(Set them both in maps.)

> So, the GET-param is being parsed fine (as evident from case a), seems I
> need to do some url-rewritings while the requests move to and from between
> nginx and upstream-server, right?

One request gets one response. If the response is a http 301, the next
request is a whole new request that should be considered separately.

If at all possible, do not design things so that you need to edit the
upstream response body before sending it to the client.

So: what is the output of "curl -v" on the first request?

What do you want the output to be, in your design?

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Ajay Garg
Hi Francis.

Thanks for your continued help.

On Sat, Apr 15, 2017 at 8:50 PM, Francis Daly <[hidden email]> wrote:
On Sat, Apr 15, 2017 at 02:47:26PM +0530, Ajay Garg wrote:

Hi there,

> If there is no forwarded-port in listening state (port 5000 in this case)
> for the upstream-server, the request suitably returns a 502 error. More
> importantly, the $arg_upstream_protocol does seem to be parsed properly ::

Why do you have $arg_upstream_protocol? What is its purpose?

After you answer that, consider: why do you not also have
$arg_forwarded_port?

If the port to connect to, and the protocol to connect with, are
conceptually analogous, they should probably be handled in the same way.


Our architecture is as follows ::

   Proxy-Server  <==>  Gateway <==>  End-Server

Proxy-Server and Gateway are connected via a ssh-reverse-tunnel.
The port over which they are connected remains the same, as long as the Gateway is same.
So, $forwarded_port can be safely set in the map.

Gateway and End-Server communicate via the "other end" of the ssh-reverse-tunnel.
The End-Server here might change, and so the communication can either be over http or https.
This information is passed as a GET-param, when making the request to the Proxy-Server.
So, $arg_upstream_protocol comes into picture.



(Set them both in maps.)


I have already tried this via

map $remote_user $forwarded_protocol {
  
    ajay           $arg_upstream_protocol
}

....
.....
                proxy_pass $forwarded_protocol://127.0.0.1:$forwarded_port;

but I get the same results as per my previous emails.






 

> So, the GET-param is being parsed fine (as evident from case a), seems I
> need to do some url-rewritings while the requests move to and from between
> nginx and upstream-server, right?

One request gets one response. If the response is a http 301, the next
request is a whole new request that should be considered separately.

If at all possible, do not design things so that you need to edit the
upstream response body before sending it to the client.

So: what is the output of "curl -v" on the first request?

Following is received ::

#####################################################
curl -v -k https://ajay:garg@1.2.3.4/?upstream_protocol=http
* Hostname was NOT found in DNS cache
*   Trying 1.2.3.4...

* Connected to 1.2.3.4 (1.2.3.4) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS Unknown, Unknown (22):
* SSLv3, TLS handshake, Client hello (1):

* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Server hello (2):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, CERT (11):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Server finished (14):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv2, Unknown (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Finished (20):
* SSLv2, Unknown (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*      subject: C=IN; ST=Delhi; L=Delhi; O=Home; OU=Home; CN=www.home.com; emailAddress=[hidden email]
*      start date: 2017-04-09 03:53:25 GMT
*      expire date: 2027-04-07 03:53:25 GMT
*      issuer: C=IN; ST=Delhi; L=Delhi; O=Home; OU=Home; CN=www.home.com; emailAddress=[hidden email]
*      SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'ajay'
* SSLv2, Unknown (23):
> GET /?upstream_protocol=http HTTP/1.1
> Authorization: Basic abcdefg
> User-Agent: curl/7.37.1
> Host: 1.2.3.4
> Accept: */*
>
* SSLv2, Unknown (23):
< HTTP/1.1 200 Ok
* Server nginx/1.11.13 is not blacklisted
< Server: nginx/1.11.13
< Date: Sun, 16 Apr 2017 03:42:22 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 75
< Connection: keep-alive
< Last-Modified: Sat, 08 Aug 2015 04:40:50 GMT
<
<script>
<!--
    window.location.href = "/cgi-bin/webproc";
-->
* Connection #0 to host 1.2.3.4 left intact
#####################################################


Strangely, when request is done by curl, absolutely nothing appears in /var/log/nignix/error.log, whereas when done through the browser, logs appear in /var/log/nginx/error.log as per my previous emails.


Beginning to feel a little lost again :-\
But I believe that the experts will sail me through..






 

What do you want the output to be, in your design?

Things work fine if I hardcode http/https in proxy_pass directive.
It's only when I need to use to parse-and-use "upstream_protocol" from the GET-param (which can only be equal to http/https) that I start facing problems.


 

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



Thanks and Regards,
Ajay

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Ajay Garg
Never mind.

I created two different credentials, one which forwards via http-protocol, and the other which forwards via https-protocol.
Thanks everyone (especially Francis) for all the help !!!


Thanks and Regards,
Ajay

On Sun, Apr 16, 2017 at 9:49 AM, Ajay Garg <[hidden email]> wrote:
Hi Francis.

Thanks for your continued help.

On Sat, Apr 15, 2017 at 8:50 PM, Francis Daly <[hidden email]> wrote:
On Sat, Apr 15, 2017 at 02:47:26PM +0530, Ajay Garg wrote:

Hi there,

> If there is no forwarded-port in listening state (port 5000 in this case)
> for the upstream-server, the request suitably returns a 502 error. More
> importantly, the $arg_upstream_protocol does seem to be parsed properly ::

Why do you have $arg_upstream_protocol? What is its purpose?

After you answer that, consider: why do you not also have
$arg_forwarded_port?

If the port to connect to, and the protocol to connect with, are
conceptually analogous, they should probably be handled in the same way.


Our architecture is as follows ::

   Proxy-Server  <==>  Gateway <==>  End-Server

Proxy-Server and Gateway are connected via a ssh-reverse-tunnel.
The port over which they are connected remains the same, as long as the Gateway is same.
So, $forwarded_port can be safely set in the map.

Gateway and End-Server communicate via the "other end" of the ssh-reverse-tunnel.
The End-Server here might change, and so the communication can either be over http or https.
This information is passed as a GET-param, when making the request to the Proxy-Server.
So, $arg_upstream_protocol comes into picture.



(Set them both in maps.)


I have already tried this via

map $remote_user $forwarded_protocol {
  
    ajay           $arg_upstream_protocol
}

....
.....
                proxy_pass $forwarded_protocol://127.0.0.1:$forwarded_port;

but I get the same results as per my previous emails.






 

> So, the GET-param is being parsed fine (as evident from case a), seems I
> need to do some url-rewritings while the requests move to and from between
> nginx and upstream-server, right?

One request gets one response. If the response is a http 301, the next
request is a whole new request that should be considered separately.

If at all possible, do not design things so that you need to edit the
upstream response body before sending it to the client.

So: what is the output of "curl -v" on the first request?

Following is received ::

#####################################################
curl -v -k https://ajay:garg@1.2.3.4/?upstream_protocol=http
* Hostname was NOT found in DNS cache
*   Trying 1.2.3.4...

* Connected to 1.2.3.4 (1.2.3.4) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS Unknown, Unknown (22):
* SSLv3, TLS handshake, Client hello (1):

* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Server hello (2):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, CERT (11):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Server finished (14):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv2, Unknown (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Finished (20):
* SSLv2, Unknown (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv2, Unknown (22):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*      subject: C=IN; ST=Delhi; L=Delhi; O=Home; OU=Home; CN=www.home.com; emailAddress=[hidden email]
*      start date: 2017-04-09 03:53:25 GMT
*      expire date: 2027-04-07 03:53:25 GMT
*      issuer: C=IN; ST=Delhi; L=Delhi; O=Home; OU=Home; CN=www.home.com; emailAddress=[hidden email]
*      SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'ajay'
* SSLv2, Unknown (23):
> GET /?upstream_protocol=http HTTP/1.1
> Authorization: Basic abcdefg
> User-Agent: curl/7.37.1
> Host: 1.2.3.4
> Accept: */*
>
* SSLv2, Unknown (23):
< HTTP/1.1 200 Ok
* Server nginx/1.11.13 is not blacklisted
< Server: nginx/1.11.13
< Date: Sun, 16 Apr 2017 03:42:22 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 75
< Connection: keep-alive
< Last-Modified: Sat, 08 Aug 2015 04:40:50 GMT
<
<script>
<!--
    window.location.href = "/cgi-bin/webproc";
-->
* Connection #0 to host 1.2.3.4 left intact
#####################################################


Strangely, when request is done by curl, absolutely nothing appears in /var/log/nignix/error.log, whereas when done through the browser, logs appear in /var/log/nginx/error.log as per my previous emails.


Beginning to feel a little lost again :-\
But I believe that the experts will sail me through..






 

What do you want the output to be, in your design?

Things work fine if I hardcode http/https in proxy_pass directive.
It's only when I need to use to parse-and-use "upstream_protocol" from the GET-param (which can only be equal to http/https) that I start facing problems.


 

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



Thanks and Regards,
Ajay



--
Regards,
Ajay

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Francis Daly
In reply to this post by Ajay Garg
On Sun, Apr 16, 2017 at 09:49:09AM +0530, Ajay Garg wrote:
> On Sat, Apr 15, 2017 at 8:50 PM, Francis Daly <[hidden email]> wrote:
> > On Sat, Apr 15, 2017 at 02:47:26PM +0530, Ajay Garg wrote:

Hi there,

> Proxy-Server and Gateway are connected via a ssh-reverse-tunnel.
> The port over which they are connected remains the same, as long as the
> Gateway is same.
> So, $forwarded_port can be safely set in the map.

Ok.

> Gateway and End-Server communicate via the "other end" of the
> ssh-reverse-tunnel.
> The End-Server here might change, and so the communication can either be
> over http or https.

I suspect that that is the piece that is breaking you now.

I think that it will be much easier to configure if the upstream is
http, or the upstream is https, and changing is a significant event,
which involves reconfiguring nginx.

> This information is passed as a GET-param, when making the request to the
> Proxy-Server.

As shown below, that design cannot work without more changes. If possible,
I suggest you remove that "changeable" desire. If not, you will need a
fuller design that accounts for it.

I would first suggest making all upstreams accessible over http. Failing
that, you may be able to build something based on cookies -- if the GET
param is present, set a cookie and use that value; else, if the cookie
is present, use that value; else do something appropriate -- perhaps
fail and tell the user why; perhaps assume http.

(If you can get a reliable "you used http instead of https" error code
from upstream, you might be able to use that information when setting
the cookie, so the thing can auto-tune.)
 
> I have already tried this via
>
> map $remote_user $forwarded_protocol {
>
>     ajay           $arg_upstream_protocol
> }

I was thinking more like

     ajay           http;

because $arg_upstream_protocol is not always present.

> > One request gets one response. If the response is a http 301, the next
> > request is a whole new request that should be considered separately.
> >
> > If at all possible, do not design things so that you need to edit the
> > upstream response body before sending it to the client.

> curl -v -k https://ajay:garg@1.2.3.4/?upstream_protocol=http

> > GET /?upstream_protocol=http HTTP/1.1
> > Authorization: Basic abcdefg

> < HTTP/1.1 200 Ok

> <script>
> <!--
>     window.location.href = "/cgi-bin/webproc";
> -->

So - the first request is handled as you want it to be. It includes
in its response body an invitation to the browser to make a new request,
to /cgi-bin/webproc.

Not to /cgi-bin/webproc?upstream_protocol=http.


Eventually, there will be a html page with many "a href" and "img src"
urls; none of them will include "?upstream_protocol=http" either. (Unless
you significantly change the upstream config to always include it --
which you do not want to do -- or you change the nginx config to edit
the upstream response body to always include it before returning it to
the client -- which you also do not want to do.)

> Strangely, when request is done by curl, absolutely nothing appears in
> /var/log/nignix/error.log, whereas when done through the browser, logs
> appear in /var/log/nginx/error.log as per my previous emails.

curl makes the one request that you ask it to make, and shows the
response. So you can see exactly what is happening.

Then, if you want to, you can use it to make the next one request that
the browser makes, so again you can see exactly what is happening.

I suspect that if you do

  curl -v -k https://ajay:garg@1.2.3.4/cgi-bin/webproc

you will see that same result in nginx as when the browser makes that
request.

But now you know that the error message is not from the first request,
so you do not need to try to fix the first request.

And if you do

  curl -v -k https://ajay:garg@1.2.3.4/cgi-bin/webproc?upstream_protocol=http

(or if you use that url in a browser) you will see what comes back and
see the next problem.

> Things work fine if I hardcode http/https in proxy_pass directive.
> It's only when I need to use to parse-and-use "upstream_protocol" from the
> GET-param (which can only be equal to http/https) that I start facing
> problems.

Yes - you only face those problems when "upstream_protocol" is not
provided in the GET-params. So you must design things so that either
"upstream_protocol" is present in every request; or design things so
that the information that it carries is available in every request --
which probably means "always http" or "tied to authentication" or "tied
to a cookie".

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Unable to use a GET url-param

Francis Daly
In reply to this post by Ajay Garg
On Mon, Apr 17, 2017 at 11:19:36AM +0530, Ajay Garg wrote:

Hi there,

> I created two different credentials, one which forwards via http-protocol,
> and the other which forwards via https-protocol.

Good that you came up with a system that works.

Cheers,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Loading...