Issue with NGINX as reverse proxy for grpc service

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez
Hi,

So we have a service exposing a grpc interface under a certain location
and we are using nginx in front of it. The config looks like the
following:

upstream grpcservers {
  server fqdn:port;
  server fqdn:port;
}

...

server {
  listen port ssl http2;
  client_max_body_size 15m;
  server_name fqdn;

  ssl_certificate /etc/certs/server.crt;
  ssl_certificate_key /etc/certs/server.key;

  location /my.location. {
    grpc_set_header X-Ip-Address $remote_addr;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    grpc_ssl_certificate /etc/ssl/mtls-client.crt;
    grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
    grpc_pass grpcs://grpcservers;
    ...
  }

  # Error responses
  include conf.d/errors.grpc_conf; # gRPC-compliant error responses
  default_type application/grpc;   # Ensure gRPC for all error
responses

} //End of the server directive

Now we just realized that each time we do a GET / to that specific port
under that specific location using curl --http2, the request is
forwarded to the backend in such a way that it makes nginx believe that
the backend has crashed, allowing anyone to DDoS this particular
service by just repeteadly sending GET / request to the endpoint.

I am seeing the following messages in the logs:

020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected request
with error 2 while reading response header from upstream, client:
X.X.X.X, server: fqdn1, request: "POST /my.location.magic.API/GetMagic
HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"

Eventually after the 2 upstream servers are marked as failed we get the
following message:

020/08/07 11:07:05 [error] 1100#1100: *96 no live upstreams while
connecting to upstream, client: X.X.X.X, server: fqdn1, request: "POST
/my.location.magic.API/GetMagic HTTP/2.0", upstream:
"grpcs://grpcservers", host: "fqdn1:PORT"

until the servers are marked as valid again. And the cycle repeats.

I am not an expert on HTTP/2 or gRPC, but it seems like nginx is unable
to negotiate the connection with the backend for those particular
requests created with curl and ends marking the backend as failed when
in fact, it is not failing. Any ideas about how can I further debug
this issue?

Thanks in advance.

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Issue with NGINX as reverse proxy for grpc service

Sergey Kandaurov

> On 7 Aug 2020, at 17:18, Víctor Enríquez <[hidden email]> wrote:
>
> Hi,
>
> So we have a service exposing a grpc interface under a certain location
> and we are using nginx in front of it. The config looks like the
> following:
>
> upstream grpcservers {
>  server fqdn:port;
>  server fqdn:port;
> }
>
> ...
>
> server {
>  listen port ssl http2;
>  client_max_body_size 15m;
>  server_name fqdn;
>
>  ssl_certificate /etc/certs/server.crt;
>  ssl_certificate_key /etc/certs/server.key;
>
>  location /my.location. {
>    grpc_set_header X-Ip-Address $remote_addr;
>    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>    grpc_ssl_certificate /etc/ssl/mtls-client.crt;
>    grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
>    grpc_pass grpcs://grpcservers;
>    ...
>  }
>
>  # Error responses
>  include conf.d/errors.grpc_conf; # gRPC-compliant error responses
>  default_type application/grpc;   # Ensure gRPC for all error
> responses
>
> } //End of the server directive
>
> Now we just realized that each time we do a GET / to that specific port
> under that specific location using curl --http2, the request is
> forwarded to the backend in such a way that it makes nginx believe that
> the backend has crashed, allowing anyone to DDoS this particular
> service by just repeteadly sending GET / request to the endpoint.
>
> I am seeing the following messages in the logs:
>
> 020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected request
> with error 2 while reading response header from upstream, client:
> X.X.X.X, server: fqdn1, request: "POST /my.location.magic.API/GetMagic
> HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"

"error 2" means that backend responded with RST_STREAM(INTERNAL_ERROR),
that is, effectively rejected processing request.
You may want to consult with backend error log to find out the reason.

--
Sergey Kandaurov

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez
Hi Sergey,

First thanks for your reply.

What I don't really understand is, shouldn't nginx be more strict by
default with the requests that are passed to grpc backends? I have been
reading a little bit about the GRPC protocol, and it's supposed to just
use POST requests (I might be wrong about this though). Shouldn't any
other kind of request be filtered by nginx directly (at least if it is
obvious that they are malformed)?

I am going to try to reproduce this with a docker compose file and the
standard hello grpc example, to see if it's a failure on our backend
implementation or if I can reproduce it with other grpc backends.

Other than this issue, the backend is working fine if the request is
not created with curl (the clients that are supposed to use the backend
are working fine), so I guess there is some checking on the golang's
grpc implementation to check if the request is valid, and it returns
that value RST_STREAM(INTERNAL_ERROR) (but I think we don't have
control over it AFAIK, again I might be wrong).

To me ,without being an expert, it sounds like there is something
wrong, either on the nginx side or on the grpc implementation side of
our backend, but we are not returning that error code directly.
(Perhaps the golang's grpc implementation should return other error
code that doesn't make nginx believe that the upstream is dead when it
receives a malformed request).

Again thanks for your help.

On Fri, 2020-08-07 at 19:28 +0300, Sergey Kandaurov wrote:

> > On 7 Aug 2020, at 17:18, Víctor Enríquez <[hidden email]> wrote:
> >
> > Hi,
> >
> > So we have a service exposing a grpc interface under a certain
> > location
> > and we are using nginx in front of it. The config looks like the
> > following:
> >
> > upstream grpcservers {
> >  server fqdn:port;
> >  server fqdn:port;
> > }
> >
> > ...
> >
> > server {
> >  listen port ssl http2;
> >  client_max_body_size 15m;
> >  server_name fqdn;
> >
> >  ssl_certificate /etc/certs/server.crt;
> >  ssl_certificate_key /etc/certs/server.key;
> >
> >  location /my.location. {
> >    grpc_set_header X-Ip-Address $remote_addr;
> >    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> >    grpc_ssl_certificate /etc/ssl/mtls-client.crt;
> >    grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
> >    grpc_pass grpcs://grpcservers;
> >    ...
> >  }
> >
> >  # Error responses
> >  include conf.d/errors.grpc_conf; # gRPC-compliant error responses
> >  default_type application/grpc;   # Ensure gRPC for all error
> > responses
> >
> > } //End of the server directive
> >
> > Now we just realized that each time we do a GET / to that specific
> > port
> > under that specific location using curl --http2, the request is
> > forwarded to the backend in such a way that it makes nginx believe
> > that
> > the backend has crashed, allowing anyone to DDoS this particular
> > service by just repeteadly sending GET / request to the endpoint.
> >
> > I am seeing the following messages in the logs:
> >
> > 020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected
> > request
> > with error 2 while reading response header from upstream, client:
> > X.X.X.X, server: fqdn1, request: "POST
> > /my.location.magic.API/GetMagic
> > HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"
>
> "error 2" means that backend responded with
> RST_STREAM(INTERNAL_ERROR),
> that is, effectively rejected processing request.
> You may want to consult with backend error log to find out the
> reason.
>

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx