disable request body buffering for file upload

classic Classic list List threaded Threaded
13 messages Options
pva
Reply | Threaded
Open this post in threaded view
|

disable request body buffering for file upload

pva
Hello, hopefully someone can tell me if i am attempting the impossible.

We use nginx and php5-fpm to handle the upload of test files used by our web
app that may be several gigabytes in size.

To achieve this we use a location block for the upload url and define the
fastcgi_pass directive to provide the location of the fastcgi socket i.e.

location /api/filter/analysis/upload {
   fastcgi_pass unix:/tmp/php5-fpm.sock;
   include fastcgi_params;
   fastcgi_params SCRIPTFILENAME $document_root/PHP/uploadFile.php;
}

The above configuration functions however we have observed that nginx will
cache the entire request body before calling the php script which
effectively copies the uploaded temporary file to a different location.

Under certain circumstances this can be a major issue for us as our host
environment is within an instrument that is designed to capture data
continuously and will fill the hard drive close to 100% capacity before
stopping. If a user instigates the upload of a 1gb file that is cached first
by nginx and then copied elsewhere by the php script we can find ourselves
out of disk space at which point nginx (and other processes) understandably
stops functioning leaving the instrument in a state that is difficult to
recover from.

If we can prevent nginx from caching the entire request body and have it
pass it straight to our php script we should be able to take preventative
steps in the script and reject the upload attempt if disk space is too low.

I have looked at the directives 'proxy_request_buffering off;' and
'fastcgi_request_buffering off;' however i have not been successful in
stopping the initial nginx upload before our php script is called.

If this is the wrong approach can anyone suggest a different one? basically
we need to intercept the file upload request and reject it if we are too low
on disk space the current setup effectively uploads the file before we can
perform such a check.

Many thanks

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,275567#msg-275567

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

Maxim Dounin
Hello!

On Wed, Jul 19, 2017 at 10:33:01AM -0400, garyc wrote:

> Hello, hopefully someone can tell me if i am attempting the impossible.
>
> We use nginx and php5-fpm to handle the upload of test files used by our web
> app that may be several gigabytes in size.
>
> To achieve this we use a location block for the upload url and define the
> fastcgi_pass directive to provide the location of the fastcgi socket i.e.
>
> location /api/filter/analysis/upload {
>    fastcgi_pass unix:/tmp/php5-fpm.sock;
>    include fastcgi_params;
>    fastcgi_params SCRIPTFILENAME $document_root/PHP/uploadFile.php;
> }
>
> The above configuration functions however we have observed that nginx will
> cache the entire request body before calling the php script which
> effectively copies the uploaded temporary file to a different location.
>
> Under certain circumstances this can be a major issue for us as our host
> environment is within an instrument that is designed to capture data
> continuously and will fill the hard drive close to 100% capacity before
> stopping. If a user instigates the upload of a 1gb file that is cached first
> by nginx and then copied elsewhere by the php script we can find ourselves
> out of disk space at which point nginx (and other processes) understandably
> stops functioning leaving the instrument in a state that is difficult to
> recover from.
>
> If we can prevent nginx from caching the entire request body and have it
> pass it straight to our php script we should be able to take preventative
> steps in the script and reject the upload attempt if disk space is too low.
>
> I have looked at the directives 'proxy_request_buffering off;' and
> 'fastcgi_request_buffering off;' however i have not been successful in
> stopping the initial nginx upload before our php script is called.

With "fastcgi_request_buffering off;" nginx will send the request
body to the FastCGI application immediately, without trying to
buffer it anywhere.

An example configuration:

    location / {
        fastcgi_pass 127.0.0.1:10002;
        fastcgi_request_buffering off;
    }

It is up to your FastCGI application to handle this though,
and PHP as well as PHP-FPM may impose additional limitations
and/or require additional configuration for this to work.

> If this is the wrong approach can anyone suggest a different one? basically
> we need to intercept the file upload request and reject it if we are too low
> on disk space the current setup effectively uploads the file before we can
> perform such a check.

Doing such checks in nginx (for example, you can use embedded perl
to do such checks) and/or keeping client_max_body_size low enough might
be a better solution.  Nevertheless, fastcgi_request_buffering is
expected to work as well.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
Thank you, it sounds like this approach may yet work.

> It is up to your FastCGI application to handle this though,
> and PHP as well as PHP-FPM may impose additional limitations
> and/or require additional configuration for this to work.

I will dig into the php configuration details and see if i can progress
this, it is reassuring to hear that the fastcgi_request_buffering directive
is intended to facilitate this.

Thanks for the quick response.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,275570#msg-275570

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
In reply to this post by Maxim Dounin
Hi Maxim,

> With "fastcgi_request_buffering off;" nginx will send the request
> body to the FastCGI application immediately, without trying to
> buffer it anywhere.

I have been monitoring the disk space while uploading a test file of 1.1 GB
in size and have confirmed that with the fastcgi_request_buffering directive
set to 'on' the disk space reduces by approximately 2.2 GB.

With the fastcgi_request_buffering directive set to 'off' the disk space
reduces by approximately 1.1GB so it does look like the
fastcgi_request_buffering directive is doing what it should however nginx
still appears to cache the entire request body before it is proxied to
fastcgi.

> It is up to your FastCGI application to handle this though,
> and PHP as well as PHP-FPM may impose additional limitations
> and/or require additional configuration for this to work.

I am still looking into PHP & php5-fpm configuration parameters in case this
having an effect but it all feels like this cache is happening before the
proxy.

From what i have researched so far it looks like nginx may always initially
cache the entire request body in some form before the request is proxied.

Another approach we could take would be to add a disk space check api to our
web app so we can confirm we have sufficient space before the upload POST
call is made.

I appreciate that running a web server in a low disk space environment is
not a common scenario however, in your opinion, should it be possible to get
details of the upload POST request to a script before the entire request
body is written to disk by nginx?

Many thanks
Gary

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,275584#msg-275584

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

Maxim Dounin
Hello!

On Thu, Jul 20, 2017 at 06:36:21AM -0400, garyc wrote:

[...]

> Another approach we could take would be to add a disk space check api to our
> web app so we can confirm we have sufficient space before the upload POST
> call is made.
>
> I appreciate that running a web server in a low disk space environment is
> not a common scenario however, in your opinion, should it be possible to get
> details of the upload POST request to a script before the entire request
> body is written to disk by nginx?

Consider the auth request module,
http://nginx.org/en/docs/http/ngx_http_auth_request_module.html.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

Francis Daly
In reply to this post by pva
On Thu, Jul 20, 2017 at 06:36:21AM -0400, garyc wrote:

Hi there,

> With the fastcgi_request_buffering directive set to 'off' the disk space
> reduces by approximately 1.1GB so it does look like the
> fastcgi_request_buffering directive is doing what it should however nginx
> still appears to cache the entire request body before it is proxied to
> fastcgi.

Why do you think that?

If the problem is nginx buffering when it should not, then the fix should
be on the nginx side. If the problem is something else -- the fastcgi
server or something else doing the buffering -- then time spent changing
nginx is wasted.

Can you find out what is really happening?

For example: change the fastcgi server to listen on a network port;
change nginx to fastcgi_pass to that; and use tcpdump to watch the
traffic involving that port.

Then do a file upload that takes 30 seconds or more.

Does the tcpdump trace show the right amount of traffic going from
nginx to the fastcgi server during that 30 seconds while the upload is
happening? Or does it show no traffic until the upload has completed,
and then lots?

With that information, it may become clear whether the problem can be
fixed in nginx, or whether it must be handled elsewhere.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
In reply to this post by Maxim Dounin
Hello, Thanks for the hint I have managed to use the auth request module to
check available disk space before accepting the request so I only attempt
the upload if there is disk space available.

I am now having another problem. Our debian environment uses a rootfs
ramdisk which has just under 1GB of memory, when i accept the upload request
I can see the rootfs disk fill steadily until it is full at which point the
upload POST request is returned with error 500 and the message 'System error
- unable to complete file upload'.

Are there any configuration settings that would instruct nginx to use
another disk when accepting the client request body?

I have experimented with client_body_in_file_only and client_body_temp_path
and can observe that temporary files are created if they are small enough
for the rootfs to handle, files that are rejected when the rootfs fill ups
have no temporary file created so presumably this happens after the initial
read to the rootfs.

Many thanks
Gary

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,276155#msg-276155

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
Please ignore the last message, having learned a bit more about probing the
file system we can now see that it is PHP that is caching the file to the
system default location (hence rootfs) a small change to the PHP
configuration has sorted this.

Thanks to everyone for your help

Gary

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,276162#msg-276162

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
garyc Wrote:
-------------------------------------------------------
> Please ignore the last message, having learned a bit more about
> probing the file system we can now see that it is PHP that is caching
> the file to the system default location (hence rootfs) a small change
> to the PHP configuration has sorted this.
>
> Thanks to everyone for your help
>
> Gary

Do you mind sharing your "PHP.ini" solution so that others know what changes
to make to their "PHP.ini" to solve the same dilemma ?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,276169#msg-276169

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

Reinis Rozitis
> Do you mind sharing your "PHP.ini" solution so that others know what
> changes
> to make to their "PHP.ini" to solve the same dilemma ?

http://php.net/manual/en/ini.core.php#ini.upload-tmp-dir

It's usually not set so by default on most linux distros ends up being /tmp

rr

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
In reply to this post by pva
Hi, Reinis has probably covered this but the default php.ini file has a
'File Upload section' with...

; Temporary directory for HTTP uploaded files (will use system default if
not
; specified).
; upload_tmp_dir =

I just uncommented the attribute and set it to a location on our main disk
e.g.

upload_tmp_dir = /opt/tmp


--

The complete solution involved using the http_auth_request_module. So in our
nginx configuration file for big file upload url...

# PHP - file upload - bigf
location /api/bigf/analysis/upload {
   auth_request /bigf/auth;
   error_page 403 =413 /bigfLowDiskSpace.html;
   error_page 413 /bigfTooBigError.html;
               
   fastcgi_request_buffering off;   # pass the request straight to php
without buffering
   fastcgi_read_timeout 1h;
   fastcgi_pass unix:/opt/tmp/php-fpm.sock;
   
   include fastcgi_params;

   # Command specific parameters
   fastcgi_param PERMITTED_FILETYPE "bigf";
   fastcgi_param HOME_FOLDER "/home/instrument";
   fastcgi_param DEST_FOLDER "Analysis/Filters";
   fastcgi_param SCRIPT_FILENAME $document_root/PHP/uploadFile.php;
}

# from the auth_request directive in the above block

location /bigf/auth {
   internal;
               
   fastcgi_pass_request_body off;
   fastcgi_pass unix:/opt/tmp/php-fpm.sock;
   fastcgi_intercept_errors on;
               
   include fastcgi_params;
   fastcgi_param BIGF_UPLOAD_SIZE $content_length;
   fastcgi_param BIGF_UPLOAD_MARGIN_BYTES 10737418240; # reject if < 10GB
free after upload
   fastcgi_param HOME_FOLDER "/home/instrument";
   fastcgi_param DEST_FOLDER "Analysis/Filters";
   fastcgi_param CONTENT_LENGTH "";
   fastcgi_param SCRIPT_FILENAME $document_root/PHP/bigfAuthUpload.php;
}

# Custom error pages for the error_page directives specified above

location /bigfLowDiskSpace.html {
   root /opt/lib/webapp/errorPages;
   allow all;
}

location /bigfTooBigError.html {
   root /opt/lib/webapp/errorPages;
   allow all;
}


---

The bigfAuthUpload.php script just checks the space available on the
destination drive and if the space available minus the approximate incoming
file size (as it includes some bytes from the request header) breaks the
allowed margin (10G in this case) we reject the upload by  calling
http_response_code(403). If there is enough space http_response_code(200) is
set which 'authorizes' the upload and allows the uploadFile.php script to be
called.

The error_page 403 =413 redirect allows us to return an error page specific
to the rejected bigfAthUpload.php call.
The error_page 413 redirect allows us to intercept the nginx file size
restriction (we set the directive 'client_max_body_size' to 5G in the server
configuration block of our nginx configuration file), i believe the use of
'fastcgi_intercept_errors on;' in our auth block facilitates this.

By using the fastcgi_pass_request_body off; directive in the 'bigf/auth'
location block the bigfAuthUpload.php script is passed the request header
without the body so we can reject the upload before the request body is
written to /opt/tmp.

The uploadFile.php script effectively copies the file from /opt/tmp to the
destination location. It allows us to handle different $_FILES content
(dependent on the client used to call our upload service we need to cope
with $_FILES['upload'], $_FILES['Data'] and $_FILES['file'] variants to
extract the file name) and to rename the file to cope with duplicates.

The fastcgi_params file we include in the above location blocks include the
lines

Hope this helps someone else out, thanks to everyone who contributed!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,276176#msg-276176

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
pva
Reply | Threaded
Open this post in threaded view
|

Re: disable request body buffering for file upload

pva
Apologies, please ignore the line

> The fastcgi_params file we include in the above location blocks include
the lines

What followed wasn't really relevant (just about overriding php.ini values
with the PHP_VALUE command) so i removed it.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275567,276178#msg-276178

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx