FastCGI application not receiving multi-part data

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

FastCGI application not receiving multi-part data

vedranf
Hello All,

I'm working on a FastCGI application the needs to support file upload. When a file is uploaded the FastCGI program is not receiving any data or envp other than METHOD. I think I have configured FastCGI correctly in the nginx config file, because other form submissions work.  Can anyone give me an idea what I could check?

nginx version: latest
FastCGI version: latest


Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4091#msg-4091


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

Daniel Rhodes
Is your FastCGI running on a socket or a port? On Linux, it should be
a socket (e.g. unix:/tmp/php.sock) , but on FreeBSD it should be a
port (e.g. 127.0.0.1:60000). I found this to be the cause of uploads
not being passed back through fastcgi one time.

-- DR

On Wed, Jul 15, 2009 at 10:31 PM, gerryw<[hidden email]> wrote:

> Hello All,
>
> I'm working on a FastCGI application the needs to support file upload. When a file is uploaded the FastCGI program is not receiving any data or envp other than METHOD. I think I have configured FastCGI correctly in the nginx config file, because other form submissions work.  Can anyone give me an idea what I could check?
>
> nginx version: latest
> FastCGI version: latest
>
>
> Thanks,
> Gerry
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4091#msg-4091
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

Almir Karic
In reply to this post by vedranf
I am running django via FCGI that has the file upload working just
fine :-), i doubt nginx is source of your problems. Did you remember
to add enctype="multipart/form-data"?


python/django hacker & sys admin
http://almirkaric.com & http://twitter.com/redduck666



On Wed, Jul 15, 2009 at 10:31 PM, gerryw<[hidden email]> wrote:

> Hello All,
>
> I'm working on a FastCGI application the needs to support file upload. When a file is uploaded the FastCGI program is not receiving any data or envp other than METHOD. I think I have configured FastCGI correctly in the nginx config file, because other form submissions work.  Can anyone give me an idea what I could check?
>
> nginx version: latest
> FastCGI version: latest
>
>
> Thanks,
> Gerry
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4091#msg-4091
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

Daniel:
>Is your FastCGI running on a socket or a port? On Linux, it should be
>a socket (e.g. unix:/tmp/php.sock) , but on FreeBSD it should be a
>port (e.g. 127.0.0.1:60000). I found this to be the cause of uploads
>not being passed back through fastcgi one time.

I was using a port. When I switched to a unix:socket I started getting some data, but it's either garbage or incomplete, because the parser segfaults trying to parse it. Could it be some kind of buffer size issue?

Almir:
> Did you remember to add enctype="multipart/form-data"?
Yes.

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4099#msg-4099


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

In order to eliminate the parser as the problem, I switched to cgicc. Unfortunately, I got the same result. I know that cgicc works, because I've used it with other web servers. I guess I'm still thinking the data is incomplete or doesn't match the length passed. Any thoughts?

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4100#msg-4100


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

Daniel Rhodes
In reply to this post by vedranf
I'm not familiar with that particular problem, although here are the
settings I use for the fcgi buffers (which has worked with me for
files > 1GB being passed on to PHP):

fastcgi_connect_timeout         60;
fastcgi_send_timeout            180;
fastcgi_read_timeout            180;
fastcgi_buffer_size             128k;
fastcgi_buffers                 4 256k;
fastcgi_busy_buffers_size       256k;
fastcgi_temp_file_write_size    256k;
fastcgi_intercept_errors        on;
fastcgi_pass_header             *;

There are also other values, like the document body size which should
be set. You can find information about that in the documentation.

On Thu, Jul 16, 2009 at 1:56 AM, gerryw<[hidden email]> wrote:

> Hello All,
>
> Daniel:
>>Is your FastCGI running on a socket or a port? On Linux, it should be
>>a socket (e.g. unix:/tmp/php.sock) , but on FreeBSD it should be a
>>port (e.g. 127.0.0.1:60000). I found this to be the cause of uploads
>>not being passed back through fastcgi one time.
>
> I was using a port. When I switched to a unix:socket I started getting some data, but it's either garbage or incomplete, because the parser segfaults trying to parse it. Could it be some kind of buffer size issue?
>
> Almir:
>> Did you remember to add enctype="multipart/form-data"?
> Yes.
>
> Thanks,
> Gerry
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4099#msg-4099
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

I also did a test with just the vanilla FastCGI API calls. The only env related variable I was able to get was "REQUEST_METHOD". None of the rest were set.

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4101#msg-4101


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

Michael Shadle
post your configuration

On Thu, Jul 16, 2009 at 2:27 AM, gerryw<[hidden email]> wrote:

> Hello All,
>
> I also did a test with just the vanilla FastCGI API calls. The only env related variable I was able to get was "REQUEST_METHOD". None of the rest were set.
>
> Thanks,
> Gerry
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4101#msg-4101
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

It looks like the config options Daniel recommended are sending the data. I think the multipart data in this case is something that most of the C/C++ cgi libs are choking on. If I just read the data from stdin and dump it, I can see the data I need. I'm going to try writing my own little parser for this. I'm using the very cool Upload module, so I don't need the actual file data.

I appreciate the help y'all have given me with this. I'll come back and post the result.

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4125#msg-4125


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

I have finally figured out what is happening, The are no envronment variables set.  For some reason getenv will return a bogus pointer at times, which will result in an invalid CONTENT_LENGTH and cause a segfault. Would there be any reason why these variables are not being set?

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4139#msg-4139


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

Igor Sysoev
On Fri, Jul 17, 2009 at 12:52:58AM -0400, gerryw wrote:

> Hello All,
>
> I have finally figured out what is happening, The are no envronment variables set.  For some reason getenv will return a bogus pointer at times, which will result in an invalid CONTENT_LENGTH and cause a segfault. Would there be any reason why these variables are not being set?

It seems you did not set or include fastcgi_param's:

    include conf/fastcgi_params;


--
Igor Sysoev
http://sysoev.ru/en/

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

Well... many hours later I am forced to give up on this. It would seem that there are some issues with the FastCGI implementation that PHP and the like must have worked around or their multipart form parser is very mature and robust. Unfortunately, php is not an option. I am working on a project that I can't delay any further. I really like nginx in general and I wish I could have used it. I appreciate those who tried to help. Maybe I can take another look when I don't have a deadline to meet.

BTW: The problem is not the fastcgi_param stuff. It has to do with the format of the multipart data. None of the current open source FastCGI compatible C libraries that I could find can parse it.

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4158#msg-4158


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

Toni Mueller-2

Hi,

On Fri, 17.07.2009 at 19:06:30 -0400, gerryw <[hidden email]> wrote:
> seem that there are some issues with the FastCGI implementation that
> PHP and the like must have worked around or their multipart form
> parser is very mature and robust.

FWIW, I found nginx's handling of FastCGI "mature enough" after
upgrading to 0.7.61. At least, it enabled me to run MovableType behind
it, which I didn't manage to do before.

You did set variables like client_body_temp_path?

> BTW: The problem is not the fastcgi_param stuff. It has to do with
> the format of the multipart data. None of the current open source
> FastCGI compatible C libraries that I could find can parse it.

It would be nice to know more about the application and data you're
working with.


Kind regards,
--Toni++

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hello All,

I have finally figured out what the problem was. For some reason when the local unix: socket was created there was an OS issue that was corrupting the data that nginx was writing to the socket. The problem was solved by specifyng an alternate location for the file. I am very happy to find that this was not an nginx issue, because I really wanted to use it.

I do have one other question though. I'm trying to find some docs that talk about multiple fcgi application instances and how nginx balances requests between them. Does anyone know where I could find some info on this?

BTW: My fcgi application is written in C++ using the fcgi library. I am also making use of the upload module.

Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4297#msg-4297


Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

merlin corey
Gerry,

Sounds like you want to make use of the upstream module (
http://wiki.nginx.org/NginxHttpUpstreamModule ); something like this
should work:

upstream supercpp {
  server   backend1.example.com    weight=5;
  server   127.0.0.1:8080          max_fails=3  fail_timeout=30s;
  server   unix:/tmp/backend3;
}

Substituing of course your backends and weight settings (if
necessary).  Minimal for you might be like:

upstream supercpp {
  server unix:/tmp/back0;
  server unix:/tmp/back1;
  ...
}

Then later, when you need to pass off to it, you simply refer to the
upstream group by name like so:

fastcgi_pass supercpp;

Thanks,
Merlin

On 7/21/09, gerryw <[hidden email]> wrote:

> Hello All,
>
> I have finally figured out what the problem was. For some reason when the
> local unix: socket was created there was an OS issue that was corrupting the
> data that nginx was writing to the socket. The problem was solved by
> specifyng an alternate location for the file. I am very happy to find that
> this was not an nginx issue, because I really wanted to use it.
>
> I do have one other question though. I'm trying to find some docs that talk
> about multiple fcgi application instances and how nginx balances requests
> between them. Does anyone know where I could find some info on this?
>
> BTW: My fcgi application is written in C++ using the fcgi library. I am also
> making use of the upload module.
>
> Thanks,
> Gerry
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4297#msg-4297
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FastCGI application not receiving multi-part data

vedranf
In reply to this post by vedranf
Hi,

Merlin:
> Sounds like you want to make use of the upstream module...

Thanks! This is exactly what I was looking for.


Thanks,
Gerry

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,4091,4400#msg-4400