nginx access logs to mysql

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

nginx access logs to mysql

SARA QUISPE MEJIA

I want to parse the log file respect to a client that means to make a report of how the client is using my application through the information provided by a log file.

So I need to filter some url where I find an especific  name of a function. For that I thought in insert my log file to database like mysql. Could I do that with my log file ?  (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx access logs to mysql

oscaretu .
Sara, why don't you process the log file just with grep / pcregrep to get just the lines that containt that function name?


On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[hidden email]> wrote:

I want to parse the log file respect to a client that means to make a report of how the client is using my application through the information provided by a log file.

So I need to filter some url where I find an especific  name of a function. For that I thought in insert my log file to database like mysql. Could I do that with my log file ?  (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx access logs to mysql

SARA QUISPE MEJIA
yes, I tried to do that and then I did the analysis with goaccess, 
but now I need to do an interface(page.php) for the user can choose
the function that want to filter and then generate a report.
So, I thought put all my information on a database like mysql

2017-04-20 9:11 GMT+02:00 oscaretu . <[hidden email]>:
Sara, why don't you process the log file just with grep / pcregrep to get just the lines that containt that function name?


On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[hidden email]> wrote:

I want to parse the log file respect to a client that means to make a report of how the client is using my application through the information provided by a log file.

So I need to filter some url where I find an especific  name of a function. For that I thought in insert my log file to database like mysql. Could I do that with my log file ?  (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx access logs to mysql

oscaretu .
If you extract the lines containing all the function names, and filter them with a script that counts the number of times that the function appears, you only have to store (for example in a JSON) the number of times that each function was used. And you can avoid storing all the logs in a database.


On Thu, Apr 20, 2017 at 9:29 AM, SARA QUISPE MEJIA <[hidden email]> wrote:
yes, I tried to do that and then I did the analysis with goaccess, 
but now I need to do an interface(page.php) for the user can choose
the function that want to filter and then generate a report.
So, I thought put all my information on a database like mysql

2017-04-20 9:11 GMT+02:00 oscaretu . <[hidden email]>:
Sara, why don't you process the log file just with grep / pcregrep to get just the lines that containt that function name?


On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[hidden email]> wrote:

I want to parse the log file respect to a client that means to make a report of how the client is using my application through the information provided by a log file.

So I need to filter some url where I find an especific  name of a function. For that I thought in insert my log file to database like mysql. Could I do that with my log file ?  (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx access logs to mysql

SARA QUISPE MEJIA
yes, I could do it but I want just store for example the customer, the function name, the date after with those information I could filter my report through a page.php
But  with json I don't know how can I do that?


2017-04-20 9:57 GMT+02:00 oscaretu . <[hidden email]>:
If you extract the lines containing all the function names, and filter them with a script that counts the number of times that the function appears, you only have to store (for example in a JSON) the number of times that each function was used. And you can avoid storing all the logs in a database.


On Thu, Apr 20, 2017 at 9:29 AM, SARA QUISPE MEJIA <[hidden email]> wrote:
yes, I tried to do that and then I did the analysis with goaccess, 
but now I need to do an interface(page.php) for the user can choose
the function that want to filter and then generate a report.
So, I thought put all my information on a database like mysql

2017-04-20 9:11 GMT+02:00 oscaretu . <[hidden email]>:
Sara, why don't you process the log file just with grep / pcregrep to get just the lines that containt that function name?


On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[hidden email]> wrote:

I want to parse the log file respect to a client that means to make a report of how the client is using my application through the information provided by a log file.

So I need to filter some url where I find an especific  name of a function. For that I thought in insert my log file to database like mysql. Could I do that with my log file ?  (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx access logs to mysql

Reinis Rozitis
In reply to this post by SARA QUISPE MEJIA
> I tried to do that with syslog-ng but it doesn't work

What exactly didn't work (before writing a longer configuration example)?

It's kind of simple with syslog-ng - you set the nginx log file as source /
provide mysql as destination and that's it.


rr

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx access logs to mysql

oscaretu .
In reply to this post by SARA QUISPE MEJIA
I was trying to say you that you don't need to store lots of unnecesary logs when you can store just the valuable information contained in the logs, in a processed way.

Why store thousands of log lines that show some called function functionname1 instead of store something like this:


{
  "date" : "20170420",
  "functionmname1": 33,
  "functionmname2": 18,
  "functionmname3": 51,
  "functionmname4": 68,
  ...
}

In JSON, in a database, or where you want...


On Thu, Apr 20, 2017 at 10:10 AM, SARA QUISPE MEJIA <[hidden email]> wrote:
yes, I could do it but I want just store for example the customer, the function name, the date after with those information I could filter my report through a page.php
But  with json I don't know how can I do that?


2017-04-20 9:57 GMT+02:00 oscaretu . <[hidden email]>:
If you extract the lines containing all the function names, and filter them with a script that counts the number of times that the function appears, you only have to store (for example in a JSON) the number of times that each function was used. And you can avoid storing all the logs in a database.


On Thu, Apr 20, 2017 at 9:29 AM, SARA QUISPE MEJIA <[hidden email]> wrote:
yes, I tried to do that and then I did the analysis with goaccess, 
but now I need to do an interface(page.php) for the user can choose
the function that want to filter and then generate a report.
So, I thought put all my information on a database like mysql

2017-04-20 9:11 GMT+02:00 oscaretu . <[hidden email]>:
Sara, why don't you process the log file just with grep / pcregrep to get just the lines that containt that function name?


On Thu, Apr 20, 2017 at 9:05 AM, SARA QUISPE MEJIA <[hidden email]> wrote:

I want to parse the log file respect to a client that means to make a report of how the client is using my application through the information provided by a log file.

So I need to filter some url where I find an especific  name of a function. For that I thought in insert my log file to database like mysql. Could I do that with my log file ?  (I use nginx )

I tried to do that with syslog-ng but it doesn't work

Do you have any ideas?

Thanks

Sara


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx