topic Re: Question on monitoring file in Getting Data In

Question on monitoring file

darksky21 — Wed, 02 Oct 2013 10:04:17 GMT

Hi i am trying to monitor some file in var/log on ubuntu. There is 4 file (auth.log,auth.log.1,auth.log.2.gz,auth.log.3.gz)

when i tried the code below it work

[monitor:///var/log/auth.log]
sourcetype= authlog
index = test
disabled = 0

but this does not work

[monitor:///var/log/auth.log.1]
sourcetype= authlog
index = test
disabled = 0

Why is that so? is there anything wrong with it?

Re: Question on monitoring file

kristian_kolb — Wed, 02 Oct 2013 10:41:59 GMT

It could be a permissions issue, check splunkd.log. Make sure that the account running splunkd has read access to /var/auth/auth.log.

The .log.n and .log.n.gz files are just rotated versions of the auth.log, so if you don't need to index the old events, you can just stick with monitoring auth.log. When the current auth.log rotates to auth.log.1, you have already indexed all those events, so you do not need to monitor the rotated files explicitly.

EDIT: typo/bad thinking.

/Kristian

Re: Question on monitoring file

sdaniels — Wed, 02 Oct 2013 12:30:55 GMT

The colon is ok, no?

Re: Question on monitoring file

kristian_kolb — Wed, 02 Oct 2013 13:08:18 GMT

Ooops. I thought I saw a semicolon, and something in my thinking process short-circuited. Colon is supposed to be there, definitely...

Edited answer to include some more clever guessing...

Re: Question on monitoring file

darksky21 — Thu, 03 Oct 2013 02:06:20 GMT

Hi thx for the reply. is there anyway to monitor all the different version of auth.log?

Re: Question on monitoring file

pradeepkumarg — Thu, 03 Oct 2013 02:16:21 GMT

You can use auth.log* to monitor all the versions and use blacklist attribute to ignore gz ones

blacklist = (\.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)

Re: Question on monitoring file

darksky21 — Thu, 03 Oct 2013 02:51:14 GMT

Hi gpradeepkuma... thx for the reply. i have tried that but it does not seem to work.It only monitor auth.log for some reason. Maybe splunk does not allow monitoring for those file?

Re: Question on monitoring file

Ayn — Thu, 03 Oct 2013 07:21:09 GMT

If the auth.log.1, auth.log.2, auth.log.3 etc files are just rotated files that have already been indexed, Splunk won't index them a second time. Might that be what's happening in your case?

Re: Question on monitoring file

darksky21 — Tue, 08 Oct 2013 09:30:43 GMT

Ayn is right. In the end i have to write a script to index all the diff auth.log instead of using monitor

Re: Question on monitoring file

Ayn — Tue, 08 Oct 2013 09:32:18 GMT

No, you don't need to do that - there are ways to make Splunk index them anyway, but very often this is NOT what you want, because it will make Splunk read the same data multiple times and I don't see why you would want that.

Re: Question on monitoring file

kristian_kolb — Tue, 08 Oct 2013 10:27:46 GMT

You have been given advice as to why that might be a bad idea. If you are absolutely sure that this is what you want... good luck.