topic Indexing a file name of which changes daily in Getting Data In

Indexing a file name of which changes daily

rvbalaji — Wed, 13 Oct 2010 01:55:12 GMT

We need to index a file which has the day's date as part of its name. How to I configure Splunk to read this file? Indexing the folder is an option, but I want to eliminate this option before taking the other route.

Re: Indexing a file name of which changes daily

dwaddle — Wed, 13 Oct 2010 02:23:16 GMT

The easiest approach to this is to make a symlink to the daily file. Tell Splunk to index the symlink, and swap the symlink out daily.

Re: Indexing a file name of which changes daily

muebel — Wed, 13 Oct 2010 02:46:03 GMT

You could monitor such a file by using a wildcard for the variable part of the file. I.E.

[monitor://<path>/daily_file*]

Presuming that the "daily_file" part is static, and what comes after that is the datestamp.

Re: Indexing a file name of which changes daily

jervin — Wed, 13 Oct 2010 20:56:40 GMT

We use blacklists/whitelists for this; for instance:

[monitor:///var/httpd]
_whitelist = (access$|access-)

That will pick up all logs under /var/httpd either ending in access or containing "access-" in the name, such as "access-20101013.log". In general I've found whitelisting to be better than blacklisting or simple wildcarding because new logs won't be picked up without administrator intervention, which helps keep license costs down if someone puts a large debug log into a directory that Splunk is indexing.

--James

Re: Indexing a file name of which changes daily

Lowell — Thu, 14 Oct 2010 05:09:13 GMT

BTW, if you want to consolidate your log file names into a single source name (because it's annoying to see a different log file each and every day), then you may find some helpful resources on this page:

http://answers.splunk.com/questions/3470/consolidate-similarly-named-log-files-into-a-single-source/4420#4420

Re: Indexing a file name of which changes daily

Lowell — Thu, 14 Oct 2010 05:15:30 GMT

Seems like there could be potential timing issues with this. For example, if you switch the symlink too soon then you miss the end of the file, and if you wait too long, then indexing gets delayed. And last time I tried indexing a symlink (managed by cronolog) back in Splunk 3.x days, this caused some indexing issues, if I'm not mistaken. Perhaps this has all be fixed now, but it never seemed worth trying again. That's my 2 cents.

Re: Indexing a file name of which changes daily

dwaddle — Thu, 14 Oct 2010 05:33:23 GMT

That is a good point about the timing - you would have to wait a reasonable time interval (and reasonable depends on how busy your forwarder / indexer is) after the last write to "yesterday's" file. I think the Splunk 3.x issues with this approach have been mostly squashed - while I'm not using this approach myself I know it's been suggested before on #splunk and seemed to worked for others.

Re: Indexing a file name of which changes daily

rvbalaji — Fri, 15 Oct 2010 02:33:50 GMT

Hey Jervin,

I tried whitelisting the file we wanted to monitor and it worked like a beauty. Thanks.

Re: Indexing a file name of which changes daily

rvbalaji — Fri, 15 Oct 2010 02:42:57 GMT

I tried configuring as you have mentioned using the Splunk\Manager\
Data Inputs\File & Directories\Add New and it did not start indexing this file.