Getting Data In

Indexing a file name of which changes daily

rvbalaji
Explorer

We need to index a file which has the day's date as part of its name. How to I configure Splunk to read this file? Indexing the folder is an option, but I want to eliminate this option before taking the other route.

Tags (3)
1 Solution

jervin
Explorer

We use blacklists/whitelists for this; for instance:

[monitor:///var/httpd]
_whitelist = (access$|access-)

That will pick up all logs under /var/httpd either ending in access or containing "access-" in the name, such as "access-20101013.log". In general I've found whitelisting to be better than blacklisting or simple wildcarding because new logs won't be picked up without administrator intervention, which helps keep license costs down if someone puts a large debug log into a directory that Splunk is indexing.

--James

View solution in original post

Lowell
Super Champion

BTW, if you want to consolidate your log file names into a single source name (because it's annoying to see a different log file each and every day), then you may find some helpful resources on this page:

http://answers.splunk.com/questions/3470/consolidate-similarly-named-log-files-into-a-single-source/...

jervin
Explorer

We use blacklists/whitelists for this; for instance:

[monitor:///var/httpd]
_whitelist = (access$|access-)

That will pick up all logs under /var/httpd either ending in access or containing "access-" in the name, such as "access-20101013.log". In general I've found whitelisting to be better than blacklisting or simple wildcarding because new logs won't be picked up without administrator intervention, which helps keep license costs down if someone puts a large debug log into a directory that Splunk is indexing.

--James

rvbalaji
Explorer

Hey Jervin,

I tried whitelisting the file we wanted to monitor and it worked like a beauty. Thanks.

0 Karma

muebel
SplunkTrust
SplunkTrust

You could monitor such a file by using a wildcard for the variable part of the file. I.E.

[monitor://<path>/daily_file*]

Presuming that the "daily_file" part is static, and what comes after that is the datestamp.

rvbalaji
Explorer

I tried configuring as you have mentioned using the Splunk\Manager\
Data Inputs\File & Directories\Add New and it did not start indexing this file.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

The easiest approach to this is to make a symlink to the daily file. Tell Splunk to index the symlink, and swap the symlink out daily.

dwaddle
SplunkTrust
SplunkTrust

That is a good point about the timing - you would have to wait a reasonable time interval (and reasonable depends on how busy your forwarder / indexer is) after the last write to "yesterday's" file. I think the Splunk 3.x issues with this approach have been mostly squashed - while I'm not using this approach myself I know it's been suggested before on #splunk and seemed to worked for others.

0 Karma

Lowell
Super Champion

Seems like there could be potential timing issues with this. For example, if you switch the symlink too soon then you miss the end of the file, and if you wait too long, then indexing gets delayed. And last time I tried indexing a symlink (managed by cronolog) back in Splunk 3.x days, this caused some indexing issues, if I'm not mistaken. Perhaps this has all be fixed now, but it never seemed worth trying again. That's my 2 cents.

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...