Monitoring Splunk

Monitoring Logfiles with random number as part of the name

rvbalaji
Explorer

Our logfiles are named in the format Log.Activity.prod.###.txt where ### is random number. Also we want to leave out previous days log which would be in the format Log.Activity.prod.###.yyyy-mm-dd.txt (using the blacklist -).

We have setup splunk light forwarders and following is what we have on our Inputs.conf file:

[monitor://d:\LogFiles\prod\Log.Activity.prod.*]
blacklist = -
disabled = false
sourcetype = Prod

[monitor://d:\LogFiles\beta\Log.Activity.beta.*]
blacklist = -
disabled = false
sourcetype = Beta

[monitor://d:\LogFiles\alpha\Log.Activity.alpha.*]
blacklist = -
disabled = false
sourcetype = Alpha

But for some reason splunk does not identify the file that is being logged to.

0 Karma
1 Solution

rvbalaji
Explorer

There weren't any issues with my Inputs.conf, but after changing the license on splunk to use the forwarder license, the access to Inputs.conf file was removed for the profile splunk was running under. Resetting the permission on the splunk folder resolved the issue. Thanks.

View solution in original post

0 Karma

rvbalaji
Explorer

There weren't any issues with my Inputs.conf, but after changing the license on splunk to use the forwarder license, the access to Inputs.conf file was removed for the profile splunk was running under. Resetting the permission on the splunk folder resolved the issue. Thanks.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You might have more success with something like this:

[monitor://d:\LogFiles\prod]
whitelist = Log.Activity.[a-zA-Z]+.[0-9]+.txt
disabled = false 
sourcetype = Prod
recursive = false

The whitelist avoids everything but your "current" logfile. (I'm not sure how wildcards in the monitor stanza and whitelist/blacklist interact -- something in the back of my mind says they don't get along, as Splunk internally might be using whitelist/blacklist to implement your wildcards.)

Strictly speaking, you aren't required to not monitor the 'older' files. As long as the first 256 bytes are the same, Splunk should recognize it as a rotated file and not re-index it even if the name changes.

0 Karma

dwaddle
SplunkTrust
SplunkTrust
0 Karma

rvbalaji
Explorer

I have not had luck with the above. But I do see the following error on the splunkd.log
10-27-2010 12:53:49.895 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
10-27-2010 12:53:49.895 INFO TailingProcessor - ...continuing.

I even tried adding "crcsalt = " with no luck.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Note - I only updated one of your inputs.conf stanzas - you should be able to make up the other two based upon it.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You can turn on sourcetype auto classification - see http://www.splunk.com/base/Documentation/latest/Admin/Aboutdefaultfields . But, if you are going to manually specify the sourcetype in an inputs.conf stanza, it can only take on one value per stanza.

0 Karma

rvbalaji
Explorer

Let me try this, but are multiple sourcetype allowed to be defined in the same inputs.conf?

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...