topic inputs.conf not picking up monitored file in Getting Data In

inputs.conf not picking up monitored file

MasterOogway — Mon, 28 Sep 2020 09:32:36 GMT

I have what appears to be a simple monitor to watch for a specific file name with a regex to define the date stamped file.
The file in question is named, /log/blahblah_0.0.0.653_9110.log

On my LWF I have the following simple inputs.conf definition:

[monitor:///log/blahblah_.*\.log]
disabled=false
sourcetype=search-log

From ../splunkd.log I get the following error.
INFO TailingProcessor - No configurations match, will ignore path='/log/blahblah_0.0.0.653_9110.log'

I have also defined a monitor based on a whitelist with the same result.

[monitor:///log]
disabled=false
whitelist = blahblah_.*.log$
sourcetype=search-log

My question is, "why does Splunk not want to index this file? I have confirmed the Regex is defined correctly, so I believe that is not it. Most likely it is something VERY simple where I can't see the forest through the trees.

Thoughts?
MasterOogway

Re: inputs.conf not picking up monitored file

fox — Tue, 10 May 2011 09:02:58 GMT

Is it possible to open up your filter or are there lots of files in this directory? I preferred your second approach, but with a more simple regex like .*log for example. You could also try the line in the input file: "crcSalt = <SOURCE>"

Re: inputs.conf not picking up monitored file

MasterOogway — Tue, 10 May 2011 11:56:47 GMT

I tried it with and without the crcSalt= line without luck. Short of naming the file explicitly I have tried just about every combination. Still looking for the "aha" answer.

Re: inputs.conf not picking up monitored file

brantramey — Fri, 05 Aug 2011 18:46:45 GMT

We are having the same issue here. Many files are not being indexed. We have tried unsuccessfully to add the "crcSalt = ". Does anyone have any other solutions to this?

Re: inputs.conf not picking up monitored file

mfrost8 — Mon, 28 Sep 2020 09:49:16 GMT

I'm not sure if I have what you're looking for, but here's some more info that might be helpful.

First, I don't believe that the crcSalt line is going to help here. My understanding is that that just helps Splunk understand when it might or might not be looking at the same file (or the events in a file) that it's indexed previously. I don't think that has anything to do with adding the file to the list of files to monitor.

So what did "splunk list monitor" report? Did it show that at least /log was being monitored but the blahblah*.log file was not being monitored?

One important thing to note is that the line

[monitor:///log/blahblah_.*\.log]

is probably not doing what you think it is. Remember that in this context, there are some regex shortcuts that are being taken. If your file really is named

blahblah_0.0.0.653_9110.log

then

[monitor:///log/blahblah_.*.log]

(i.e. no escaped '.' since '.' is the literal period character) should work. Remember that this is creating an implicit whitelist. I'm actually not sure what the '.' would do, but I'd guess that it might actually look for a backslash followed by a dot.

I would have thought your whitelist version might have worked assuming you were using a version of newer version of Splunk where "whitelist" replaced the older "_whitelist" keyword.

Lowell and gkanapathy were kind enough to write up some nice details to a question I had a while back about the use of wildcards and regexes in inputs.conf at

http://splunk-base.splunk.com/answers/2775/regexs-and-windows-paths-in-inputsconf-and-propsconf

in case that might also provide some further clues.