Inputs.conf: Why is my file not getting picked up with my monitor configuration?

[a212830]

Hi,

I have a file in the format /apps/logs/YYYY/MM/DD/system-hostname.log - so, /apps/logs/2014/06/30/system-pf-us123-mgmt.log

My inputs is the following:

[monitor:///apps/logs////system-pf.log]
recursive = Yes
index=network
sourcetype = netscreen_syslog
followTail = 0
disabled = 0

The file is not getting picked up. Did I do something wrong?

[kfeagans_splunk]

Assuming you are trying to regex the date, or the current date of the day (generally, regex does not have access to system time/date however)? I don't see anything that will match the file to the monitor statement either. You could also break this into a monitor statement at the high level (/apps/logs) then dive into the regex with individual whitelist(s).

Perhaps something like (should match a valid date until 2099) :

[monitor:///apps/logs/(19|20)\d\d/(0[1-9]|1[012])/(0[1-9]|[12][0-9]|3[01])/system-\w+-\w+-\w+.log$]
recursive = Yes
index=network
sourcetype = netscreen_syslog
followTail = 0
disabled = 0

--

Kelly

[a212830]

/apps/logs/2014/06/30/system-pf-us123-mgmt.log
/apps/logs/2014/06/30/system-psh-us123.log
/apps/logs/2014/06/30/system-ive123.log
/apps/logs/2014/06/30/system-pf-us299-mgmt.log

and so on, and so on, and so on....

The directory structure will always be the same, and the log file will always begin with system- and end with .log - other than that, it's beyond my control, but it's usually a combination of characters and numbers - sometimes there's an underscore, sometimes a dash...

[kfeagans_splunk]

What does the directory structure look like? What do the system-pf-hosts files inside the dir look like? I was assuming that the files start with "system" then have some series of characters separated by hyphens. Is that not the case? Hence my pattern "system-w+-w+-w+.log$" ... which is system, hyphen, some characters, hyphen, characters, hyphen, charactes, ending with .log ...

In your example above, will "system-pf.log" ever match anything?

Kelly

[a212830]

I want anything under /apps/logs - I control that filesystem, so it shouldn't be a problem. I can't have the system-w+, as that will grab any word, and I have lots of different files in this structure. The pf hosts match one sourcetype, and another name would match a different sourcetype.

[kfeagans_splunk]

So you don't want to validate the date? Careful with '*' ... can be very greedy, and can interfere with regex processing depending on the location. Take a look here: http://docs.splunk.com/Documentation/Splunk/6.1.1/Data/Specifyinputpathswithwildcards

Generally, it's always better to use regex rather than wildcard. With regex you can be much more targeted in what you are after.

Kelly

[a212830]

The asterisk should handle the date - I have plenty of these setup already, and they get picked up.

[bmunson_splunk]

I would have used

[monitor:///apps/logs/.../system-*.log]
host_regex = /system-([^/]).log$
recursive  = Yes
index=network
sourcetype = netscreen_syslog
followTail = 0
disabled = 0

[a212830]

can't be system-*.log, as there are many other inputs that follow the same type of format, based upon the hostname.

[ppablo]

yes you are totally correct. Sorry, I've been looking at too many forward and backslashes. Getting cross eyed 😛

[a212830]

no - should be 3 /, no? two for the stanza, one for the actual filesystem.

[ppablo]

Hi @a212830

Do you have one extra / before "apps"?