I have 3 types of log file names, ones that simply end with
.log.2018 (eg: dc1-sms.log.2018), others end with
-error.log.2018 (eg: dc1-sms-error.log.2018),
I am trying to match files that end with:
dc1-sms(!access)*.log.2018 would suffice, but obviously this does not work.
I tried several combinations ofr wildcards and regex, but without success.
I want to be able to do this from the
[monitor://<path>] part of the config because the way our app is built, I only have access to that part of Splunk config.
This is a regex form that "should" work. I have not tried it, but the syntax in pure PCRE would be:
But Splunk does some manipulation of the string before processing it. That is why something like
*.log works even though it is not a valid regular expression normally. I don't have the same environment that you have to be able to test it out, but it should at least get you looking in the right direction.
So you are using only the
[monitor:....] line designation of the files to be monitored, not even the
whitelist (which is available through the "Add Data" link on the main Splunk page)? What I provided above works with the
whitelist. If you have access to the
whitelist, then you should have access to the
blacklist, but if you don't have access to either, then you cannot do what you want because the
[monitor:...] doesn't do regular expressions like the
blacklist. If that is your restriction, then you will have to designate each file individually, or accept that the
-access files will also be indexed.
I did create an environment like you have described and the only way I got it to work was with the
whitelist (I didn't try the
blacklist, but it should work as well).
The inputs.conf monitoring stanza has
blacklist attribute using which you can blacklist a file pattern from being ingested. You can try something like this
inputs.conf on the forwarder
[monitor://<Your log file folder>/*.log.*] index= yourIndexerHere sourcetype=yourSourcetypeHere blacklist = -access\.log\.\d+$
Basically, monitor every file which is in form
*.log.* but exclude any file with it's full path ending in
*-access.log.<somenumber>. See this for more details:
did you tried with whitelists and blacklists?
Anyway, try something like this
[monitor:///<your_path/*.log.2018] index = my_index sourcetype = my_sourcetype blacklist = .*-access\.log\.2018
if you cannot use blacklists, the only way is to create more stanzas finding some rules for your files with the extension *.log.2018:
extension *-error.log.2018 hasn't any problem so you can create a monitor stanza with
instead for the other files you have to find more rules, e,g,:
and so on
Otherwise you could take all files
and then filter them in the Indexers or Heavy Forwarders