I have 3 types of log file names, ones that simply end with .log.2018
(eg: dc1-sms.log.2018), others end with -error.log.2018
(eg: dc1-sms-error.log.2018), -access.log.2018
(dc1-sms-access.log.2018).
I am trying to match files that end with:
.log.2018
or
-error.log.2018
and not
-access.log.2018
Basically a dc1-sms(!access)*.log.2018
would suffice, but obviously this does not work.
I tried several combinations ofr wildcards and regex, but without success.
I want to be able to do this from the [monitor://<path>]
part of the config because the way our app is built, I only have access to that part of Splunk config.
Try:
*(?<!-access).log.*
This is a regex form that "should" work. I have not tried it, but the syntax in pure PCRE would be:
.*(?<!access)\.log\..*
But Splunk does some manipulation of the string before processing it. That is why something like *.log
works even though it is not a valid regular expression normally. I don't have the same environment that you have to be able to test it out, but it should at least get you looking in the right direction.
Doesn't seem to be working
So you are using only the [monitor:....] line
designation of the files to be monitored, not even the whitelist
(which is available through the "Add Data" link on the main Splunk page)? What I provided above works with the whitelist
. If you have access to the whitelist
, then you should have access to the blacklist
, but if you don't have access to either, then you cannot do what you want because the [monitor:...]
doesn't do regular expressions like the whitelist
and blacklist
. If that is your restriction, then you will have to designate each file individually, or accept that the -access
files will also be indexed.
I did create an environment like you have described and the only way I got it to work was with the whitelist
(I didn't try the blacklist
, but it should work as well).
The inputs.conf monitoring stanza has blacklist
attribute using which you can blacklist a file pattern from being ingested. You can try something like this
inputs.conf on the forwarder
[monitor://<Your log file folder>/*.log.*]
index= yourIndexerHere
sourcetype=yourSourcetypeHere
blacklist = -access\.log\.\d+$
Basically, monitor every file which is in form *.log.*
but exclude any file with it's full path ending in *-access.log.<somenumber>
. See this for more details:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Whitelistorblacklistspecificincomingdata
As mentioned in the OP, I cannot use blacklist, only the monitor path
Hi ionicabalaurul,
did you tried with whitelists and blacklists?
see http://docs.splunk.com/Documentation/Splunk/7.0.2/admin/Inputsconf
Anyway, try something like this
[monitor:///<your_path/*.log.2018]
index = my_index
sourcetype = my_sourcetype
blacklist = .*-access\.log\.2018
Bye.
Giuseppe
As mentioned in the OP, I cannot use blacklist, only the monitor path
Hi ionicabalaurul,
if you cannot use blacklists, the only way is to create more stanzas finding some rules for your files with the extension *.log.2018:
extension *-error.log.2018 hasn't any problem so you can create a monitor stanza with
[monitor:///<your_path>/*-error.log.2018]
instead for the other files you have to find more rules, e,g,:
[monitor:///<your_path>/dc*-*.log.2018]
and
[monitor:///<your_path>/dhcp*-*.log.2018]
and so on
Otherwise you could take all files
[monitor:///<your_path>/*.log.2018]
and then filter them in the Indexers or Heavy Forwarders
see http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad
Bye.
Giuseppe