topic Re: too many sources in Getting Data In

too many sources

gnovak — Wed, 23 Feb 2011 06:51:33 GMT

Hi,

I just noticed something today and have an idea of maybe how to fix it but figured i would toss it by here too.

In splunk on 2 servers I have in the inputs.conf the following to be monitored:

[monitor:///opt/drs/var/log]
disabled = false
sourcetype = drs7

the problem i have noticed is that in this directory logs are generated every day with a different name. The date is included in the name so what's happening is in splunk under Sources on the summary page there are tons of entries for each and every one of these logs, instead of 1 source just being /opt/drs/var/log.

2 examples of some of the logs are below:

/opt/drs/var/log/drs.me-20110131_1525.log
/opt/drs/var/log/drs.gi-20110222_1559.log

If I was to change the entry in the inputs.conf to say:

[monitor:///opt/drs/var/log/drs.*.log]
disabled = false
sourcetype = drs7

Would this eliminate every single log being listed as a source?

Re: too many sources

dwest — Wed, 23 Feb 2011 08:07:19 GMT

Not sure if there's some reason not to do this versus ron's answer, but I have a monitor pointing to a directory and I simply have source=sourcename in the stanza in inputs.conf, which causes there to be only one source set.

inputs.conf:

[monitor://my_directory]
source=singlesourcename
sourcetype=mysourcetype

Re: too many sources

Ron_Naken — Wed, 23 Feb 2011 08:14:00 GMT

I removed mine, in lieu of this answer.

Re: too many sources

Lowell — Wed, 23 Feb 2011 23:20:56 GMT

You may also want to take a look a the following question/answer:

Consolidate similarly named log files into a single source

If you want to consolidate all files in a directory as a single source then using the [monitor] stanza that has an explicit source= setting will work, but if you have different types of log files in a single directory, then I recommend using a transformer-based approach (as detailed in the link above).