topic Re: direct files in /var/log/atpco to different indexes and sourcetypes in Splunk Search

direct files in /var/log/atpco to different indexes and sourcetypes

steveirogers — Mon, 28 Sep 2020 10:04:08 GMT

I am running Splunk 4.2.3.

I have a directory called "/var/log/atpco" which contains numerous log files.

I have played with all types of coding for whitelist and blacklist configurations but nothing seems to work.
My goals are as follows:
I would like to direct these specific files to a separate index and sourcetype as index=rules, soucetype=RulesOffline:
fmgpjob01_RulesApplyBC01.log
fmgpjob01_RulesCopyBC01.log
fmgpjob01_RulesQueryBC01.log
fmgpjob01_RulesQueryBC02.log
fmgpjob01_RulesQuickJobsBC01.log
fmgpjob01_RulesQuickJobsBC02.log
fmgpjob01_RulesRBDTableSaveBC01.log

I would like to direct these specific files to a separate index and sourcetype as index=fares, soucetype=FaresOffline:
fmgpjob01_ApplyBC01.log
fmgpjob01_CriteriaSetBC01.log
fmgpjob01_InquiryBC01.log
fmgpjob01_LoadBC01.log
fmgpjob01_QuickJobsBC01.log
fmgpjob01_QuickJobsBC02.log
fmgpjob01_StrikeoverBC01.log
fmgpjob01_ValidationJobsBC01.log
fmgpjob01_ValidationJobsBC02.log
fmgpjob01_ValidationJobsBC03.log

I want to ignore any other files in the /var/log/atpco directory.

Could anyone please provide some guidance on how to accomplish the above? Currently I get nothing with the existing configuration. Do I need to do this in props.conf?

Here is what I have now:

[monitor:///var/log/atpco]
disabled = false
index = rules
sourcetype = RulesOffline
whitelist = fmgpjob01_Rules[^/]*.log$

[monitor:///var/log/atpco]
disabled = false
index = fares
sourcetype = FaresOffline
whitelist = fmgpjob01_Apply*.log$
whitelist = fmgpjob01_CriteriaSet*.log$
whitelist = fmgpjob01_DDSAllAdds*.log$
whitelist = fmgpjob01_Inquiry*.log$
whitelist = fmgpjob01_QuickJobs*.log$
whitelist = fmgpjob01_Strikeover*.log$
whitelist = fmgpjob01_ValidationJobs*.log$

Re: direct files in /var/log/atpco to different indexes and sourcetypes

MuS — Tue, 08 Nov 2011 13:12:31 GMT

Hi steveirogers

with the monitor stanza you have to do this in inputs.conf.

props.conf and transforms.conf can also be used for route and filter into different indexes, but then this would be for any kind of input.

cheers

Re: direct files in /var/log/atpco to different indexes and sourcetypes

steveirogers — Tue, 08 Nov 2011 15:01:36 GMT

Thanks MuS, but I do not quite understand. Should my monitor stanza work as coded?

Re: direct files in /var/log/atpco to different indexes and sourcetypes

MuS — Tue, 08 Nov 2011 15:11:42 GMT

basically: yes. you can run 'splunk cmd btool --debug inputs list monitor' to see what your monitor stanzas will look at the end. I think the problem is that you have two monitor stanzas for the same directory.

Re: direct files in /var/log/atpco to different indexes and sourcetypes

steveirogers — Wed, 09 Nov 2011 18:44:53 GMT

Thanks. I will do some more research on overlapping monitor stanza's then.