topic How to monitor multiple source types in same folder in Getting Data In

How to monitor multiple source types in same folder

ilv2splunk — Mon, 28 Sep 2020 11:57:15 GMT

BlackBerry servers have many different .txt log files all created in the one folder.

I have a universal forwarder installed on a win 2k8 server which I have setup the following inputs.conf

[monitor://C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20*]
Sourcetype=BES_Server_Logs

I get errors like the following.

06-18-2012 14:10:33.062 +1000 ERROR TailingProcessor - matching C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20120606\ against ^C:\\Program Files (x86)\\Research In Motion\\BlackBerry Enterprise Server\\Logs\\20[^\\]*\\$

I was hoping to setup multiple monitor stanzas for the different log files to have different sourcetypes. Is this possible?

eg:
[monitor://C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20*\server_name_MAGT_*_001.txt]

Log files are named like this

C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20120101\server_name_MAGT_20120101_001.txt

C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20120101\server_name_ALRT_20120101_001.txt

C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20120101\server_name_BBIM_20120101_001.txt

Should I use props.conf to rename the sourcetype, if so where should the props.conf live and how specific should the regex be for the files?

Thanks

Re: How to monitor multiple source types in same folder

lguinn2 — Mon, 18 Jun 2012 08:38:30 GMT

You could do it like this

[default]
hostname=yourservername

[monitor://C:\\Program Files (x86)\\Research In Motion\\BlackBerry Enterprise Server\\Logs\\...\\*_MAGT_*_001.txt]
sourcetype=BES_magt

[monitor://C:\\Program Files (x86)\\Research In Motion\\BlackBerry Enterprise Server\\Logs\\...\\*_ALRT_*_001.txt]
sourcetype=BES_alrt

    [monitor://C:\\Program Files (x86)\\Research In Motion\\BlackBerry Enterprise Server\\Logs\\...\\*_BBIM_*_001.txt]
    sourcetype=BES_bbim

This would all part of inputs.conf. You could put it under
C:\Program Files\Splunkforwarder\etc\system\local

Re: How to monitor multiple source types in same folder

ilv2splunk — Mon, 18 Jun 2012 22:46:44 GMT

Thats what I thought I could do but when I do that i get the following errors in splunkd.log

06-19-2012 07:41:00.066 +1000 ERROR TailingProcessor - matching C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\20120619\ against ^C:\Program Files (x86)\Research In Motion\BlackBerry Enterprise Server\Logs\.\[^\]MAGT[^\]*_001.txt$

Re: How to monitor multiple source types in same folder

lguinn2 — Wed, 20 Jun 2012 00:16:33 GMT

This looks like a known issue: SPL-47988 " ERROR TailingProcessor - matching X against Y "

It is scheduled to be fixed in 4.3.4

Here is another person with the same question, and a work-around from support:

ERROR - TailingProcessor - matching...

As it turns out, my original answer was correct - if this bug didn't exist...

Re: How to monitor multiple source types in same folder

Dev999 — Mon, 10 Mar 2014 21:00:43 GMT

I am trying to do this with 6.0.1. Just wonder if you get it working. Thanks.

Re: How to monitor multiple source types in same folder

ddrillic — Mon, 07 Mar 2016 15:20:11 GMT

Our expert said -

Yeah this would have to be done on a heavy indexer,
Which is also good for doing the parsing CPU processing on a heavy forwarder instead of the indexer.

We could send this file(s) through syslog (/etc/rsyslog.conf) to heavy forwarder too, then the heavy forwarder would transform the file.

Only thing I would ask if the timestamps are going to be different. That would propose a new problem to solve. Having three different date formats in one file?

Re: How to monitor multiple source types in same folder

ddrillic — Mon, 07 Mar 2016 15:23:17 GMT

Sorry - wrong thread ; -)