topic Differentiating between Sourcetypes in Getting Data In

Differentiating between Sourcetypes

rturk — Mon, 14 Nov 2011 01:14:01 GMT

Greetings Splunkers!

I am currently collecting logs centrally for a content delivery platform for indexing into Splunk.

The vendor in their infinite wisdom has decided that the service_monitor logs should use the same file naming convention, despite the format of the logs differing based on the type of device that is generating it.

Example:

Filename: service_monitor_10.10.18.49_20110920_204501_00363
Originating Device Type: Service Engine
Fields: date time movie-streamer-threshold-exceeded movie-streamer-augment-threshold-exceeded movie-streamer-stopped...

Filename: service_monitor_10.10.18.41_20111026_225501_03295
Originating Device Type: Service Router
Fields: date time sr-cpu-percentage sr-mem(bytes) requests-received http-normal-requests-received...

Keeping in mind that these files are kept in the same directory. It would no doubt be possible to determine the role by the IP address, however this would involve a LARGE inputs.conf with a stanza something like:

[host://<ip_address>_service_monitor' sourcetype=service_monitor_se ...

For every device on the platform.

Is there a way I can differentiate between the two automatically?

Many thanks in advance 🙂

Re: Differentiating between Sourcetypes

Takajian — Mon, 28 Sep 2020 10:05:58 GMT

If you extract each hosts by using a regular expression, does it work for you? The setting will be following in inputs.conf.

[monitor://]
host_regex = $YOUR_REGEX

Re: Differentiating between Sourcetypes

rturk — Mon, 14 Nov 2011 03:48:03 GMT

Thanks Takajian. I can see what you tried to do there, but it's not quite what I was after. Fortunately I have found an swer that does what I need it to do... see below 🙂

Thanks again for your answer!

Re: Differentiating between Sourcetypes

rturk — Mon, 14 Nov 2011 03:54:28 GMT

Playing around has come up with the goods.

With a list of the Service Routers, we were able to come up with the following stanza in our inputs.conf:

[batch:///...path_to_file.../service_monitor_(10.10.10.163_|192.168.159.68_|10.10.10.172_|192.168.159.76_|10.10.0.76_|192.168.159.172_|10.10.10.67_|192.168.159.164_)*]
host_regex = service_monitor_(\d+.\d+.\d+.\d+)_\d+_\d+_\d+
sourcetype = service_monitor_sr
index = service_monitor_sr
crcSalt = <SOURCE>
disabled = false
move_policy = sinkhole

Because configuration files are processed sequentially, having a "catch-all" below it for the other files captures the rest of them as service_monitor_se sourcetypes:

[batch:///...path_to_file.../service_monitor*]
host_regex = service_monitor_(\d+.\d+.\d+.\d+)_\d+_\d+_\d+
sourcetype = cds_service_monitor_engine
index = cds_service_monitor_engine
crcSalt = <SOURCE>
disabled = false
move_policy = sinkhole

I hope this helps someone 🙂