topic Re: Can segmentation be done for sourcetypes and indexes? in Splunk Search

Can segmentation be done for sourcetypes and indexes?

theouhuios — Fri, 28 Feb 2014 15:23:01 GMT

Hello

I have a syslog server which is being used to collect various network oriented data. For example if its a Aruba device then its location is /var/log/splunk/<deviceType>/<deviceName>/<devicename>.log and for Cisco devices its /var/log/splunk/<deviceType>/<deviceName>/<devicename>.log . What I have now is

host_segment=5
sourcetype =<deviceType>

and

 host_segment=5
 sourcetype = <deviceType>.

So if I have data from about 20-30 different devices I need to write the same number of stanza's in inputs.conf. To avoid this can I do segmentation based on sourcetype and host so that I can use just one stanza? Something like

host_segment=5
soucetype_segment=4

Any ideas?

Update: Below config testing is for index along with sourcetype extraction.

transforms.conf

[replace_index_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = index::$1
DEST_KEY = _MetaData:Index



[replace_sourcetype_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

props.conf

[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replace = replace_sourcetype_with_segment_5_from_source

[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replaceindex = replace_index_with_segment_4_from_source

inputs.conf

[monitor:///home/K23780/splunk/.../.../.../*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_5_from_source
#index= replace_index_with_segment_4_from_source

Re: Can segmentation be done for sourcetypes and indexes?

somesoni2 — Fri, 28 Feb 2014 16:07:22 GMT

How about using following (one input stanza for each sourcetype)

[monitor:///var/log/splunk/Aruba/*/*.log]
disabled = false
followTail = 0
host_segment=5
sourcetype = Aruba

[monitor:///var/log/splunk/Cisco/*/*.log]
disabled = false
followTail = 0
host_segment=5
sourcetype = Cisco

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Mon, 28 Sep 2020 16:00:53 GMT

I am doing the same as of now. Its just that I have about 35 stanzas with more to come. So I will have to keep updating inputs.conf file for each device type. But it I can do segmenting for host and sourcetype then I can set something like

[monitor:///var/log/splunk/.../.../.log]
disabled = false
followTail = 0
host_segment=5
sourcetype_segment = 4
and that will be the only stanza I would need .

Re: Can segmentation be done for sourcetypes and indexes?

somesoni2 — Fri, 28 Feb 2014 17:06:44 GMT

The comment window ate up the '*' I put earliest. Updated it.

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Fri, 28 Feb 2014 18:48:23 GMT

You could do something like this (amended to include index override):

inputs.conf

[monitor:///var/log/splunk/*/*/*/*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_4_and_index_with_segment_5_from_source

props.conf

[replace_sourcetype_with_segment_4_and_index_with_segment_5_from_source]
TRANSFORMS-replaceSourcetype = replace_sourcetype_with_segment_4_from_source
TRANSFORMS-replaceIndex = replace_index_with_segment_5_from_source

transforms.conf

[replace_sourcetype_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[replace_index_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = $1
DEST_KEY = _MetaData:Index

That should take a look at your source's fourth segment and write it into your sourcetype, and write segment 5 into the index.

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Fri, 28 Feb 2014 19:15:06 GMT

Works very well. Thanks.

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Mon, 28 Sep 2020 16:02:14 GMT

@martin_mueller, Can I do the same for the Index too. In case I add index at /var/log/splunk/test-index/.../.../.log as the path name. Would it be possible to even extract the index from it?

[replace_index_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = index::$1
DEST_KEY = _MetaData:Index

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Mon, 28 Sep 2020 16:02:17 GMT

I am doing it the same way as martin mentioned it for the sourcetype. But I get this error

Search peer aabbcc has the following message: received event for unconfigured/disabled/deleted index='replace_index_with_segment_4_from_source' with source='source::/var/log/splunk/test_web/Cisco/.../newtest01.log' host='host::abc' sourcetype='sourcetype::Cisco'

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Tue, 04 Mar 2014 20:21:44 GMT

Do post the corresponding inputs.conf, I suspect you set index=replace... there while your stanza in transforms.conf is looking for sourcetype=...

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Mon, 28 Sep 2020 16:02:19 GMT

[monitor:///var/log/splunk/.../.../.../*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_5_from_source
index= replace_index_with_segment_4_from_source

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Tue, 04 Mar 2014 21:03:29 GMT

How do you trigger the transforms.conf [replace_index_with_segment_4_from_source] stanza from your props.conf?

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Mon, 28 Sep 2020 16:02:28 GMT

as [replace_index_with_segment_4_from_source]
TRANSFORMS-replace = replace_index_with_segment_4_from_source

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Mon, 28 Sep 2020 16:02:39 GMT

Yeah, that's not going to match your incoming events because that's looking for a sourcetype called replace_index_with_segment_4_from_source.

If you always want to grab the index from segment 4 for sourcetype replace_sourcetype_with_segment_5_from_source than you can drop that TRANSFORMS-replace into its transforms.conf stanza.

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Wed, 05 Mar 2014 12:18:02 GMT

Isn't the replace just a stanza name
Or is it a function which is mentioning the source to extract the values. Both source type and index are from the source. Can you please share any documentation on how the replace function is working.

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Mon, 28 Sep 2020 16:02:47 GMT

By writing [foo] in props.conf you specify properties for sourcetype "foo". As a result, your TRANSFORMS-replace = ... is only applied to events with a sourcetype of "replace_index_with_segment_4_from_source". That doesn't match your events because their sourcetype is "replace_sourcetype_with_segment_5_from_source"

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Wed, 05 Mar 2014 13:42:24 GMT

So how does the inputs and props look? I shouldn't mention the index in inputs.conf file ?

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Wed, 05 Mar 2014 13:49:24 GMT

Updated the original question with the configs.

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Wed, 05 Mar 2014 14:30:50 GMT

That updated props.conf defines TRANSFORMS-replace twice for sourcetype replace_sourcetype_with_segment_5_from_source, change one to TRANSFORMS-replaceIndex

Re: Can segmentation be done for sourcetypes and indexes?

theouhuios — Wed, 05 Mar 2014 14:39:14 GMT

It still fails for the index. Goes into default. Should I mention anything for the index in inputs.conf?

Re: Can segmentation be done for sourcetypes and indexes?

martin_mueller — Wed, 05 Mar 2014 15:01:15 GMT

Any index you put into the inputs.conf is going to be overwritten by the transforms.conf. Your issue right now appears to be that the transforms.conf stanza isn't being executed.