Archive

Can segmentation be done for sourcetypes and indexes?

Motivator

Hello

I have a syslog server which is being used to collect various network oriented data. For example if its a Aruba device then its location is /var/log/splunk/<deviceType>/<deviceName>/<devicename>.log and for Cisco devices its /var/log/splunk/<deviceType>/<deviceName>/<devicename>.log . What I have now is

host_segment=5
sourcetype =<deviceType>

and

 host_segment=5
 sourcetype = <deviceType>.

So if I have data from about 20-30 different devices I need to write the same number of stanza's in inputs.conf. To avoid this can I do segmentation based on sourcetype and host so that I can use just one stanza? Something like

host_segment=5
soucetype_segment=4

Any ideas?

Update: Below config testing is for index along with sourcetype extraction.

transforms.conf

[replace_index_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = index::$1
DEST_KEY = _MetaData:Index



[replace_sourcetype_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

props.conf

[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replace = replace_sourcetype_with_segment_5_from_source

[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replaceindex = replace_index_with_segment_4_from_source

inputs.conf

[monitor:///home/K23780/splunk/.../.../.../*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_5_from_source
#index= replace_index_with_segment_4_from_source
Tags (1)
0 Karma

SplunkTrust
SplunkTrust

You could do something like this (amended to include index override):

inputs.conf

[monitor:///var/log/splunk/*/*/*/*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_4_and_index_with_segment_5_from_source

props.conf

[replace_sourcetype_with_segment_4_and_index_with_segment_5_from_source]
TRANSFORMS-replaceSourcetype = replace_sourcetype_with_segment_4_from_source
TRANSFORMS-replaceIndex = replace_index_with_segment_5_from_source

transforms.conf

[replace_sourcetype_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

[replace_index_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = $1
DEST_KEY = _MetaData:Index

That should take a look at your source's fourth segment and write it into your sourcetype, and write segment 5 into the index.

Motivator

Looks like I have another issue in the same case. Now that the host_segment is extracting the host name, I am trying to modify the host name. As of now we are getting the hostname as host.company.com for all the devices. I am trying to just get the host value. I know I can do just the extract-host in props.conf which goes to the HF(where indexing is happening) but that doesn't seem to work.

My config is

[source::/var/log/splunk////.log]
EXTRACT-host = (?P\d+.\d+.\d+.\d+|([A-Za-z0-9]+))

as I want it to be applied to any source under it. Any idea on where I am missing it

0 Karma

Motivator

Works now. Had to send it to HF and Indexers for it to work. Thanks @martin_mueller for all your help.

0 Karma

Motivator

It still send it to default @martin. Looks like it can't be done for index.

0 Karma

Motivator

testing it. Its the other way around. index as 4,sourcetype as 5 and host as 6

0 Karma

SplunkTrust
SplunkTrust

I've updated my answer to load the sourcetype from segment 4, the index from segment 5, and the host from segment 6.

0 Karma

SplunkTrust
SplunkTrust

Any index you put into the inputs.conf is going to be overwritten by the transforms.conf. Your issue right now appears to be that the transforms.conf stanza isn't being executed.

0 Karma

Motivator

It still fails for the index. Goes into default. Should I mention anything for the index in inputs.conf?

0 Karma

SplunkTrust
SplunkTrust

That updated props.conf defines TRANSFORMS-replace twice for sourcetype replace_sourcetype_with_segment_5_from_source, change one to TRANSFORMS-replaceIndex

0 Karma

Motivator

Updated the original question with the configs.

0 Karma

Motivator

So how does the inputs and props look? I shouldn't mention the index in inputs.conf file ?

0 Karma

SplunkTrust
SplunkTrust

By writing [foo] in props.conf you specify properties for sourcetype "foo". As a result, your TRANSFORMS-replace = ... is only applied to events with a sourcetype of "replace_index_with_segment_4_from_source". That doesn't match your events because their sourcetype is "replace_sourcetype_with_segment_5_from_source"

0 Karma

Motivator

Isn't the replace just a stanza name
Or is it a function which is mentioning the source to extract the values. Both source type and index are from the source. Can you please share any documentation on how the replace function is working.

0 Karma

SplunkTrust
SplunkTrust

Yeah, that's not going to match your incoming events because that's looking for a sourcetype called replace_index_with_segment_4_from_source.

If you always want to grab the index from segment 4 for sourcetype replace_sourcetype_with_segment_5_from_source than you can drop that TRANSFORMS-replace into its transforms.conf stanza.

0 Karma

Motivator

as [replace_index_with_segment_4_from_source]
TRANSFORMS-replace = replace_index_with_segment_4_from_source

0 Karma

SplunkTrust
SplunkTrust

How do you trigger the transforms.conf [replace_index_with_segment_4_from_source] stanza from your props.conf?

0 Karma

Motivator

[monitor:///var/log/splunk/.../.../.../*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_5_from_source
index= replace_index_with_segment_4_from_source

0 Karma

SplunkTrust
SplunkTrust

Do post the corresponding inputs.conf, I suspect you set index=replace... there while your stanza in transforms.conf is looking for sourcetype=...

0 Karma

Motivator

I am doing it the same way as martin mentioned it for the sourcetype. But I get this error

Search peer aabbcc has the following message: received event for unconfigured/disabled/deleted index='replace_index_with_segment_4_from_source' with source='source::/var/log/splunk/test_web/Cisco/.../newtest01.log' host='host::abc' sourcetype='sourcetype::Cisco'

0 Karma

Motivator

@martin_mueller, Can I do the same for the Index too. In case I add index at /var/log/splunk/test-index/.../.../.log as the path name. Would it be possible to even extract the index from it?

[replace_index_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = index::$1
DEST_KEY = _MetaData:Index

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!