topic How to dynamically route logs uto multiple indexes and sourcetypes based on file path and filename in Getting Data In

How to dynamically route logs uto multiple indexes and sourcetypes based on file path and filename

harshal_chakran — Wed, 30 Sep 2020 03:13:31 GMT

Hi,
I am working on OS log onboarding data under multiple hostname folders and these hostname folders are located at same file path.
My plan is to dynamically onboard these logs to indexes based on relevant hostname with dynamic sourcetype set based on filename text.

My logs directory structure:
\opt\myAPP\host1\filename_type1.log
\opt\myAPP\host2\filename_type2.log
\opt\myAPP\host3\filename_type3.log

Expected index name from foldername:
indexname_host1
indexname_host2
indexname_host3

Expected sourcetype name from filename :
sourcetype_type1
sourcetype_type2
sourcetype_type3

Following are the configuration am using at inputs.conf , where index=route is just placeholder and no such index is created:
`[monitor:///opt/myAPP/.../*.log]
host_segment = 3
index = route
sourcetype = reroute_1
whitelist = (host1|host4|host5)

[monitor:///opt/myAPP/.../*.log]
host_segment = 3
index = route
sourcetype = reroute_2
whitelist = (host2)

[monitor:///opt/myAPP/.../*.log]
host_segment = 3
index = route
sourcetype = reroute_3
whitelist = (host3)`

At props.conf
`[reroute_1]
TRANSFORMS-sourcetype = overridesourcetype1
TRANSFORMS-index = overrideindex

[reroute_2]
TRANSFORMS-sourcetype = overridesourcetype2
TRANSFORMS-index = overrideindex

[reroute_3]
TRANSFORMS-sourcetype = overridesourcetype3
TRANSFORMS-index = overrideindex
at transforms.conf :[overridesourcetype1]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::\/opt\/myAPP\/\w+\/filename_(\w+).*
FORMAT = sourcetype::sourcetype_$1

[overridesourcetype2]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::\/opt\/myAPP\/\w+\/filename_(\w+).*
FORMAT = sourcetype::sourcetype_$1

[overridesourcetype3]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::\/opt\/myAPP\/\w+\/filename_(\w+).*
FORMAT = sourcetype::sourcetype_$1

[overrideindex]
SOURCE_KEY = MetaData:Source
REGEX = source::\/opt\/myAPP\/(\w+).*
DEST_KEY = MetaData:Index
FORMAT = index$1`

However, all the log files are indexed into the index="indexname_host3".

Is there any way to route this as mentioned under 'Expected'.

Kindly help...

Re: How to dynamically route logs uto multiple indexes and sourcetypes based on file path and filename

manjunathmeti — Tue, 03 Dec 2019 06:42:12 GMT

You don't need 3 separate monitors here, just create one and check with following configurations.

inputs.conf

[monitor:///opt/myAPP/.../*.log] 
host_segment = 3
index = route 
sourcetype = reroute_1

props.conf:

[reroute_1]
TRANSFORMS-index_routing = route_to_index
TRANSFORMS-sourcetype_routing = route_to_sourcetype

transforms.conf

[route_to_index]
SOURCE_KEY = MetaData:Host
REGEX = host::(.*)
DEST_KEY = _MetaData:Index
FORMAT = indexname_$1

[route_to_sourcetype]
SOURCE_KEY = MetaData:Host
REGEX = host::(.*)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype_$1

Re: How to dynamically route logs uto multiple indexes and sourcetypes based on file path and filename

gcusello — Tue, 03 Dec 2019 08:33:33 GMT

Hi @harshal_chakranarayan,
remember that all the knowledge objects in Splunk are related to sourcetype, so if you have different sourcetypes you cannot use (or it's very difficoult) field extractions, eventtypes, tags, etc...; this means that it isn't a good idea to use different sourcetypes!

At the same time, why do you want to put logs from hosts in different indexes?
Usually indexes are choosen based on retention policies and access right, eventually based on quantity of data (e.g. large data flows aren't stored in indexes together with few data flows), not other.

In other words, Splunk isn't a database and usually logs are stored in indexes which common retention policies and access rights using a limited number of sourcetypes that permits to manage knowledge objects.
Logs are searcheable using all their fields like sourcetype host and others.

Ciao.
Giuseppe