Getting Data In

How to dynamically route logs uto multiple indexes and sourcetypes based on file path and filename

harshal_chakran
Builder

Hi,
I am working on OS log onboarding data under multiple hostname folders and these hostname folders are located at same file path.
My plan is to dynamically onboard these logs to indexes based on relevant hostname with dynamic sourcetype set based on filename text.

My logs directory structure:
\opt\myAPP\host1\filename_type1.log
\opt\myAPP\host2\filename_type2.log
\opt\myAPP\host3\filename_type3.log

Expected index name from foldername:
indexname_host1
indexname_host2
indexname_host3

Expected sourcetype name from filename :
sourcetype_type1
sourcetype_type2
sourcetype_type3

Following are the configuration am using at inputs.conf , where index=route is just placeholder and no such index is created:
`[monitor:///opt/myAPP/.../*.log]
host_segment = 3
index = route
sourcetype = reroute_1
whitelist = (host1|host4|host5)

[monitor:///opt/myAPP/.../*.log]
host_segment = 3
index = route
sourcetype = reroute_2
whitelist = (host2)

[monitor:///opt/myAPP/.../*.log]
host_segment = 3
index = route
sourcetype = reroute_3
whitelist = (host3)`

At props.conf
`[reroute_1]
TRANSFORMS-sourcetype = overridesourcetype1
TRANSFORMS-index = overrideindex

[reroute_2]
TRANSFORMS-sourcetype = overridesourcetype2
TRANSFORMS-index = overrideindex

[reroute_3]
TRANSFORMS-sourcetype = overridesourcetype3
TRANSFORMS-index = overrideindex

at transforms.conf :
[overridesourcetype1]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::\/opt\/myAPP\/\w+\/filename_(\w+).*
FORMAT = sourcetype::sourcetype_$1

[overridesourcetype2]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::\/opt\/myAPP\/\w+\/filename_(\w+).*
FORMAT = sourcetype::sourcetype_$1

[overridesourcetype3]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::\/opt\/myAPP\/\w+\/filename_(\w+).*
FORMAT = sourcetype::sourcetype_$1

[overrideindex]
SOURCE_KEY = MetaData:Source
REGEX = source::\/opt\/myAPP\/(\w+).*
DEST_KEY = MetaData:Index
FORMAT = index
$1`

However, all the log files are indexed into the index="indexname_host3".

Is there any way to route this as mentioned under 'Expected'.

Kindly help...

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @harshal_chakranarayan,
remember that all the knowledge objects in Splunk are related to sourcetype, so if you have different sourcetypes you cannot use (or it's very difficoult) field extractions, eventtypes, tags, etc...; this means that it isn't a good idea to use different sourcetypes!

At the same time, why do you want to put logs from hosts in different indexes?
Usually indexes are choosen based on retention policies and access right, eventually based on quantity of data (e.g. large data flows aren't stored in indexes together with few data flows), not other.

In other words, Splunk isn't a database and usually logs are stored in indexes which common retention policies and access rights using a limited number of sourcetypes that permits to manage knowledge objects.
Logs are searcheable using all their fields like sourcetype host and others.

Ciao.
Giuseppe

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

You don't need 3 separate monitors here, just create one and check with following configurations.

inputs.conf

[monitor:///opt/myAPP/.../*.log] 
host_segment = 3
index = route 
sourcetype = reroute_1 

props.conf:

[reroute_1]
TRANSFORMS-index_routing = route_to_index
TRANSFORMS-sourcetype_routing = route_to_sourcetype

transforms.conf

[route_to_index]
SOURCE_KEY = MetaData:Host
REGEX = host::(.*)
DEST_KEY = _MetaData:Index
FORMAT = indexname_$1

[route_to_sourcetype]
SOURCE_KEY = MetaData:Host
REGEX = host::(.*)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype_$1
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...