topic Re: How to route data to certain indexes based on host, sourcetype, and index? in Getting Data In

How to route data to certain indexes based on host, sourcetype, and index?

mahesh_ravji1 — Wed, 24 Feb 2016 09:48:51 GMT

Hi.

I have a requirement to route events to index based on the fields host, sourcetype, and index.

Field host format is dev-customerA, dev-customerB, etc
Field sourcetype is typeA, typeB, etc

The following routing rules are required:
- If event index is NOT 'main' then don't do any routing (i.e. let the event go to the index set in the event).
- Set index to customer part from host field (e.g. customerA, customerB, etc)
- For sourcetype = typeA and typeB, append '-keep' to the index (e.g. the index becomes customerA-keep, customerB-keep, etc)

Examples:
Event1 index=firewall host=dev-customerA sourcetype=ASA. Should not be routed as index does not equal 'main'
Event2 index=main host=dev-customerA sourcetype=ASA. Should be routed to index=customerA
Event3 index=main host=dev-customerA sourcetype=typeA. Should be routed to index=customerA-keep
Event4 index=main host=dev-customerA sourcetype=typeB. Should be routed to index=customerA-keep
Event5 index=main host=dev-customerB sourcetype=ASA. Should be routed to index=customerB
Event6 index=main host=dev-customerB sourcetype=typeA. Should be routed to index=customerB-keep
Event7 index=main host=dev-customerB sourcetype=typeB. Should be routed to index=customerB-keep

Any idea how this can be achieved with props.conf and transforms.conf (or by other means)?

Re: How to route data to certain indexes based on host, sourcetype, and index?

somesoni2 — Wed, 24 Feb 2016 18:37:01 GMT

This might help you http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Routeandfilterdatad#Filter_and_route_event_data_to_target_groups

Re: How to route data to certain indexes based on host, sourcetype, and index?

gyslainlatsa — Tue, 29 Sep 2020 08:55:36 GMT

hi mahesh_ravji,

This examples routes events of windows_snare_logsource type to the appropriate index based on their log types. "Application"logs will go to an alternate index, while all other log types, such as "Security", will go to the default index.
To make this determination, it uses props.conf to direct events of windows_snare_log source type through the transforms.conf stanza named "AppRedirect", where a regex then looks for the log type, "Application". Any event with a match on "Application"in the appropriate location is routed to the alternate index, "applogindex". All other events go to the default index.

Edit props.conf
Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

[windows_snare_syslog]
TRANSFORMS-index = AppRedirect

This directs events of windows_snare_syslog sourcetype to the AppRedirectstanza in transforms.conf

Edit transforms.conf
Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

[AppRedirect]
REGEX = MSWinEventLog\s+\d+\s+Application
DEST_KEY = _MetaData:Index
FORMAT = applogindex

This stanza processes the events directed here by props.conf. Events that match the regex (because they contain the string "Application" in the specified location) get routed to the alternate index, "applogindex". All other events route as usual to the default index.

try to use this example.

For more informations, follow this link:

http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Setupmultipleindexes

and read paragraph **Route specific events to a different index**

Re: How to route data to certain indexes based on host, sourcetype, and index?

mahesh_ravji1 — Tue, 29 Sep 2020 08:55:52 GMT

Thanks. I've looked at this but still cant figure out how to write the transform so that the REGEX looks at two fields before directing to a new index (i.e. the REGEX needs to look at the current index and host fields). Example if the index is NOT main, dont do a transform. If the index=main then set index to customerX portion from host field.

I need to apply the following transform but only if the current index=main, otherwise dont apply transform.

[override-index-by-host]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(?i)dev-(.+)$
FORMAT = $1
DEST_KEY = _MetaData:Index

Re: How to route data to certain indexes based on host, sourcetype, and index?

mahesh_ravji1 — Thu, 25 Feb 2016 11:35:34 GMT

Hi gyslainlatsa,

Please see my comment above...

Re: How to route data to certain indexes based on host, sourcetype, and index?

gyslainlatsa — Thu, 25 Feb 2016 12:09:54 GMT

find the regular expression that returns the events of the index main I call regex1.

next try like this:

1-props.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

 [Event_By_Index]
 TRANSFORMS-index = override-index-by-host

2-transform.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

[override-index-by-host]
SOURCE_KEY = MetaData:Host
REGEX = regex1                            // regular expression that returns the events
FORMAT = name_index               // name of the index that will receive data
DEST_KEY = _MetaData:Index