How to route data to certain indexes based on host, sourcetype, and index?

[mahesh_ravji1]

Hi.

I have a requirement to route events to index based on the fields host, sourcetype, and index.

Field host format is dev-customerA, dev-customerB, etc
Field sourcetype is typeA, typeB, etc

The following routing rules are required:
- If event index is NOT 'main' then don't do any routing (i.e. let the event go to the index set in the event).
- Set index to customer part from host field (e.g. customerA, customerB, etc)
- For sourcetype = typeA and typeB, append '-keep' to the index (e.g. the index becomes customerA-keep, customerB-keep, etc)

Examples:
Event1 index=firewall host=dev-customerA sourcetype=ASA. Should not be routed as index does not equal 'main'
Event2 index=main host=dev-customerA sourcetype=ASA. Should be routed to index=customerA
Event3 index=main host=dev-customerA sourcetype=typeA. Should be routed to index=customerA-keep
Event4 index=main host=dev-customerA sourcetype=typeB. Should be routed to index=customerA-keep
Event5 index=main host=dev-customerB sourcetype=ASA. Should be routed to index=customerB
Event6 index=main host=dev-customerB sourcetype=typeA. Should be routed to index=customerB-keep
Event7 index=main host=dev-customerB sourcetype=typeB. Should be routed to index=customerB-keep

Any idea how this can be achieved with props.conf and transforms.conf (or by other means)?

[gyslainlatsa]

hi mahesh_ravji,

This examples routes events of windows_snare_logsource type to the appropriate index based on their log types. "Application"logs will go to an alternate index, while all other log types, such as "Security", will go to the default index.
To make this determination, it uses props.conf to direct events of windows_snare_log source type through the transforms.conf stanza named "AppRedirect", where a regex then looks for the log type, "Application". Any event with a match on "Application"in the appropriate location is routed to the alternate index, "applogindex". All other events go to the default index.

1. Edit props.conf
   Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

   [windows_snare_syslog]
   TRANSFORMS-index = AppRedirect

This directs events of windows_snare_syslog sourcetype to the AppRedirectstanza in transforms.conf

1. Edit transforms.conf
   Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

   [AppRedirect]
   REGEX = MSWinEventLog\s+\d+\s+Application
   DEST_KEY = _MetaData:Index
   FORMAT = applogindex

This stanza processes the events directed here by props.conf. Events that match the regex (because they contain the string "Application" in the specified location) get routed to the alternate index, "applogindex". All other events route as usual to the default index.

try to use this example.

For more informations, follow this link:

http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Setupmultipleindexes

and read paragraph **Route specific events to a different index**

[mahesh_ravji1]

Hi gyslainlatsa,

Please see my comment above...

[somesoni2]

This might help you http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Routeandfilterdatad#Filter_and_route_ev...

[mahesh_ravji1]

Thanks. I've looked at this but still cant figure out how to write the transform so that the REGEX looks at two fields before directing to a new index (i.e. the REGEX needs to look at the current index and host fields). Example if the index is NOT main, dont do a transform. If the index=main then set index to customerX portion from host field.

I need to apply the following transform but only if the current index=main, otherwise dont apply transform.

[override-index-by-host]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(?i)dev-(.+)$
FORMAT = $1
DEST_KEY = _MetaData:Index

[gyslainlatsa]

find the regular expression that returns the events of the index main I call regex1.

next try like this:

1-props.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

 [Event_By_Index]
 TRANSFORMS-index = override-index-by-host

2-transform.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

[override-index-by-host]
SOURCE_KEY = MetaData:Host
REGEX = regex1                            // regular expression that returns the events
FORMAT = name_index               // name of the index that will receive data
DEST_KEY = _MetaData:Index