topic Re: How do I route data to specific index based on a field? in Splunk Enterprise

How do I route data to specific index based on a field?

melonman — Mon, 18 Jun 2012 14:28:54 GMT

Hi,

I would like to know how to route data to a specific index based on a value in a field.

I have a series of data that look like this:

2012/06/07 10:45:50 service=srvc1 server=node3 score=50 seq=55041
2012/06/07 10:45:50 service=srvc3 server=node1 score=17 seq=55042
2012/06/07 10:45:50 service=srvc2 server=node1 score=67 seq=55043
2012/06/07 10:45:50 service=srvc2 server=node4 score=43 seq=55044
2012/06/07 10:45:50 service=srvc3 server=node2 score=11 seq=55045
2012/06/07 10:45:50 service=srvc3 server=node2 score=60 seq=55046
2012/06/07 10:45:50 service=srvc1 server=node0 score=28 seq=55047
2012/06/07 10:45:50 service=srvc1 server=node0 score=4 seq=55048

Then, I want to route date to srvc1, srvc2 or srvc3 depending on the value in service field.
I found several answers that explains how to route data based on host or source(IP), but I could not find an answer for my questions.

I really appreciate any comment on this...

Thank you,

(JA) イベントの任意のフィールドの値に基づいて保存するIndexを変えるにはどうするのか。

Re: How do I route data to specific index based on a field?

jeff — Mon, 18 Jun 2012 15:08:49 GMT

You need to use a transforms as follows:

In props.conf

[host::(host1|host2|host3)]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

and/or

[some_sourcetype]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

In transforms.conf

[index_srvc1]
REGEX    = .*service=srvc1.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc1

[index_srvc2]
REGEX    = .*service=srvc2.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc2

[index_srvc3]
REGEX    = .*service=srvc3.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc3

In my case I needed to base on host from a centralized rsyslog server. YMMV... you may be able to use a replacement expression instead of a separate transform for each index, but I'll leave that as an exercise for you.

Re: How do I route data to specific index based on a field?

melonman — Tue, 19 Jun 2012 03:02:22 GMT

Thanks, I could make my config work.

Re: How do I route data to specific index based on a field?

melonman — Tue, 19 Jun 2012 03:03:34 GMT

Hi jeff,

I could get it work with the following config.

-- props.conf

[sample1]
TRANSFORMS-index_routing = route_data_to_index_by_field_service

-- transforms.conf

[route_data_to_index_by_field_service]
REGEX = .*service=(.*?)[ ]
DEST_KEY = _MetaData:Index
FORMAT = $1

-- Result

$ ./splunk search 'index=* sourcetype=sample1 | head limit=10 | table index, service, server'
index service server
----- ------- ------
srvc2 srvc2   node1
srvc2 srvc2   node0
srvc3 srvc3   node1
srvc2 srvc2   node4
srvc3 srvc3   node0
srvc2 srvc2   node4
srvc2 srvc2   node0
srvc1 srvc1   node4
srvc2 srvc2   node1
srvc1 srvc1   node0

now I can move forward to configure RBAC thing... thanks!

Re: How do I route data to specific index based on a field?

jeff — Wed, 20 Jun 2012 02:03:26 GMT

cool... glad I could help. Thanks for posting your regex- may have a use for that later.

Re: How do I route data to specific index based on a field?

andrey2007 — Tue, 10 Mar 2015 15:16:38 GMT

Ir really works, thanks!

Re: How do I route data to specific index based on a field?

andrey2007 — Mon, 30 Mar 2015 13:27:22 GMT

Hello, splunkers
Have you tryed method described above when you recieve data from forwarder?
with local file it works but in case of forwarder and indexer does not.
I put transforms.conf and props.conf on indexer may be I do something wrong

Re: How do I route data to specific index based on a field?

tkomatsubara_sp — Mon, 30 Mar 2015 16:20:34 GMT

If you want to do somthing by using transforms.conf and props.conf, you need to use "Splunk Enterprise" as "Heavy Forwarder". It means, "just use Splunk Enterprise and make it send data to indexers".

Be carefull. "Splunk Forwarder" which is different binary from "Enterprise Splunk" can't do anything by using "props.con and transforms.conf".

Re: How do I route data to specific index based on a field?

andrey2007 — Tue, 31 Mar 2015 11:45:23 GMT

Thanks, really I use HF which parses my data so I can do nothing with them on indexer