How do I route data to specific index based on a field?

[melonman]

Hi,

I would like to know how to route data to a specific index based on a value in a field.

I have a series of data that look like this:

2012/06/07 10:45:50 service=srvc1 server=node3 score=50 seq=55041
2012/06/07 10:45:50 service=srvc3 server=node1 score=17 seq=55042
2012/06/07 10:45:50 service=srvc2 server=node1 score=67 seq=55043
2012/06/07 10:45:50 service=srvc2 server=node4 score=43 seq=55044
2012/06/07 10:45:50 service=srvc3 server=node2 score=11 seq=55045
2012/06/07 10:45:50 service=srvc3 server=node2 score=60 seq=55046
2012/06/07 10:45:50 service=srvc1 server=node0 score=28 seq=55047
2012/06/07 10:45:50 service=srvc1 server=node0 score=4 seq=55048

Then, I want to route date to srvc1, srvc2 or srvc3 depending on the value in service field.
I found several answers that explains how to route data based on host or source(IP), but I could not find an answer for my questions.

I really appreciate any comment on this...

Thank you,

(JA) イベントの任意のフィールドの値に基づいて保存するIndexを変えるにはどうするのか。

[melonman]

Hi jeff,

I could get it work with the following config.

-- props.conf

[sample1]
TRANSFORMS-index_routing = route_data_to_index_by_field_service

-- transforms.conf

[route_data_to_index_by_field_service]
REGEX = .*service=(.*?)[ ]
DEST_KEY = _MetaData:Index
FORMAT = $1

-- Result

$ ./splunk search 'index=* sourcetype=sample1 | head limit=10 | table index, service, server'
index service server
----- ------- ------
srvc2 srvc2   node1
srvc2 srvc2   node0
srvc3 srvc3   node1
srvc2 srvc2   node4
srvc3 srvc3   node0
srvc2 srvc2   node4
srvc2 srvc2   node0
srvc1 srvc1   node4
srvc2 srvc2   node1
srvc1 srvc1   node0

• now I can move forward to configure RBAC thing... thanks!

[andrey2007]

Hello, splunkers
Have you tryed method described above when you recieve data from forwarder?
with local file it works but in case of forwarder and indexer does not.
I put transforms.conf and props.conf on indexer may be I do something wrong

[tkomatsubara_sp]

If you want to do somthing by using transforms.conf and props.conf, you need to use "Splunk Enterprise" as "Heavy Forwarder". It means, "just use Splunk Enterprise and make it send data to indexers".

Be carefull. "Splunk Forwarder" which is different binary from "Enterprise Splunk" can't do anything by using "props.con and transforms.conf".

[andrey2007]

Thanks, really I use HF which parses my data so I can do nothing with them on indexer

[andrey2007]

Ir really works, thanks!

[jeff]

cool... glad I could help. Thanks for posting your regex- may have a use for that later.

[jeff · Accepted solution]

You need to use a transforms as follows:

In props.conf

[host::(host1|host2|host3)]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

and/or

[some_sourcetype]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

In transforms.conf

[index_srvc1]
REGEX    = .*service=srvc1.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc1

[index_srvc2]
REGEX    = .*service=srvc2.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc2

[index_srvc3]
REGEX    = .*service=srvc3.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc3

In my case I needed to base on host from a centralized rsyslog server. YMMV... you may be able to use a replacement expression instead of a separate transform for each index, but I'll leave that as an exercise for you.

[melonman]

Thanks, I could make my config work.