Splunk Enterprise

How do I route data to specific index based on a field?

melonman
Motivator

Hi,

I would like to know how to route data to a specific index based on a value in a field.

I have a series of data that look like this:

2012/06/07 10:45:50 service=srvc1 server=node3 score=50 seq=55041
2012/06/07 10:45:50 service=srvc3 server=node1 score=17 seq=55042
2012/06/07 10:45:50 service=srvc2 server=node1 score=67 seq=55043
2012/06/07 10:45:50 service=srvc2 server=node4 score=43 seq=55044
2012/06/07 10:45:50 service=srvc3 server=node2 score=11 seq=55045
2012/06/07 10:45:50 service=srvc3 server=node2 score=60 seq=55046
2012/06/07 10:45:50 service=srvc1 server=node0 score=28 seq=55047
2012/06/07 10:45:50 service=srvc1 server=node0 score=4 seq=55048

Then, I want to route date to srvc1, srvc2 or srvc3 depending on the value in service field.
I found several answers that explains how to route data based on host or source(IP), but I could not find an answer for my questions.

I really appreciate any comment on this...

Thank you,

(JA) イベントの任意のフィールドの値に基づいて保存するIndexを変えるにはどうするのか。

Tags (2)
1 Solution

jeff
Contributor

You need to use a transforms as follows:

In props.conf

[host::(host1|host2|host3)]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

and/or

[some_sourcetype]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

In transforms.conf

[index_srvc1]
REGEX    = .*service=srvc1.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc1

[index_srvc2]
REGEX    = .*service=srvc2.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc2

[index_srvc3]
REGEX    = .*service=srvc3.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc3

In my case I needed to base on host from a centralized rsyslog server. YMMV... you may be able to use a replacement expression instead of a separate transform for each index, but I'll leave that as an exercise for you.

View solution in original post

melonman
Motivator

Hi jeff,

I could get it work with the following config.

-- props.conf

[sample1]
TRANSFORMS-index_routing = route_data_to_index_by_field_service

-- transforms.conf

[route_data_to_index_by_field_service]
REGEX = .*service=(.*?)[ ]
DEST_KEY = _MetaData:Index
FORMAT = $1

-- Result

$ ./splunk search 'index=* sourcetype=sample1 | head limit=10 | table index, service, server'
index service server
----- ------- ------
srvc2 srvc2   node1
srvc2 srvc2   node0
srvc3 srvc3   node1
srvc2 srvc2   node4
srvc3 srvc3   node0
srvc2 srvc2   node4
srvc2 srvc2   node0
srvc1 srvc1   node4
srvc2 srvc2   node1
srvc1 srvc1   node0
  • now I can move forward to configure RBAC thing... thanks!

andrey2007
Contributor

Hello, splunkers
Have you tryed method described above when you recieve data from forwarder?
with local file it works but in case of forwarder and indexer does not.
I put transforms.conf and props.conf on indexer may be I do something wrong

0 Karma

tkomatsubara_sp
Splunk Employee
Splunk Employee

If you want to do somthing by using transforms.conf and props.conf, you need to use "Splunk Enterprise" as "Heavy Forwarder". It means, "just use Splunk Enterprise and make it send data to indexers".

Be carefull. "Splunk Forwarder" which is different binary from "Enterprise Splunk" can't do anything by using "props.con and transforms.conf".

0 Karma

andrey2007
Contributor

Thanks, really I use HF which parses my data so I can do nothing with them on indexer

0 Karma

andrey2007
Contributor

Ir really works, thanks!

0 Karma

jeff
Contributor

cool... glad I could help. Thanks for posting your regex- may have a use for that later.

0 Karma

jeff
Contributor

You need to use a transforms as follows:

In props.conf

[host::(host1|host2|host3)]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

and/or

[some_sourcetype]
TRANSFORMS-index1            = index_srvc1
TRANSFORMS-index2            = index_srvc2
TRANSFORMS-index3            = index_srvc3

In transforms.conf

[index_srvc1]
REGEX    = .*service=srvc1.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc1

[index_srvc2]
REGEX    = .*service=srvc2.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc2

[index_srvc3]
REGEX    = .*service=srvc3.*
DEST_KEY = _MetaData:Index
FORMAT   = srvc3

In my case I needed to base on host from a centralized rsyslog server. YMMV... you may be able to use a replacement expression instead of a separate transform for each index, but I'll leave that as an exercise for you.

melonman
Motivator

Thanks, I could make my config work.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...