Getting Data In

What is the difference between DEST_KEY= _TCP_ROUTING and DEST_KEY = _MetaData:Index

manikanta66
Explorer

Please give me a practical explanation of DEST_KEY usage in transforms.conf

0 Karma
1 Solution

harsmarvania57
Ultra Champion

Hi @manikanta66,

_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.

Example: _TCP_ROUTING

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

outputs.conf

[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997

In above example splunk will find error word from test sourcetype raw data and it will send it to tcpout group errorGroup which will send data to 2 indexers in load balanced way.

Example: _MetaData:Index

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index

Let's assume that test sourcetype is sending data to ok_index index, now based on above example splunk will find raw data from test sourcetype with word error and it will write it to error_index index.

I hope this helps.

Thanks,
Harshil

View solution in original post

harsmarvania57
Ultra Champion

Hi @manikanta66,

_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.

Example: _TCP_ROUTING

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

outputs.conf

[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997

In above example splunk will find error word from test sourcetype raw data and it will send it to tcpout group errorGroup which will send data to 2 indexers in load balanced way.

Example: _MetaData:Index

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index

Let's assume that test sourcetype is sending data to ok_index index, now based on above example splunk will find raw data from test sourcetype with word error and it will write it to error_index index.

I hope this helps.

Thanks,
Harshil

manikanta66
Explorer

Hi @harsmarvania57,

Thanks for your response. I have a doubt in your answer.

as per your 2nd example, the UF contain inputs.conf like below?

  1. [monitor://var/www/testing.log]
  2. disabled = 0
  3. sourcetype = test
  4. index = ok_index

you said "test sourcetype with word error will write it to error_index"
for the FORMAT key value, you mention error_index

here my doubts are
1.which name I choose for a new index in the indexer
ok index OR error_index?
2.If I have two indexers, How HF will find index without specifying the target-group in the FORMAT?

0 Karma

harsmarvania57
Ultra Champion

Based on 2nd example both index will require on your Indexers (ok_index and error_index)

Let's say you have below events in your log files

30-10-2017GMT17:12:00 ERROR This is error
30-10-2017GMT17:12:50 INFO This is info

In this case 1st line(event) will got to error_index and 2nd line(event) will go to ok_index

You need to specify index name in FORMAT on HF in 2nd example, otherwise it will not work.

0 Karma

manikanta66
Explorer

Thank you @harsmarvania57

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...