Getting Data In

What is the difference between DEST_KEY= _TCP_ROUTING and DEST_KEY = _MetaData:Index

manikanta66
Explorer

Please give me a practical explanation of DEST_KEY usage in transforms.conf

0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi @manikanta66,

_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.

Example: _TCP_ROUTING

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

outputs.conf

[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997

In above example splunk will find error word from test sourcetype raw data and it will send it to tcpout group errorGroup which will send data to 2 indexers in load balanced way.

Example: _MetaData:Index

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index

Let's assume that test sourcetype is sending data to ok_index index, now based on above example splunk will find raw data from test sourcetype with word error and it will write it to error_index index.

I hope this helps.

Thanks,
Harshil

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

Hi @manikanta66,

_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.

Example: _TCP_ROUTING

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

outputs.conf

[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997

In above example splunk will find error word from test sourcetype raw data and it will send it to tcpout group errorGroup which will send data to 2 indexers in load balanced way.

Example: _MetaData:Index

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index

Let's assume that test sourcetype is sending data to ok_index index, now based on above example splunk will find raw data from test sourcetype with word error and it will write it to error_index index.

I hope this helps.

Thanks,
Harshil

manikanta66
Explorer

Hi @harsmarvania57,

Thanks for your response. I have a doubt in your answer.

as per your 2nd example, the UF contain inputs.conf like below?

  1. [monitor://var/www/testing.log]
  2. disabled = 0
  3. sourcetype = test
  4. index = ok_index

you said "test sourcetype with word error will write it to error_index"
for the FORMAT key value, you mention error_index

here my doubts are
1.which name I choose for a new index in the indexer
ok index OR error_index?
2.If I have two indexers, How HF will find index without specifying the target-group in the FORMAT?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Based on 2nd example both index will require on your Indexers (ok_index and error_index)

Let's say you have below events in your log files

30-10-2017GMT17:12:00 ERROR This is error
30-10-2017GMT17:12:50 INFO This is info

In this case 1st line(event) will got to error_index and 2nd line(event) will go to ok_index

You need to specify index name in FORMAT on HF in 2nd example, otherwise it will not work.

0 Karma

manikanta66
Explorer

Thank you @harsmarvania57

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...