Getting Data In

What is the difference between DEST_KEY= _TCP_ROUTING and DEST_KEY = _MetaData:Index

manikanta66
Explorer

Please give me a practical explanation of DEST_KEY usage in transforms.conf

0 Karma
1 Solution

harsmarvania57
Ultra Champion

Hi @manikanta66,

_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.

Example: _TCP_ROUTING

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

outputs.conf

[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997

In above example splunk will find error word from test sourcetype raw data and it will send it to tcpout group errorGroup which will send data to 2 indexers in load balanced way.

Example: _MetaData:Index

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index

Let's assume that test sourcetype is sending data to ok_index index, now based on above example splunk will find raw data from test sourcetype with word error and it will write it to error_index index.

I hope this helps.

Thanks,
Harshil

View solution in original post

harsmarvania57
Ultra Champion

Hi @manikanta66,

_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.

Example: _TCP_ROUTING

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup

outputs.conf

[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997

In above example splunk will find error word from test sourcetype raw data and it will send it to tcpout group errorGroup which will send data to 2 indexers in load balanced way.

Example: _MetaData:Index

props.conf

[test]
TRANSFORMS-routing=errorRouting

transforms.conf

[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index

Let's assume that test sourcetype is sending data to ok_index index, now based on above example splunk will find raw data from test sourcetype with word error and it will write it to error_index index.

I hope this helps.

Thanks,
Harshil

manikanta66
Explorer

Hi @harsmarvania57,

Thanks for your response. I have a doubt in your answer.

as per your 2nd example, the UF contain inputs.conf like below?

  1. [monitor://var/www/testing.log]
  2. disabled = 0
  3. sourcetype = test
  4. index = ok_index

you said "test sourcetype with word error will write it to error_index"
for the FORMAT key value, you mention error_index

here my doubts are
1.which name I choose for a new index in the indexer
ok index OR error_index?
2.If I have two indexers, How HF will find index without specifying the target-group in the FORMAT?

0 Karma

harsmarvania57
Ultra Champion

Based on 2nd example both index will require on your Indexers (ok_index and error_index)

Let's say you have below events in your log files

30-10-2017GMT17:12:00 ERROR This is error
30-10-2017GMT17:12:50 INFO This is info

In this case 1st line(event) will got to error_index and 2nd line(event) will go to ok_index

You need to specify index name in FORMAT on HF in 2nd example, otherwise it will not work.

0 Karma

manikanta66
Explorer

Thank you @harsmarvania57

0 Karma
Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...