Please give me a practical explanation of DEST_KEY usage in transforms.conf
Hi @manikanta66,
_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.
Example: _TCP_ROUTING
props.conf
[test]
TRANSFORMS-routing=errorRouting
transforms.conf
[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup
outputs.conf
[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997
In above example splunk will find error
word from test
sourcetype raw data and it will send it to tcpout group errorGroup
which will send data to 2 indexers in load balanced way.
Example: _MetaData:Index
props.conf
[test]
TRANSFORMS-routing=errorRouting
transforms.conf
[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index
Let's assume that test
sourcetype is sending data to ok_index
index, now based on above example splunk will find raw data from test
sourcetype with word error
and it will write it to error_index
index.
I hope this helps.
Thanks,
Harshil
Hi @manikanta66,
_TCP_ROUTING routes set of data to set of servers which you have defined in outputs.conf with stanza starting from [tcpout:....]
_MetaData:Index will route set of data to different index.
Example: _TCP_ROUTING
props.conf
[test]
TRANSFORMS-routing=errorRouting
transforms.conf
[errorRouting]
REGEX=error
DEST_KEY=_TCP_ROUTING
FORMAT=errorGroup
outputs.conf
[tcpout:errorGroup]
server=10.10.0.1:9997, 10.10.0.2:9997
In above example splunk will find error
word from test
sourcetype raw data and it will send it to tcpout group errorGroup
which will send data to 2 indexers in load balanced way.
Example: _MetaData:Index
props.conf
[test]
TRANSFORMS-routing=errorRouting
transforms.conf
[errorRouting]
REGEX=error
DEST_KEY=_MetaData:Index
FORMAT= error_index
Let's assume that test
sourcetype is sending data to ok_index
index, now based on above example splunk will find raw data from test
sourcetype with word error
and it will write it to error_index
index.
I hope this helps.
Thanks,
Harshil
Hi @harsmarvania57,
Thanks for your response. I have a doubt in your answer.
you said "test sourcetype with word error will write it to error_index"
for the FORMAT key value, you mention error_index
here my doubts are
1.which name I choose for a new index in the indexer
ok index OR error_index?
2.If I have two indexers, How HF will find index without specifying the target-group in the FORMAT?
Based on 2nd example both index will require on your Indexers (ok_index and error_index)
Let's say you have below events in your log files
30-10-2017GMT17:12:00 ERROR This is error
30-10-2017GMT17:12:50 INFO This is info
In this case 1st line(event) will got to error_index and 2nd line(event) will go to ok_index
You need to specify index name in FORMAT
on HF in 2nd example, otherwise it will not work.
Thank you @harsmarvania57