topic Re: Heavyforwarder transforms.conf split data into multiple indexes in Getting Data In

Heavyforwarder transforms.conf split data into multiple indexes

nareshinsvu — Tue, 29 Sep 2020 23:20:53 GMT

Hello experts,

Need help. My requirement is to extract 1st set of lines into 1st index and 2nd set into 2nd index. And ignore all other lines from a log file.

Below is my configuration which is obviously failing.

I have seen other blogs' solution - successfully able to separate events into two indexes without using [discardAll] from transforms.conf and unspecified index in inputs.conf. But it will redirect all my ignored lines into main idx which I don't want.

inputs.conf
[monitor://D:\splunk_test\target.log]
disabled = false
sourcetype = Custom_S
index = target_index_one
interval = 10
crcSalt =

props.conf
[Custom_S]
TRANSFORMS-set = discardAll,index2one,index2two

transforms.conf
[discardAll]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[index2one]
REGEX=(First_Filter)
DEST_KEY=_MetaData:Index
FORMAT=target_index_one

[index2two]
REGEX=(Second_Variant)
DEST_KEY=_MetaData:Index
FORMAT=target_index_two

Re: Heavyforwarder transforms.conf split data into multiple indexes

FrankVl — Mon, 25 Feb 2019 08:32:46 GMT

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two

Re: Heavyforwarder transforms.conf split data into multiple indexes

nareshinsvu — Mon, 25 Feb 2019 23:55:08 GMT

Just Awesome. I should have asked this before scratching my head for half a day and trying multiple options with transforms.conf and eventually failing.

Thanks Frank