Heavyforwarder transforms.conf split data into multiple indexes

[nareshinsvu]

Hello experts,

Need help. My requirement is to extract 1st set of lines into 1st index and 2nd set into 2nd index. And ignore all other lines from a log file.

Below is my configuration which is obviously failing.

I have seen other blogs' solution - successfully able to separate events into two indexes without using [discardAll] from transforms.conf and unspecified index in inputs.conf. But it will redirect all my ignored lines into main idx which I don't want.

inputs.conf
[monitor://D:\splunk_test\target.log]
disabled = false
sourcetype = Custom_S
index = target_index_one
interval = 10
crcSalt =

props.conf
[Custom_S]
TRANSFORMS-set = discardAll,index2one,index2two

transforms.conf
[discardAll]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[index2one]
REGEX=(First_Filter)
DEST_KEY=_MetaData:Index
FORMAT=target_index_one

[index2two]
REGEX=(Second_Variant)
DEST_KEY=_MetaData:Index
FORMAT=target_index_two

[FrankVl · Accepted solution]

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two

[nareshinsvu]

Just Awesome. I should have asked this before scratching my head for half a day and trying multiple options with transforms.conf and eventually failing.

Thanks Frank