Getting Data In

Heavyforwarder transforms.conf split data into multiple indexes

Builder

Hello experts,

Need help. My requirement is to extract 1st set of lines into 1st index and 2nd set into 2nd index. And ignore all other lines from a log file.

Below is my configuration which is obviously failing.

I have seen other blogs' solution - successfully able to separate events into two indexes without using [discardAll] from transforms.conf and unspecified index in inputs.conf. But it will redirect all my ignored lines into main idx which I don't want.

inputs.conf
[monitor://D:\splunktest\target.log]
disabled = false
sourcetype = Custom
S
index = targetindexone
interval = 10
crcSalt =

props.conf
[Custom_S]
TRANSFORMS-set = discardAll,index2one,index2two

transforms.conf
[discardAll]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[index2one]
REGEX=(FirstFilter)
DEST
KEY=MetaData:Index
FORMAT=target
index_one

[index2two]
REGEX=(SecondVariant)
DEST
KEY=MetaData:Index
FORMAT=target
index_two

0 Karma
1 Solution

Ultra Champion

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two

View solution in original post

0 Karma

Ultra Champion

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two

View solution in original post

0 Karma

Builder

Just Awesome. I should have asked this before scratching my head for half a day and trying multiple options with transforms.conf and eventually failing.

Thanks Frank

0 Karma