Getting Data In

Heavyforwarder transforms.conf split data into multiple indexes

nareshinsvu
Builder

Hello experts,

Need help. My requirement is to extract 1st set of lines into 1st index and 2nd set into 2nd index. And ignore all other lines from a log file.

Below is my configuration which is obviously failing.

I have seen other blogs' solution - successfully able to separate events into two indexes without using [discardAll] from transforms.conf and unspecified index in inputs.conf. But it will redirect all my ignored lines into main idx which I don't want.

inputs.conf
[monitor://D:\splunk_test\target.log]
disabled = false
sourcetype = Custom_S
index = target_index_one
interval = 10
crcSalt =

props.conf
[Custom_S]
TRANSFORMS-set = discardAll,index2one,index2two

transforms.conf
[discardAll]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[index2one]
REGEX=(First_Filter)
DEST_KEY=_MetaData:Index
FORMAT=target_index_one

[index2two]
REGEX=(Second_Variant)
DEST_KEY=_MetaData:Index
FORMAT=target_index_two

0 Karma
1 Solution

FrankVl
Ultra Champion

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two

View solution in original post

0 Karma

FrankVl
Ultra Champion

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two
0 Karma

nareshinsvu
Builder

Just Awesome. I should have asked this before scratching my head for half a day and trying multiple options with transforms.conf and eventually failing.

Thanks Frank

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...