Getting Data In

Heavyforwarder transforms.conf split data into multiple indexes

nareshinsvu
Builder

Hello experts,

Need help. My requirement is to extract 1st set of lines into 1st index and 2nd set into 2nd index. And ignore all other lines from a log file.

Below is my configuration which is obviously failing.

I have seen other blogs' solution - successfully able to separate events into two indexes without using [discardAll] from transforms.conf and unspecified index in inputs.conf. But it will redirect all my ignored lines into main idx which I don't want.

inputs.conf
[monitor://D:\splunk_test\target.log]
disabled = false
sourcetype = Custom_S
index = target_index_one
interval = 10
crcSalt =

props.conf
[Custom_S]
TRANSFORMS-set = discardAll,index2one,index2two

transforms.conf
[discardAll]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[index2one]
REGEX=(First_Filter)
DEST_KEY=_MetaData:Index
FORMAT=target_index_one

[index2two]
REGEX=(Second_Variant)
DEST_KEY=_MetaData:Index
FORMAT=target_index_two

0 Karma
1 Solution

FrankVl
Ultra Champion

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two

View solution in original post

0 Karma

FrankVl
Ultra Champion

The problem is that the "discardAll" changes the queue for all messages, so all messages are dropped. Changing the index for the messages you want to keep doesn't change the queue back from null queue.

So you need 2 additional transforms (or combine the 2 regexes to do it in one):

[queue_one] 
REGEX=(First_Filter)
DEST_KEY=queue
FORMAT=indexQueue

[queue_two] 
REGEX=(Second_Variant)
DEST_KEY=queue
FORMAT=indexQueue

And then of course update your props.conf:

[Custom_S]
TRANSFORMS-set = discardAll,queue_one,queue_two,index2one,index2two
0 Karma

nareshinsvu
Builder

Just Awesome. I should have asked this before scratching my head for half a day and trying multiple options with transforms.conf and eventually failing.

Thanks Frank

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...