Getting Data In

How to filter data from a single file and write to two different indexes?

payal23
Path Finder

I am trying to filter a set of data from a single file with the below conditions and send the filtered data to different indexes.

Events are like: [ file.txt]

<85>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

<25>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

So, event with 85 one should go to index A

and 25 one should go to index B

 

Labels (3)
0 Karma
1 Solution

thambisetty
SplunkTrust
SplunkTrust

create props.conf On HF or Indexer

Note: you should change [currentsourcetype] below with your sourcetype events are coming in.

[currentsourcetype]
TRANSFORMS-routetoindexes = firstindex, secondindex

transforms.conf on same HF or Indexers

[firstindex]
REGEX = ^\<85\> 
#to change index if event matches with the above regex
DEST_KEY = _MetaData:Index 
#indexA is index where matching data will be indexed
FORMAT = indexA

[secondindex]
REGEX = ^\<25\>
#to change index if event matches with the above regex
DEST_KEY = _MetaData:Index 
#indexA is index where matching data will be indexed
FORMAT = indexB

 

 

————————————
If this helps, give a like below.

View solution in original post

thambisetty
SplunkTrust
SplunkTrust

create props.conf On HF or Indexer

Note: you should change [currentsourcetype] below with your sourcetype events are coming in.

[currentsourcetype]
TRANSFORMS-routetoindexes = firstindex, secondindex

transforms.conf on same HF or Indexers

[firstindex]
REGEX = ^\<85\> 
#to change index if event matches with the above regex
DEST_KEY = _MetaData:Index 
#indexA is index where matching data will be indexed
FORMAT = indexA

[secondindex]
REGEX = ^\<25\>
#to change index if event matches with the above regex
DEST_KEY = _MetaData:Index 
#indexA is index where matching data will be indexed
FORMAT = indexB

 

 

————————————
If this helps, give a like below.

payal23
Path Finder

Thanks @thambisetty I will try this. And which index i should write in inputs.conf (Splunk UF integration) as Splunk will be monitoring the same file.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

@payal23 ,

you can give any index based on your requirement, if you give different index other than indexA and indexB used in transforms, you will be ending up with total 3 indexes for same input. hope this clears your doubt.

 

up vote if my solution works for you.

————————————
If this helps, give a like below.

payal23
Path Finder

Thanks for the explanation @thambisetty .. and its working!!! I added WRITE_META=true as well.

I have one more question: This configuration will work in both HF and Indexer, so which one should i use for this scenario?

Read this link and so I am confused: https://aditumpartners.com/5-splunk-myths-busted/

0 Karma

thambisetty
SplunkTrust
SplunkTrust

Data flow between universal forwarder and Indexer

universal forwarder -> Heavy forwarder(optional) -> Indexer

if you have heavy forwarder in place, I recommend using HF because the purpose of placing HF in between UF and Indexer is to parse/clean the data. Indexer is busy in serving requests coming from Search Head.

if you don't have HF in the flow , you can apply on Indexer.

Hope this helps...

up vote, if it solves your issue.

————————————
If this helps, give a like below.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...