topic How to filter data from a single file and write to two different indexes? in Getting Data In

How to filter data from a single file and write to two different indexes?

payal23 — Mon, 10 Aug 2020 14:36:31 GMT

I am trying to filter a set of data from a single file with the below conditions and send the filtered data to different indexes.

Events are like: [ file.txt]

<85>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

<25>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

So, event with 85 one should go to index A

and 25 one should go to index B

Re: How to filter data from a single file and write to two different indexes?

thambisetty — Mon, 10 Aug 2020 15:43:55 GMT

create props.conf On HF or Indexer

Note: you should change [currentsourcetype] below with your sourcetype events are coming in.

[currentsourcetype] TRANSFORMS-routetoindexes = firstindex, secondindex

transforms.conf on same HF or Indexers

[firstindex] REGEX = ^\<85\> #to change index if event matches with the above regex DEST_KEY = _MetaData:Index #indexA is index where matching data will be indexed FORMAT = indexA [secondindex] REGEX = ^\<25\> #to change index if event matches with the above regex DEST_KEY = _MetaData:Index #indexA is index where matching data will be indexed FORMAT = indexB

Re: How to filter data from a single file and write to two different indexes?

payal23 — Mon, 10 Aug 2020 15:34:45 GMT

Thanks @thambisetty I will try this. And which index i should write in inputs.conf (Splunk UF integration) as Splunk will be monitoring the same file.

Re: How to filter data from a single file and write to two different indexes?

thambisetty — Mon, 10 Aug 2020 15:46:19 GMT

@payal23 ,

you can give any index based on your requirement, if you give different index other than indexA and indexB used in transforms, you will be ending up with total 3 indexes for same input. hope this clears your doubt.

up vote if my solution works for you.

Re: How to filter data from a single file and write to two different indexes?

payal23 — Tue, 11 Aug 2020 03:43:45 GMT

Thanks for the explanation @thambisetty .. and its working!!! I added WRITE_META=true as well.

I have one more question: This configuration will work in both HF and Indexer, so which one should i use for this scenario?

Read this link and so I am confused: https://aditumpartners.com/5-splunk-myths-busted/

Re: How to filter data from a single file and write to two different indexes?

thambisetty — Tue, 11 Aug 2020 08:51:24 GMT

Data flow between universal forwarder and Indexer

universal forwarder -> Heavy forwarder(optional) -> Indexer

if you have heavy forwarder in place, I recommend using HF because the purpose of placing HF in between UF and Indexer is to parse/clean the data. Indexer is busy in serving requests coming from Search Head.

if you don't have HF in the flow , you can apply on Indexer.

Hope this helps...

up vote, if it solves your issue.