realtime parse data from one index to another index

HEYan123 — Wed, 16 Jun 2021 15:03:57 GMT

Thanks first for your checking !

the case is :

we have one index named : indexA with the realtime data come from application event log, but the data is not clean enough (i know we can do some parse logic before the data get into the indexA, the situation is we can't control that part logic. what we received is only the indexA with the not well formatted data)

the requirement is:

1. we need create a new index named: indexB.

2. when there is new data come into indexA, a parse logic is processed to the new data (the logic will figure out new fields what we want) and feed the data from indexA into indexB.

It is more like take indexA as the datasource and forward data into indexB ...

as i know, we can write the search query and use the collect function to insert to data from indexA to indexB

however my concern is:

1. seems it is not realtime. - how to make it realtime (schedule ? or is there some steam logic will automatically do this ? not sure ..) ?

2. it only happened when we do the search with the full data - how to only get the incremental data (new coming data) ?

Here, need your suggestion - how to handle our requirement. thanks in advance

topic realtime parse data from one index to another index in Splunk Enterprise

realtime parse data from one index to another index