topic Using collect command: Why is some data missing? in Splunk Enterprise

Using collect command: Why is some data missing?

welcome — Thu, 17 Aug 2023 18:00:21 GMT

I am using collect command to transfer data data from one index to another index
The query is like index=A source=sourceA sourcetype:sourcetypeA host=hostA | collect index=B source=sourceA sourcetype:sourcetypeA host=hostA.
But some data is missing why

Re: Using collect command

bowesmana — Thu, 17 Aug 2023 04:21:58 GMT

What sort of data is missing?

Have you tried using output_format=hec for the collect command

https://docs.splunk.com/Documentation/Splunk/9.0.3/SearchReference/Collect#arg-options

Re: Using collect command

welcome — Thu, 17 Aug 2023 04:25:24 GMT

I am getting events properly but the total number of A events not matching the B events

Re: Using collect command

bowesmana — Thu, 17 Aug 2023 04:47:08 GMT

Can you find some information around the missing data - e.g. does it transfer a fixed number of events, but miss some others.

Are the time stamps the same and is the time window you are searching the same on both indexes?

Re: Using collect command

welcome — Thu, 17 Aug 2023 06:19:02 GMT

Can we have any other command to transfer one index data into another index

Re: Using collect command

bowesmana — Thu, 17 Aug 2023 08:24:37 GMT

Is it possible to re-ingest the data to the new index? Other than that, the collect command should do the trick.