Knowledge Management

Can the Collect command create duplicates in the target summary index if run multiple times over the same source data and time range?

bill_chatfield
Explorer

If it is run multiple times over the same source data and time range?

1 Solution

sowings
Splunk Employee
Splunk Employee

Yes, it can create duplicates.

View solution in original post

chris
Motivator

This depends on your base search and your definition of duplicates.

If your base search creates or retains the _raw field then the events will be duplicated but the source will be set to some random stash file and the sourcetype will be set to stash and the internal field _indextime will be different too

If your base search does not contain a _raw field the _raw will be created from the fields in your search a timestamp will be added by default aswell as a info_search_time field which will be different for every execution

The documentation is quite good in my opinion:

http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchReference/Collect

bill_chatfield
Explorer

My definition of duplicates would be two events in the summary index that represent the exact same data set from the source index. For example if I have my report which populates the summary index scheduled to run once a day, but then someone also goes and runs it manually on Tuesday, will the data in the summary index be doubled up for Tuesday as opposed to the Monday data when it only ran as scheduled (no additional manual run).

0 Karma

chris
Motivator

Yes the data will be doubled. You have the possibility to identify and mark the extra data as deleted but the collect command does not have any 'intelligence' to detect that situation.

sowings
Splunk Employee
Splunk Employee

Yes, it can create duplicates.

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...