Knowledge Management
Highlighted

Can the Collect command create duplicates in the target summary index if run multiple times over the same source data and time range?

Explorer

If it is run multiple times over the same source data and time range?

Highlighted

Re: Can the Collect command create duplicates in the target summary index if run multiple times over the same source data and time range?

Splunk Employee
Splunk Employee

Yes, it can create duplicates.

View solution in original post

Highlighted

Re: Can the Collect command create duplicates in the target summary index if run multiple times over the same source data and time range?

Motivator

This depends on your base search and your definition of duplicates.

If your base search creates or retains the _raw field then the events will be duplicated but the source will be set to some random stash file and the sourcetype will be set to stash and the internal field _indextime will be different too

If your base search does not contain a raw field the _raw will be created from the fields in your search a timestamp will be added by default aswell as a infosearch_time field which will be different for every execution

The documentation is quite good in my opinion:

http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchReference/Collect

Highlighted

Re: Can the Collect command create duplicates in the target summary index if run multiple times over the same source data and time range?

Explorer

My definition of duplicates would be two events in the summary index that represent the exact same data set from the source index. For example if I have my report which populates the summary index scheduled to run once a day, but then someone also goes and runs it manually on Tuesday, will the data in the summary index be doubled up for Tuesday as opposed to the Monday data when it only ran as scheduled (no additional manual run).

0 Karma
Highlighted

Re: Can the Collect command create duplicates in the target summary index if run multiple times over the same source data and time range?

Motivator

Yes the data will be doubled. You have the possibility to identify and mark the extra data as deleted but the collect command does not have any 'intelligence' to detect that situation.