topic Re: Why is summary index search returning duplicate and incorrect counts of data? in Splunk Search

Why is summary index search returning duplicate and incorrect counts of data?

smaran06 — Thu, 02 Feb 2017 00:29:24 GMT

Hi Team,

I am populating the data in summary index using the following Splunk search

index=data"  | sistats count as total by  appName,trueclient, httpstatus,request_uri

when, I do stats over this

index=summary_ |stats count as total by appName.

We are getting lot of difference in counts.

When, I run the search directly the app counts are very low, then on summary index it's very high. Why summary index data is returning wrong data? is it because, I added trueclient, httpstatus,request_uri in sistats?

Re: Why is summary index search returning duplicate and incorrect counts of data?

somesoni2 — Thu, 02 Feb 2017 05:36:54 GMT

When you use the si* command for summary index, you need to use the same aggregation command on the summary index data. Give this a try and see if the counts are matching.

index=summary | stats count as total by  appName,trueclient, httpstatus,request_uri | stats sum(total) as total by appName

compare with this.

index=data |stats count as total by appName

Re: Why is summary index search returning duplicate and incorrect counts of data?

smaran06 — Thu, 02 Feb 2017 23:18:21 GMT

Thanks, still the count is not matching, summary index is at very high when compare to data which is not is summary index

Re: Why is summary index search returning duplicate and incorrect counts of data?

briancronrath — Thu, 02 Feb 2017 23:22:36 GMT

When you search for just

index=summary_

How many different source values are you getting? Are you getting sources outside of the search you used to populate it? If so, limit down to just the name of the search as your source when you search and see if the numbers look better.

Re: Why is summary index search returning duplicate and incorrect counts of data?

graa1005 — Tue, 29 Sep 2020 18:11:20 GMT

I have exactly the same problem. Multiple entries in the summary index for the same data. Only one value for info_search_time So it looks like it is one search. only outputs on multiple indexers.
If i deleted the summary data and re runtje job to add the results to the summary index.I get double data only different as the previous summary data. So only a part of the data is double.

Re: Why is summary index search returning duplicate and incorrect counts of data?

graa1005 — Fri, 23 Feb 2018 14:10:42 GMT

Some data can exist multiple times. IN my case up to 5 times. (i have 5 indexers)

Re: Why is summary index search returning duplicate and incorrect counts of data?

ddrillic — Fri, 23 Feb 2018 14:55:57 GMT

We normally put a safe-guard to avoid duplicates. A left join in the spirit of - | join type=left <field> [search index=<summary index name>| eval matched="Y"]

Re: Why is summary index search returning duplicate and incorrect counts of data?

graa1005 — Fri, 23 Feb 2018 15:00:08 GMT

ddrillic my search does NOT generate duplicates. I execute the search ones a day to generate a summary of the records of the day before.

Re: Why is summary index search returning duplicate and incorrect counts of data?

ddrillic — Fri, 23 Feb 2018 15:03:21 GMT

Sorry ; -)