topic Re: Problem using eventstats - not populating all fields in Splunk Search

Problem using eventstats - not populating all fields

dbryan — Thu, 21 Jun 2012 09:43:04 GMT

I have the following search string (which I've obfuscated slightly):

sourcetype=NetworkImpression
| fields User_ID IP Advertiser_ID Buy_ID Ad_ID Creative_ID Site_ID Page_ID
| eval key = User_ID + IP + Advertiser_ID + Buy_ID + Ad_ID + Creative_ID + Site_ID + Page_ID
| eventstats count by key

If I run it over about fifteen minutes of my data (~45,000 events), it behaves as expected. I get a count field in 100% of them. See here:

However, anything much more and I start to see this:

That was over half an hour - ~90,000 events - I see this:

It continues to go down over a longer time frame.

Can anyone give me an indication of what's going on here? There are no errors that I can see in the internal index, and the search inspector doesn't show anything that stands out to me.

Re: Problem using eventstats - not populating all fields

Ayn — Thu, 21 Jun 2012 10:02:43 GMT

Does the "key" field exist in all the events?

Re: Problem using eventstats - not populating all fields

dbryan — Thu, 21 Jun 2012 10:22:42 GMT

Yes, I meant to mention- it's always on 100% of events.

Re: Problem using eventstats - not populating all fields

itinney — Thu, 21 Jun 2012 11:05:05 GMT

What happens if you add the following to the end of your search: | search NOT count="*"

Re: Problem using eventstats - not populating all fields

dbryan — Fri, 22 Jun 2012 01:36:37 GMT

I can see all the events missing the count field, corresponding to how many are missing in the search in my question. I don't understand though - I can't find any limits options for eventstats, and 100% of them have count.

Re: Problem using eventstats - not populating all fields

itinney — Fri, 22 Jun 2012 09:13:19 GMT

There is a "configurable" limit of 50,000 search results returned when you Submit a search:

[searchresults]
maxresultrows = 50000

Although the README says that you should not increase this figure. Can you sample smaller time-ranges?

Re: Problem using eventstats - not populating all fields

itinney — Fri, 22 Jun 2012 09:16:53 GMT

I would run the search more frequently (so less than 50K results) and save the results to a summary index if this is something you need to report often.

Re: Problem using eventstats - not populating all fields

dbryan — Sat, 23 Jun 2012 10:32:18 GMT

The thing is that the search still returns all my results - just many of them are not decorated with the data from the eventstats command.

Re: Problem using eventstats - not populating all fields

dbryan — Sun, 24 Jun 2012 23:35:50 GMT

I just checked the docs, and it looks like this option only affects search commands that grow the result set, so hopefully this isn't it.

Re: Problem using eventstats - not populating all fields

dbryan — Mon, 25 Jun 2012 03:15:59 GMT

Now that I've done some more googling it looks like you might be right - there's a [stats] stanza documented for limits.conf which has a maxresultrows option. It's still specified as only applying to searches that create additional rows, but if it's applying to my eventstats command that'd make sense why I stop seeing values after ~50,000 rows, as that's the universal default for maxresultrows.