Problem using eventstats - not populating all fields

[dbryan]

I have the following search string (which I've obfuscated slightly):

sourcetype=NetworkImpression
| fields User_ID IP Advertiser_ID Buy_ID Ad_ID Creative_ID Site_ID Page_ID
| eval key = User_ID + IP + Advertiser_ID + Buy_ID + Ad_ID + Creative_ID + Site_ID + Page_ID
| eventstats count by key

If I run it over about fifteen minutes of my data (~45,000 events), it behaves as expected. I get a count field in 100% of them. See here:

However, anything much more and I start to see this:

That was over half an hour - ~90,000 events - I see this:

It continues to go down over a longer time frame.

Can anyone give me an indication of what's going on here? There are no errors that I can see in the internal index, and the search inspector doesn't show anything that stands out to me.

[itinney]

What happens if you add the following to the end of your search: | search NOT count="*"

[dbryan]

Now that I've done some more googling it looks like you might be right - there's a [stats] stanza documented for limits.conf which has a maxresultrows option. It's still specified as only applying to searches that create additional rows, but if it's applying to my eventstats command that'd make sense why I stop seeing values after ~50,000 rows, as that's the universal default for maxresultrows.

[dbryan]

I just checked the docs, and it looks like this option only affects search commands that grow the result set, so hopefully this isn't it.

[dbryan]

The thing is that the search still returns all my results - just many of them are not decorated with the data from the eventstats command.

[itinney]

I would run the search more frequently (so less than 50K results) and save the results to a summary index if this is something you need to report often.

[itinney]

There is a "configurable" limit of 50,000 search results returned when you Submit a search:

[searchresults]
maxresultrows = 50000

Although the README says that you should not increase this figure. Can you sample smaller time-ranges?

[dbryan]

I can see all the events missing the count field, corresponding to how many are missing in the search in my question. I don't understand though - I can't find any limits options for eventstats, and 100% of them have count.

[dbryan]

Yes, I meant to mention- it's always on 100% of events.

[Ayn]

Does the "key" field exist in all the events?