topic Issue with count -- How can I search a large data set without Splunk truncating the data? in Splunk Search

Issue with count -- How can I search a large data set without Splunk truncating the data?

closeset — Mon, 28 Aug 2017 20:53:14 GMT

Hi,

I would like query all data over the past year and then use "stats count by some fields" to calculate the counts.

However, the data is too large (at least a few millions) and Splunk truncates data when querying, so the number of counts is inaccurate.

Does anyone know a good way to fix it?

PS. I tried 'sistats' and set a report run every hour to query data from the previous year.
Ideally, I hope the report can collect data in a smaller time interval accurately, and the aggregate the result.
However, in each hour, the report query the whole previous data inaccurately and then added up all counts as the result.

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

cmerriman — Mon, 28 Aug 2017 21:17:01 GMT

can you provide the original query that ended up being truncated as well as what query you're using to try summary indexing? replace any sensitive information. This will help the community answer your question more accurately.

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

skoelpin — Mon, 28 Aug 2017 21:44:14 GMT

Are you referring to the number of rows getting truncated?

If so, I had a simialr problem a while back where it would truncate anything more than 50,000 rows and lead to inaccurate results. Luckily this is a simple fix to limits.conf

maxresultrows = <integer>
* Configures the maximum number of events are generated by search commands which 
grow the size of your result set (such as multikv) or that create events. Other search commands are explicitly 
controlled in specific stanzas below.
* This limit should not exceed 50000. Setting this limit higher than 50000 causes instability.
* Defaults to 50000.

http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/Limitsconf

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

closeset — Tue, 29 Sep 2020 15:33:07 GMT

Hi, Thanks for reminding me. The code is here:

The code to create summary indexing report:
sourcetype=my_source event_id =*
| sistats count by event_id field1 field2

The name of the report is "my_report_name."

The code to retrieve the result:
index=summary search_name="my_report_name"
|stats count by event_id field1 field2

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

closeset — Mon, 28 Aug 2017 22:30:44 GMT

Thank you skoelpin!
This is one possible solution for me. In this case, because increasing the limit might cause some instability, do you happen to know other possible methods?

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

skoelpin — Wed, 30 Aug 2017 13:19:52 GMT

Are the number of rows getting truncated after 50k? If so then this may be your only solution

I've increased the limit before and haven't seen any instability issues. I would contact support and get their opinion before trying this in production

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

jkat54 — Thu, 31 Aug 2017 01:10:30 GMT

The limit is there to protect your browser from locking up (amoung other reasons... or at least that's what I believe). When you load than much into memory things can get funny. "Unstable" even!

Re: Issue with count -- How can I search a large data set without Splunk truncating the data?

jkat54 — Thu, 31 Aug 2017 07:33:09 GMT

I think you have several options.

Create and accelerate a data model.
Create summary indexes (using searches that run every day, or more frequently (like every 5-15 minutes) and then use a backfill script).
Do both of the above, create an accelerated DM and summary indexes from the DM using the tstats command, etc.

Number one being the easiest approach. Number 2 being a faster approach. Number three being necessary if you need to correlate data from more than one really large data set.