Getting Data In

Why does the custom search command display only 1000 events?

jibanes
Path Finder

The following custom search command (which should return 100,000 displays) returns only 1000 events in Splunk. The rest of the events seems to be accounted for, but are not displayed; Splunk 6.x and 7.x:

import splunk.clilib.cli_common as spcli
import splunk.Intersplunk
import sys
import time

keywords, options = splunk.Intersplunk.getKeywordsAndOptions()

def main(args):
  results = []
  row = {}
  for i in range(0,100000):
    record = {}
    record['_time'] = time.time()
    record['_raw'] = "{" + str(i) + "}"
    results.append(record)
  splunk.Intersplunk.outputStreamResults(results)
  exit()
main(sys.argv)

commands.conf:

[test]
filename = test.py
local = true
overrides_timeorder = true
streaming = true
supports_multivalues = true
generating = stream

alt text

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

This is because by default it's limited at 1000 in code.

You can increase this value in limits.conf:
[search]
max_events_per_bucket = xxxx

Please refer to this answer:
https://answers.splunk.com/answers/92979/the-flashtimeline-dashboard-only-shows-first-1000-events.ht...

0 Karma

Lazarix
Communicator

look for limits.conf

Configure:

[searchresults]
maxresultrows = 100000
0 Karma

jibanes
Path Finder

No that doesn't seem to change the behavior, I added this in limits.conf, then restarted splunk.

/opt/splunk/bin/splunk btool limits list | grep -A10 searchresults
[searchresults]
compression_level = 1
max_mem_usage_mb = 200
maxresultrows = 100000
tocsv_maxretry = 5
tocsv_retryperiod_ms = 500
[set]
max_mem_usage_mb = 200
maxresultrows = 50000
[show_source]
distributed = true

Same behavior as previously reported though... only 10,000 results visible.

Note that the same behavior is observed with the default splunk command:

| streambag chunks=100

there are no events passed page #20. Exact same behavior.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...