One of our teams is running Java script that uses REST API to fetch data from Splunk Cloud using the search.
They run it around 1am EST, once per day. Usually it runs successfully but sometimes, not clear due to what reason, the script fails due to the following error:
"Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk."
Here is the more detailed output from the script log:
2020-09-09 01:59:17,172 [INFO ] [mcsplunk] Connecting to splunk
2020-09-09 01:59:17,173 [INFO ] [mcsplunk] Setting up proxy to establish connection
2020-09-09 01:59:17,234 [INFO ] [mcsplunk] Successfully established connection
with Splunk on host "our_host" using User "our_user" in app conext "our_app_name"
2020-09-09 01:59:17,234 [INFO ] [mcsplunk] Starting to fetch data from Splunk
2020-09-09 01:59:29,740 [INFO ] [mcsplunk] Splunk query execution has completed.
Scanned:723492, Matched:723036,Results:1, Run Duration:11.146
09-09-2020 05:01:28.899 +0000 ERROR SearchMessages - orig_component="CursoredSearch"
sid="1234567.123jobid" peer_name="indexer01.ourdomain.com" message=[indexer01.ourdomain.com] Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk.
But when we attempt to run the same splunk search in Splunk Cloud GUI - it works fine, no errors.
Any idea what could be the cause of such behavior?
