Splunk Search

Why is a join of a large dataset (350k records) and limits.conf configuration not working?

splunkIT
Splunk Employee
Splunk Employee

We are joining a large set of information using the join command and are only getting 50000 results. The indexes look like this:
"index=first" has 350k records each with an ID field
"index=second" has 125k records and a similar ID field

We are doing a search like this:

index=first | join ID [search index=second] 

and we only get 50000 records. we have already copied default/limits.conf to local/limits.conf and changed subsearch_maxout to 500000, and then restarted the server. The inner join still only returns 50k results.

Note: We are already investigating using 'stats' to join the data instead of 'join' but some of the data ends up as multi-value. so, ultimately, we want to use a join.

0 Karma

kieren_hamps_tl
Explorer

Thanks to a previous post you need to change two stanzas in the limits.conf.
[join] & [searchresults].

So for example I now have in $SPLUNK_HOME/etc/system/local/limits.conf:

[join]
subsearch_maxout = 500000

[searchresults]
maxresultrows = 500000

Note the above value is 500,000. I have added an extra 0 to the defaul

I believe this is because when you run the "join" in your query you are also using the "search" command so both parts are limiting you to the default of 50,000. i.e. joint FIELDNAME [search index=.....

somesoni2
Revered Legend

Try setting this as well

[searchresults]
* This stanza controls search results for a variety of Splunk search commands.

maxresultrows = <integer>
0 Karma

hexx
Splunk Employee
Splunk Employee

Could you please show us the output of the following command?

$SPLUNK_HOME/bin/splunk cmd btool limits list join --debug
0 Karma

kieren_hamps_tl
Explorer

I have the same issue. A join query returns only 50000 out of the 87000 that I expect. So using the answer above I set system/local/limits.conf and increased the maxout to 5M. Restarted splunk and still having the same issue.

Output of "splunk cmd btool limits list join" gives:
[join]
subsearch_maxout = 500000
subsearch_maxtime = 60
subsearch_timeout = 120

Any other ideas? Or is this a bug? (FYI. I am using Splunk Enterprise v6.3.0)

0 Karma

robertlynch2020
Motivator

Did you find a fix for this?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...