Splunk Search

How do I get around the issue of the Segmentation and Subsearch limit if I have 30000 results?

DanielFordWA
Contributor

Hi,

I have a index of raw usage data (iis) and a separate index of entitlement data (rest_ent_prod), both indexes have a unique identifier for each user "GUID".

I would like to be able to ad hoc search the raw usage index for user behavior of users with certain entitlements and also create summary indexes.

The issue I have is the number of unique users with certain entitlements is around 30k and subsearches max out at 10.5k.

Can anyone advise how I can get around this issue?

Thanks,

Dan

0 Karma
1 Solution

jeffland
Champion

Joining may be more comfortable, but you can always get the same mechanics going with a simple stats on a search comprising both sources, split by the field you would usually join on. I.e., instead of

index=iis | join GUID [search index=rest_ent_prod]

you would do

index=iis OR index=rest_ent_prod | stats values(something) by GUID

Check this cool post for more detail!

View solution in original post

woodcock
Esteemed Legend

The answer by @jeffland is absolutely the correct way but if you cannot make that work, and you can deal with using a 2-stage process to pump some of your data out to file or KV-store (using outputlookup), then you can use this trick to escape append/subsearch limits:

https://answers.splunk.com/answers/318428/how-can-i-escape-the-50k-subsearch-limit-while-lin.html

0 Karma

jeffland
Champion

Joining may be more comfortable, but you can always get the same mechanics going with a simple stats on a search comprising both sources, split by the field you would usually join on. I.e., instead of

index=iis | join GUID [search index=rest_ent_prod]

you would do

index=iis OR index=rest_ent_prod | stats values(something) by GUID

Check this cool post for more detail!

View solution in original post

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!