Splunk Search

How do I get around the issue of the Segmentation and Subsearch limit if I have 30000 results?

DanielFordWA
Contributor

Hi,

I have a index of raw usage data (iis) and a separate index of entitlement data (rest_ent_prod), both indexes have a unique identifier for each user "GUID".

I would like to be able to ad hoc search the raw usage index for user behavior of users with certain entitlements and also create summary indexes.

The issue I have is the number of unique users with certain entitlements is around 30k and subsearches max out at 10.5k.

Can anyone advise how I can get around this issue?

Thanks,

Dan

0 Karma
1 Solution

jeffland
Champion

Joining may be more comfortable, but you can always get the same mechanics going with a simple stats on a search comprising both sources, split by the field you would usually join on. I.e., instead of

index=iis | join GUID [search index=rest_ent_prod]

you would do

index=iis OR index=rest_ent_prod | stats values(something) by GUID

Check this cool post for more detail!

View solution in original post

woodcock
Esteemed Legend

The answer by @jeffland is absolutely the correct way but if you cannot make that work, and you can deal with using a 2-stage process to pump some of your data out to file or KV-store (using outputlookup), then you can use this trick to escape append/subsearch limits:

https://answers.splunk.com/answers/318428/how-can-i-escape-the-50k-subsearch-limit-while-lin.html

0 Karma

jeffland
Champion

Joining may be more comfortable, but you can always get the same mechanics going with a simple stats on a search comprising both sources, split by the field you would usually join on. I.e., instead of

index=iis | join GUID [search index=rest_ent_prod]

you would do

index=iis OR index=rest_ent_prod | stats values(something) by GUID

Check this cool post for more detail!

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...