search records limitation of 800000 records

[kishen2018]

Hi All,

Facing one issue with splunk for an search query records getting limited to 800000.
The SLR001 total count is not getting displayed more than 800000 records but actually its index has more than 900000 records

I tried changing maxresultrows value in limits.conf but its not working.
Also tried append maxout command in query its not giving expected result.

Appreciate any help display more than 800000 records for SLR001.

Query Used:

(index=sumidx_slr006 search_stage=slr006) OR (index=sumidx_slr002 stage=transaction slr=slr002) OR (index=sumidx_slr003 slr=slr003 stage=transaction) OR (index=sumidx_slr004 search_name="sumidx_slr004") |append [search index="sumidx_slr001" search_name="sumidx_slr001" |dedup isoClearSysRef]
| eval SLR_name=case(index="sumidx_slr006","SLR006",search_name="sumidx_slr001_change2","SLR001",index="sumidx_slr002","SLR002",index="sumidx_slr003","SLR003",index="sumidx_slr004","SLR004")
| stats count(eval(SLR_status="Breached")) AS Breached,count(eval(SLR_status="Breached" OR SLR_status="Not Breached")) as Total by SLR_name

Output Below:

SLR_name Breached Total

SLR001 315 800000
SLR002 141 1378539
SLR003 1792 1349458
SLR004 17 231518
SLR006 13 220741

[logloganathan]

Please refer the answer. it will help you

https://answers.splunk.com/answers/54674/how-to-increase-the-maximum-number-of-concurrent-historical...

[kishen2017]

Any help on this query is highly appreciable !!!

[niketn]

@kishen2018 you will have to provide some more details about your query. For example what is the purpose of performing dedup on sumidx_slr001 index with field isoClearSysRef. Do all indexes 001, 002... 006 have same field/type of data? Why do other indexes do not have duplicates.

With your current search, if you do not need dedup for 001, I could say use multisearch instead of append as it is not impacted by sub-search limitation. But then you can have index slr001 in the base search itself. If the isoCLearSysRef is present in all indexes and you can do a dedup across all you can try the following. Also as a search performance you can move eval SLR_name after the stats by index and then use rename as your final pipe i.e.

(index=sumidx_slr006 search_stage=slr006) OR (index=sumidx_slr002 stage=transaction slr=slr002) OR (index=sumidx_slr003 slr=slr003 stage=transaction) OR (index=sumidx_slr004 search_name="sumidx_slr004")  (index="sumidx_slr001" search_name="sumidx_slr001") 
| dedup isoClearSysRef
| stats count(eval(SLR_status="Breached")) AS Breached,count(eval(SLR_status="Breached" OR SLR_status="Not Breached")) as Total by index
| eval index=case(index="sumidx_slr006","SLR006",search_name="sumidx_slr001_change2","SLR001",index="sumidx_slr002","SLR002",index="sumidx_slr003","SLR003",index="sumidx_slr004","SLR004") 
| rename index as SLR_Name

[kishen2017]

Hi @niketn,

Thanks for the update.
duplicate values are available with field isoClearSysRef and its only for sumidx_slr001, rest of indexes 002...006 don't have duplicates and also the field isoClearSysRef is available only in sumidx_slr001 and not with other indexes.

[niketn]

@kishen2017 may I know the reasons for duplicates in sumidx_slr01 and not others? Is the duplicate inserted only once or on regular basis? Why not in other indexes?

In any case can you try with union command instead of append with maxout option to test?