savedsearch in parsing forever end ever

[sistemistiposta]

Hello,

  I would like to summary index some data from heaving searches.

The savedsearch is

[Summary - servizi BIND by Clients]
action.email.useNSSubject = 1
action.summary_index = 1
action.summary_index._name = summary_ldap
action.summary_index.instance = servizi
action.summary_index.type = client
alert.track = 0
cron_schedule = 0 2 * * *
description = BIND by clients on servizi
dispatch.earliest_time = -1d@d
dispatch.latest_time = @d
display.general.timeRangePicker.show = 0
enableSched = 1
schedule_window = 60
search = index=ldap sourcetype=ldap:syslog instance=servizi\
[search index=ldap sourcetype=ldap:syslog instance=servizi op_type=BIND\
| where dn!="cn=replication manager,cn=config"\
| fields conn host instance]\
| where in(op_type,"connection","closed","BIND")\
| transaction maxopenevents=-1 maxopentxn=-1 maxevents=-1 mvlist=t startswith=client_ip=* conn host instance\
| lookup dnslookup clientip as client_ip OUTPUT clienthost as client\
| eval client_ip=mvindex(client_ip,mvfind(client_ip, "^\d+")), client=if(isnull(client),client_ip,client), dn=lower(dn)\
| mvexpand dn\
| where dn!="null"\
| sichart count(dn) over dn by client

The subsearch returns million of results, so I have already increased maxout (and maxresultrows in [searchresults].

The problem is that this search never ends. In the "Job Manager" the search is always in parsing mode. 

This search is still running and is approximately 0% complete.

(SID: scheduler__nobody__DS389__RMD58ebfeb8123af6f21_at_1643763600_45904) 

The search log terminated with:

02-02-2022 02:02:24.087 INFO  SearchOperator:kv [182237 searchOrchestrator] - Extractor stats: name=dnexplicitanom, probes=225, total_time_ms=5, max_time_ms=1
02-02-2022 02:02:24.091 INFO  ISearchOperator [182237 searchOrchestrator] - 0x7f3315be3400 PREAD_HISTOGRAM: usec_1_8=6754 usec_8_64=7150 usec_64_512=76 usec_512_4096=25 usec_4096_32768=28 usec_32768_262144=1 usec_262144_INF=0 
02-02-2022 02:02:24.092 INFO  SearchStatusEnforcer [182237 searchOrchestrator] - SearchStatusEnforcer is already terminated

It's all INFO rows, no WARN, no ERROR. Looking at the log I see that the subsearch terminated well:

audit.log:02-02-2022 02:02:50.993 +0100 INFO AuditLogger - Audit:[timestamp=02-02-2022 02:02:50.993, user=splunk-system-user, action=search, info=completed, search_id='subsearch_scheduler__nobody__DS389__RMD58ebfeb8123af6f21_at_1643763600_45904_1643763603.1', has_error_warn=false, fully_completed_search=true, total_run_time=140.00, event_count=3337731, result_count=3337731, available_count=3337731, scan_count=3340592, drop_count=0, exec_time=1643763603, api_et=1643670000.000000000, api_lt=1643756400.000000000, api_index_et=N/A, api_index_lt=N/A, search_et=1643670000.000000000, search_lt=1643756400.000000000, is_realtime=0, savedsearch_name="", search_startup_time="740", is_prjob=false, acceleration_id="A3473993-F504-40E9-8902-DDB9B00D2B1B_DS389_nobody_d9f7ec6dda7fbf4b", app="DS389", provenance="N/A", mode="historical", is_proxied=false, searched_buckets=13, eliminated_buckets=0, considered_events=3341037, total_slices=139625, decompressed_slices=125689, duration.command.search.index=4522, invocations.command.search.index.bucketcache.hit=0, duration.command.search.index.bucketcache.hit=0, invocations.command.search.index.bucketcache.miss=0, duration.command.search.index.bucketcache.miss=0, invocations.command.search.index.bucketcache.error=0, duration.command.search.rawdata=40772, invocations.command.search.rawdata.bucketcache.hit=0, duration.command.search.rawdata.bucketcache.hit=0, invocations.command.search.rawdata.bucketcache.miss=0, duration.command.search.rawdata.bucketcache.miss=0, invocations.command.search.rawdata.bucketcache.error=0, sourcetype_count__ldap:syslog=3337731, roles='admin+power+splunk-system-role+user', search='search index=ldap sourcetype=ldap:syslog instance=servizi op_type=BIND | where dn!="cn=replication manager,cn=config" | fields conn host instance']

Height hours have been passed, and no others logs have been written.

The Job Manager still shows the process as "parsing". I see no errors, but in the system I see no process running about this scheduled search. So I suspect that it's not in progress.

I don't know how to debug this issue. Do you have any hints?

The Splunk Enterprise version is 8.2.4.

Thank you very much

Kind Regards

Marco