Splunk Enterprise

savedsearch in parsing forever end ever

sistemistiposta
Path Finder

Hello,

  I would like to summary index some data from heaving searches.

The savedsearch is

 

 

[Summary - servizi BIND by Clients]
action.email.useNSSubject = 1
action.summary_index = 1
action.summary_index._name = summary_ldap
action.summary_index.instance = servizi
action.summary_index.type = client
alert.track = 0
cron_schedule = 0 2 * * *
description = BIND by clients on servizi
dispatch.earliest_time = -1d@d
dispatch.latest_time = @d
display.general.timeRangePicker.show = 0
enableSched = 1
schedule_window = 60
search = index=ldap sourcetype=ldap:syslog instance=servizi\
[search index=ldap sourcetype=ldap:syslog instance=servizi op_type=BIND\
| where dn!="cn=replication manager,cn=config"\
| fields conn host instance]\
| where in(op_type,"connection","closed","BIND")\
| transaction maxopenevents=-1 maxopentxn=-1 maxevents=-1 mvlist=t startswith=client_ip=* conn host instance\
| lookup dnslookup clientip as client_ip OUTPUT clienthost as client\
| eval client_ip=mvindex(client_ip,mvfind(client_ip, "^\d+")), client=if(isnull(client),client_ip,client), dn=lower(dn)\
| mvexpand dn\
| where dn!="null"\
| sichart count(dn) over dn by client

 

 

 

The subsearch returns million of results, so I have already increased maxout (and maxresultrows in [searchresults].

The problem is that this search never ends. In the "Job Manager" the search is always in parsing mode. 

This search is still running and is approximately 0% complete.

(SID: scheduler__nobody__DS389__RMD58ebfeb8123af6f21_at_1643763600_45904) 

The search log terminated with:

02-02-2022 02:02:24.087 INFO  SearchOperator:kv [182237 searchOrchestrator] - Extractor stats: name=dnexplicitanom, probes=225, total_time_ms=5, max_time_ms=1
02-02-2022 02:02:24.091 INFO  ISearchOperator [182237 searchOrchestrator] - 0x7f3315be3400 PREAD_HISTOGRAM: usec_1_8=6754 usec_8_64=7150 usec_64_512=76 usec_512_4096=25 usec_4096_32768=28 usec_32768_262144=1 usec_262144_INF=0 
02-02-2022 02:02:24.092 INFO  SearchStatusEnforcer [182237 searchOrchestrator] - SearchStatusEnforcer is already terminated

 

It's all INFO rows, no WARN, no ERROR. Looking at the log I see that the subsearch terminated well:

 

audit.log:02-02-2022 02:02:50.993 +0100 INFO AuditLogger - Audit:[timestamp=02-02-2022 02:02:50.993, user=splunk-system-user, action=search, info=completed, search_id='subsearch_scheduler__nobody__DS389__RMD58ebfeb8123af6f21_at_1643763600_45904_1643763603.1', has_error_warn=false, fully_completed_search=true, total_run_time=140.00, event_count=3337731, result_count=3337731, available_count=3337731, scan_count=3340592, drop_count=0, exec_time=1643763603, api_et=1643670000.000000000, api_lt=1643756400.000000000, api_index_et=N/A, api_index_lt=N/A, search_et=1643670000.000000000, search_lt=1643756400.000000000, is_realtime=0, savedsearch_name="", search_startup_time="740", is_prjob=false, acceleration_id="A3473993-F504-40E9-8902-DDB9B00D2B1B_DS389_nobody_d9f7ec6dda7fbf4b", app="DS389", provenance="N/A", mode="historical", is_proxied=false, searched_buckets=13, eliminated_buckets=0, considered_events=3341037, total_slices=139625, decompressed_slices=125689, duration.command.search.index=4522, invocations.command.search.index.bucketcache.hit=0, duration.command.search.index.bucketcache.hit=0, invocations.command.search.index.bucketcache.miss=0, duration.command.search.index.bucketcache.miss=0, invocations.command.search.index.bucketcache.error=0, duration.command.search.rawdata=40772, invocations.command.search.rawdata.bucketcache.hit=0, duration.command.search.rawdata.bucketcache.hit=0, invocations.command.search.rawdata.bucketcache.miss=0, duration.command.search.rawdata.bucketcache.miss=0, invocations.command.search.rawdata.bucketcache.error=0, sourcetype_count__ldap:syslog=3337731, roles='admin+power+splunk-system-role+user', search='search index=ldap sourcetype=ldap:syslog instance=servizi op_type=BIND | where dn!="cn=replication manager,cn=config" | fields conn host instance']

 

 

 

Height hours have been passed, and no others logs have been written.

The Job Manager still shows the process as "parsing". I see no errors, but in the system I see no process running about this scheduled search. So I suspect that it's not in progress.

I don't know how to debug this issue. Do you have any hints?

 

The Splunk Enterprise version is 8.2.4.

Thank you very much

Kind Regards

Marco

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...