Monitoring Splunk

Why is disabled report showing results in Monitoring Console "Aggregate Scheduled Search Runtime" chart?

postit
New Member

Hi,

I inherited a Splunk Enterprise environment.

It is composed of 10 machines, divided into development and production (the latter with 2 clustered indexes). One machine serves as the Monitoring Console.

I find an app (with several reports) present on both development and production. The report has a cron job 44 * * * * in production and 12 1/13 * * * in development and produces a KVStore lookup (with the exact same name as the report). Other reports in other apps make use of the lookup.

On the Monitoring Console Search>Scheduler Activity>Scheduler Activity:Instance "Aggregate Scheduled Search Runtime" chart I see that same report displaying >60 Runtime(seconds) in 1 minute bins.

How is that possible if the lookup (and not the report) is scheduled to run?

If I click on the 1-minute bar on the chart, the drill-down opens another chart with, among others, fields PID, PPID as well as Elapsed Time (e.g. 744617.8700 within 50 seconds! Are these seconds at all?).

Trying to understand where these values come from (and what is running the report), I only find similar results with this query:

 

index=_introspection 20664 14912

 

and this is an example of the results (edited):

 

{"datetime":"08-02-2023 11:24:37.275 +0200","log_level":"INFO","component":"PerProcess","data":{"pid":"14912","ppid":"20664","status":"W","t_count":"12","mem_used":"61.352","pct_memory":"0.53","page_faults":"0","pct_cpu":"0.00","normalized_pct_cpu":"0.00","read_mb":"0.000","written_mb":"0.109","fd_used":"28","elapsed":"754858.4800","process":"splunkd","process_type":"search","search_props":{"sid":"scheduler__nobody_Q0dJLXNlYXJjaGhlYWRzLWdscGktc2VhcmNoZXM__RMD53efdbadd3a98c46d_at_1690213440_46074","user":"splunk-system-user","app":"biz-searchheads-glpi-searches","label":"glpi_states_table_lookup","provenance":"scheduler","scan_count":"0","delta_scan_count":"0","role":"head","mode":"historical","type":"scheduled"}}}

 

I disabled the report in both development and production but the Monitoring Console chart above keeps showing the same results.

Can somebody help me understand what is going on? How to find out where the results on the Monitoring Console for that report come from? Is this from the lookup (and not the report)? Is there some hidden mechanism running the report even if it is disabled?

Thanks!

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...