Monitoring Splunk

Why is disabled report showing results in Monitoring Console "Aggregate Scheduled Search Runtime" chart?

postit
New Member

Hi,

I inherited a Splunk Enterprise environment.

It is composed of 10 machines, divided into development and production (the latter with 2 clustered indexes). One machine serves as the Monitoring Console.

I find an app (with several reports) present on both development and production. The report has a cron job 44 * * * * in production and 12 1/13 * * * in development and produces a KVStore lookup (with the exact same name as the report). Other reports in other apps make use of the lookup.

On the Monitoring Console Search>Scheduler Activity>Scheduler Activity:Instance "Aggregate Scheduled Search Runtime" chart I see that same report displaying >60 Runtime(seconds) in 1 minute bins.

How is that possible if the lookup (and not the report) is scheduled to run?

If I click on the 1-minute bar on the chart, the drill-down opens another chart with, among others, fields PID, PPID as well as Elapsed Time (e.g. 744617.8700 within 50 seconds! Are these seconds at all?).

Trying to understand where these values come from (and what is running the report), I only find similar results with this query:

 

index=_introspection 20664 14912

 

and this is an example of the results (edited):

 

{"datetime":"08-02-2023 11:24:37.275 +0200","log_level":"INFO","component":"PerProcess","data":{"pid":"14912","ppid":"20664","status":"W","t_count":"12","mem_used":"61.352","pct_memory":"0.53","page_faults":"0","pct_cpu":"0.00","normalized_pct_cpu":"0.00","read_mb":"0.000","written_mb":"0.109","fd_used":"28","elapsed":"754858.4800","process":"splunkd","process_type":"search","search_props":{"sid":"scheduler__nobody_Q0dJLXNlYXJjaGhlYWRzLWdscGktc2VhcmNoZXM__RMD53efdbadd3a98c46d_at_1690213440_46074","user":"splunk-system-user","app":"biz-searchheads-glpi-searches","label":"glpi_states_table_lookup","provenance":"scheduler","scan_count":"0","delta_scan_count":"0","role":"head","mode":"historical","type":"scheduled"}}}

 

I disabled the report in both development and production but the Monitoring Console chart above keeps showing the same results.

Can somebody help me understand what is going on? How to find out where the results on the Monitoring Console for that report come from? Is this from the lookup (and not the report)? Is there some hidden mechanism running the report even if it is disabled?

Thanks!

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...