Hi,
I inherited a Splunk Enterprise environment.
It is composed of 10 machines, divided into development and production (the latter with 2 clustered indexes). One machine serves as the Monitoring Console.
I find an app (with several reports) present on both development and production. The report has a cron job 44 * * * * in production and 12 1/13 * * * in development and produces a KVStore lookup (with the exact same name as the report). Other reports in other apps make use of the lookup.
On the Monitoring Console Search>Scheduler Activity>Scheduler Activity:Instance "Aggregate Scheduled Search Runtime" chart I see that same report displaying >60 Runtime(seconds) in 1 minute bins.
How is that possible if the lookup (and not the report) is scheduled to run?
If I click on the 1-minute bar on the chart, the drill-down opens another chart with, among others, fields PID, PPID as well as Elapsed Time (e.g. 744617.8700 within 50 seconds! Are these seconds at all?).
Trying to understand where these values come from (and what is running the report), I only find similar results with this query:
index=_introspection 20664 14912
and this is an example of the results (edited):
{"datetime":"08-02-2023 11:24:37.275 +0200","log_level":"INFO","component":"PerProcess","data":{"pid":"14912","ppid":"20664","status":"W","t_count":"12","mem_used":"61.352","pct_memory":"0.53","page_faults":"0","pct_cpu":"0.00","normalized_pct_cpu":"0.00","read_mb":"0.000","written_mb":"0.109","fd_used":"28","elapsed":"754858.4800","process":"splunkd","process_type":"search","search_props":{"sid":"scheduler__nobody_Q0dJLXNlYXJjaGhlYWRzLWdscGktc2VhcmNoZXM__RMD53efdbadd3a98c46d_at_1690213440_46074","user":"splunk-system-user","app":"biz-searchheads-glpi-searches","label":"glpi_states_table_lookup","provenance":"scheduler","scan_count":"0","delta_scan_count":"0","role":"head","mode":"historical","type":"scheduled"}}}
I disabled the report in both development and production but the Monitoring Console chart above keeps showing the same results.
Can somebody help me understand what is going on? How to find out where the results on the Monitoring Console for that report come from? Is this from the lookup (and not the report)? Is there some hidden mechanism running the report even if it is disabled?
Thanks!