About rdhdr

rdhdr · ‎09-13-2025

Hello experts, I have a dashboard in simple xml that shows single number charts which reflect, by host and application, whether OS tools are running on a particular set of servers. The hosts and applications are maintained in a lookup file and the metrics are contained in index em_metrics which is populated by Splunk_TA_Nix. The number charts represent how many servers associated with an application are currently NOT running that particular tool. Clicking on that number will drill down to a table which lists all of the servers and their status vis-a-vis that tool. The lookup file format is: host,application host01,app01 host02,app01 host03,app02 A sample from the em_metrics index (populated by Splunk_TA_Nix): | mstats count WHERE index="em_metrics" AND metric_name=ps_metric* host IN (host01) BY host, COMMAND span=15m "_time",host,COMMAND,count "2025-09-12T11:45:00.000-0400",host01,"(sd-pam)",32 "2025-09-12T11:45:00.000-0400",host01,NetworkManager,16 "2025-09-12T11:45:00.000-0400",host01,tool01,224 "2025-09-12T11:45:00.000-0400",host01,tool01,32 "2025-09-12T11:45:00.000-0400",host01,"[acpi_thermal_pm]",16 "2025-09-12T11:45:00.000-0400",host01,"[audit_prune_tre]",16 "2025-09-12T11:45:00.000-0400",host01,"[blkcg_punt_bio]",16 "2025-09-12T11:45:00.000-0400",host01,"[bnx2i_thread/0]",16 "2025-09-12T11:45:00.000-0400",host01,"[bnx2i_thread/10]",16 Here is a Splunk query which works, but which only considers the hosts which show up in the em_metrics index. In other words, if an application has 8 serrvers associated with it in the lookup file, but only 6 of them have metrics in the em_metrics index, and only 5 of those servers are running the tool, then the single number chart should show a 3, reflecting the 1 server not running the tool, as well as the 2 servers missing from the index (because they are not reporting any metrics at all). base query: | mstats count WHERE index="em_metrics" AND metric_name=ps_metric* host IN (*) BY host, COMMAND span=1h Query for single: | lookup hostlist host | where application="app01" | rename COMMAND AS process | eval expected_process_found=if(match(process,"(?i)tool01"),1,0) | stats max(expected_process_found) AS expected_process_found first(process) AS Process BY host | eval Process=if(expected_process_found=1, "tool01 Found Running", "tool01 not running") | search Process="tool01 not running" | stats count Query for table; | lookup hostlist host | where application="app01" | rename COMMAND AS process | eval expected_process_found=if(match(process,"(?i)tool01"),1,0) | stats max(expected_process_found) AS expected_process_found first(process) AS Process BY host | eval Process=if(expected_process_found=1, "tool01 Found Running", "tool01 not running") | table host Process expected_process_found application I have two changes I need to make. 1. How can I modify this to check for also include hosts in the lookup file that do not have metrics in the em_metrics index? 2. How can I convert the single chart into a timechart so that it can show the current count, but also a trendline underneath the number?

rdhdr · ‎09-24-2024

Wow, that was a lot simpler than the solutions I was trying to get working. Thank you!

rdhdr · ‎09-24-2024

Hello, I have the following dataset. It consists of configuration parameters from multiple systems. Each system has somewhere in the neighborhood of 3000-5000 parameters, some of which will not exist in all systems. I am trying to come up with a list of unique combinations of parameters with an Matching flag which shows whether the value is identical between both systems. It should indicate a false flag if the parameter exists in either system, but not the other, or if the parameter exists in both systems but with different values. The parameters are identified by a unique combination of SERVICE_NAME, FILE_NAME, SECTION and KEY (all four are required to be the same). And the system is identified by SID. The data look like this: SID SERVICE_NAME FILE_NAME SECTION KEY VALUE AAA index global.ini global timezone_dataset 123 AAA dpserver index.ini password policy minimal_password_length 16 AAA index index.ini flexible_table reclaim_interval 3600 AAA dpserver global.ini abstract_sql_plan max_count 1000000 BBB dpserver index.ini password policy minimal_password_length 16 BBB index index.ini password policy minimal_password_length 25 BBB dpserver global.ini abstract_sql_plan max_count 1000000 BBB index index.ini mergedog check_interval 60000 The data is in a dashboard, along with drop-downs to select two systems to be compared. One a user selects system AAA and system BBB, I would like the result to show: SERVICE_NAME FILE_NAME SECTION KEY Match index global.ini global timezone_dataset No dpserver index.ini password policy minimal_password_length Yes index index.ini flexible_table reclaim_interval No dpserver global.ini abstract_sql_plan max_count Yes index index.ini password policy minimal_password_length No index index.ini mergedog check_interval No I have tried many different SPL searches, but none have provided the intended result. I would greatly appreciate any assistance or guidance. Cheers, David

rdhdr · ‎06-14-2024

This requirement was solved with the following syntax: index = indxtst | table _time source EVENT_TYPE EVENT_SUBTYPE UID EVENT | eval diff=now()-_time | eval type=case(EVENT=="START","START",EVENT="END","END") | eventstats dc(type) as dc_type by UID | search dc_type=1 AND (type=START AND diff>300)

rdhdr · ‎06-14-2024

Hi, I guess the question I still need an answer to is, how can I apply a time restriction to the START event, but not the END event? Cheers, David

rdhdr · ‎06-12-2024

Thanks for the input, Giuseppe. I have not considered a max time between START and END events. I may need to think about that requirement. I notice that you put earliest==-15m AND latest==-5m at the start of the query. It seems to me that this would check whether both START and END events are > 5 minutes old, which would be subject to the same issue I have today, in which the alert fires between START and END events. What I think I need is to find a START event > 5 minutes old, with a corresponding END event of any age. Cheers, David

rdhdr · ‎06-12-2024

Hello, I have programs which write status events to Splunk. At the beginning they write EVENT=START and at the end, they write EVENT=END, both with a matching UID. I have created an alert which monitors for a START event without a corresponding END event, in order to find when a program may terminate abruptly. The alert is: index=indxtst | table _time source EVENT_TYPE EVENT_SUBTYPE UID EVENT | eval stat=case(EVENT=="START","START",EVENT=="END","END") | eventstats dc(stat) as dc_stat by UID | search dc_stat=1 AND stat=START This alert works fine, except sometimes it catches it while the program is running and simply hasn't written an END event yet. To fix this, I would like to add a delay, but that is not working. index=indxtst | table _time source EVENT_TYPE EVENT_SUBTYPE UID EVENT | eval stat=case(EVENT=="START","START",EVENT=="END","END") | eventstats dc(stat) as dc_stat by UID | search dc_stat=1 AND stat=START AND earliest==-15m AND latest==-5m This pulls back no records at all, even when appropriate testing data is created. What am I doing wrong?

Posts	7
Solutions	1
Karma Given	1
Karma Received	0
Member Since	‎06-12-2024

Online Status	Offline
Date Last Visited	‎09-15-2025 03:21 PM

mstats with Splunk_TA_Nix

How to Compare Values Based on Multiple Field Matc...

Match One Without Another, with Delay

mstats with Splunk_TA_Nix

Re: How to Compare Values Based on Multiple Field ...

How to Compare Values Based on Multiple Field Matc...

Re: Match One Without Another, with Delay

Re: Match One Without Another, with Delay

Re: Match One Without Another, with Delay

Match One Without Another, with Delay

Join the Conversation