Hi.   I see rather strange (for me, newbie) issue with number of *nix devices. After the UF agent install devices reported data for couple of days, but showed status "unstable". A day later devices stopped updating in Splunk. On devices I found an error message. splunk.service - SYSV: Splunk indexer service    Loaded: loaded (/etc/rc.d/init.d/splunk; bad; vendor preset: disabled)    Active: inactive (dead)      Docs: man:systemd-sysv-generator(8) Warning: splunk.service changed on disk. Run 'systemctl daemon-reload' to reload -------------- I found that some people experienced similar issue and fixed with update of init.d script ------------- splunk_start() {   echo Starting Splunk...   ulimit -Hn 20240   ulimit -Sn 10240 ---------------- I implemented the proposed change and it did help for few days. Now I see devices being updated in Splunk on regular bases, but reported as "unstable" and no CPU/MEMORY/DISK data being reported. Thank you in advance  ... View more

Hi.   I am monitoring service status on number of paired servers. While service is running on server1 report on service stopped on server 2 is a false positive But if it stopped on server 1 and did not start on server 2 it's a case when I need to be alerted. Any example of alert logic you can share? Thank you ... View more

Hi. I have two panel dashboard. One is general status of cluster and another one details for selected cluster.  --------- Panel 1 -------- `myapp_get_index` sourcetype="myapp:pce:metadata" myapp_type="myapp:pce:health" |stats values(status) as status by fqdn |rename fqdn as FQDN | eval "Cluster Status" = upper(substr(status,0,1)) + lower(substr(status,2)) |fields - status ----------------------- Panel 2 -------------- `myapp_get_index` sourcetype="myapp:pce:metadata" myapp_type="myapp:pce:health" fqdn=$fqdn$ | head 1 |spath path="nodes{}" output=nodes | mvexpand nodes |table nodes |spath input=nodes |eval "Uptime Day"=round(uptime_seconds/60/60,0) |table hostname, ip_address, type, cpu.percent, disk{}.location, disk{}.value.percent,memory.percent,services.running{},services.status, "Uptime Day" ----- This code works. I added transpose command to panel 1 -------------- `myapp_get_index` sourcetype="myapp:pce:metadata" myapp_type="myapp:pce:health" |stats values(status) as status by fqdn |rename fqdn as FQDN |eval "Cluster Status" = upper(substr(status,0,1)) + lower(substr(status,2)) |fields - status |transpose 5 |fields - column | rename column as FQDN,"row 1" as "FQDN 1", "row 2" as "FQDN 2", "row 3" as "FQDN 3", "row 4" as "FQDN 4", "row 5" as "FQDN 5" ------------------ It shows data as I wanted in panel1, but in panel 2 it shows details only for first FQDN regardless what I click on. ----------------------- I could not pinpoint what is missing. Thank you in advance. ... View more

Hi. How I can compare load during the same time every day for business days? I.e. time 11:oo AM - 7:00 PM on Monday, Tuesday -----, Friday ----------------- Monday - 3200 Tuesday  - 3300 Wednesday - 5400 Thursday - 3200 Friday - 3100 -------------------- TIA ... View more