Need suggestions or advice to check status of same device on two systems

[eholz1]

Hello All,

I have a generic question on using splunk. I have two systems, system A, and system B.

If a device changes state on system A I want to check the state of the same device on system B. Currently I have log file data from system B being forwarder to the indexer. I have an application on system A, that can

send an http event to the indexer with UP or DOWN status of device on system A. I was thinking about creating a search when the hec message comes to the indexer, and searching  the log data on system B for the same device. If there is a match, send an alert, and if no match i.e system A idevice  shows DOWN, and system B show UP for the device, send an alert.

I know there are at least two ways to do this. But I am  open for suggestions.

Thanks,

ewholz

[PickleRick]

Your problem is a bit vague. Remember that Splunk deals with data represented as events. So when you're talking about some external "systems" we have no idea what describes those systems or their state within Splunk. Do you have some input actively monitoring those systems? (kinda like an active nagios/zabbix/whatever probe) Or do those systems simply send their state as some kind of a keepalive messages?

[bowesmana]

When you talk about 'systems' is that some field in an index representing that system? It's not clear what you're after, when you say 'if no match, i.e. System A shows DOWN and System B shows UP' - clearly that means there WAS a match.

What do you actually want to alert on given the potential states? 

System A - UP - System B - not found
System A - UP - System B - UP
System A - UP - System B - DOWN
System A - DOWN - System B - not found
System A - DOWN - System B - UP
System A - DOWN - System B - DOWN

and how does System B record it's state for a device. What about if the state of data in System B shows multiple different states for a device, e.g.

SYS B:2026-01-13 08:00:00 - Device X DOWN
SYS B:2026-01-13 08:01:30 - Device X UP
SYS A:2026-01-13 08:00:30 - Device X DOWN

and your search runs at 08:02 - what should occur?

You can't trigger a search "when the message comes", searches run on a schedule, i.e. at predefined times, so presumably you will need to take time into account, so in the above example when the search runs, A shows DOWN, but B shows a LATER time of UP.

[eholz1]

Hello,

Thank you for the replies to my post. By "system" I mean a virtual machine running Linux with each server running an application that monitory network switch and router status.  I have a universal forwarder running

on System B. I thought I could use the Splunk hec to send device status to Splunk, and check the system B log data on the indexer for a match, primarily the "down" state. 

Both virtual machines have the same application running, and monitor the same devices. Here is the base issue, these vms are in different locations and on different networks. If a switch show as "down" on system A, it does not mean the switch is really down, but sometimes the network path to the device is interrupted, so the idea is to check the same device from the other location over a different network path. If both systems indicate a "down" condition, then the device may reall be down.

I can put a forwarder on system A, and it might be better to compare that way.

Sorry for long-winded reply.

eholz1

[bowesmana]

If you can get data in Splunk that describes the states of the systems, you can write SPL to determine what is UP and what is down between your reporting tools.

However, whatever data you have, if you have one system that says UP and one system that says DOWN, how can you determine which of those is correct if the UP message comes in from system A 10 seconds before the DOWN state from system B and would you generate an alert to say it's down, or do you simply want to alert if both states are DOWN?

Anyway, if there is data in Splunk, you can make a decision.