Alerting when a remote device shows zero bytes_seen at two different collectors for same remote device

[rholm01]

Several thousand remote hosts connecting to a collector at 2 geographically distanced datacenters.

One of the collectors should be getting bytes_seen > zero

If neither collector sees bytes_seen > zero, then the controller is assumed to be down and an alert needs to be raised.

Trying to table

RemoteDevice  DatacenterCollector BytesSeenWestCoast    DatacenterCollector    BytesSeenEastCoast                RemoteDeviceStatus

Device-1              Collector-WestCoast      50                              Collector-EastCoast                     0                              Up

Device-2              Collector-WestCoast      104                            Collector-EastCoast                     0                              Up

Device-3              Collector-WestCoast      0                                 Collector-EastCoast                     0                              Down

In this scenario, Device-3 should be marked down, and an alert generated.

Expecting to use a lookup table file for a list of all the remote devices Splunk needs to receive the events for from the datacenter collectors.

[sravankaripe]

try this 

-------- | table RemoteDevice  DatacenterCollector BytesSeenWestCoast    DatacenterCollector    BytesSeenEastCoast    | eval   RemoteDeviceStatus=if(BytesSeenWestCoast=0 AND BytesSeenEastCoast=0,"Down","Up") | table RemoteDevice  DatacenterCollector BytesSeenWestCoast    DatacenterCollector    BytesSeenEastCoast  RemoteDeviceStatus | where  RemoteDeviceStatus like "Down"

[burwell]

Hi. I don't know what your data looks like but it might be something like 

<search to get your logs...>

| stats latest(CollectorWestCoastCount) AS WestCoastCount,  latest(CollectorEastCoastCount) AS EastCoastCount by RemoteDevice
| search WestCoastCount=0 AND EastCoastCount=0

And then alert when you have count > 0

If you show your log samples we can refine from here

[rholm01]

To be less cryptic. . . 

d1xhop01(CollectorWestCoast) and d2xhop01 (CollectorEastCoast)

RemoteDevice is the store controller. In this case: "CC4001-(Store 4001 CC Controller Div 35)"

Here is a sample message seen from d2xhop01 (CollectorEastCoast) that was forwarded to Splunk.

{"eda_source":"d2xhop01","device":"CC4001-(Store 4001 CC Controller Div 35)","bytes_seen":52}

Show syntax highlighted

The challenge is to compare bytes_seen from both collectors at the same time for each RemoteDevice and alert if bytes_seen is zero from both collectors for the period defined.