The LogicalDeviceId is the unique name for each device being indexed. I want to use that field as the basis for the main search.
host="Jetstream" and sourcetype="ObjectEvent" earliest=-30h@h [ subsearch here ] | fields LogicalDeviceId, EventTime
For reference, an ObjectEvent looks like: <Jetstream xmlns="http://Jetstream.TersoSolutions.com/v1.0/ObjectEvent"><Header EventId="02d42360-d0df-48f4-aa85-dc5cede9cc4a" EventTime="2011-05-16T05:53:31Z" LogicalDeviceId="10000092" ReceivedTime="2011-05-16T05:57:10Z" /><ObjectEvent><DeviceExtensionList /></ObjectEvent></Jetstream>
The resulting list would be those LogicalDeviceIds that have not sent in an ObjectEvent in the last 30 hours.
I'm seeing a couple problems. One is obviously speed on the subsearch. Any suggestions how to speed it up? I'm guessing there's a way to do unique on the LogicalDeviceId... The second is I'm not getting any results back.
A workable approach for this might be to build state in a lookup table. If you can periodically (and incrementally) update a lookup table with the last time a LogicalDeviceId checked in, then you can easily search that lookup table to figure out when a checkin was over 30 hours ago. Araitz covers this approach for firewall state in http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/