topic TriggerTime eval with SolarWinds Index in Splunk Search

TriggerTime eval with SolarWinds Index

daviswk — Thu, 12 Dec 2024 18:49:21 GMT

I am creating a dashboard with Splunk to monitor offline assets in my environment with SolarWinds. I have the add-on and incorporate solarwinds:nodes and solarwinds:alerts into my query. I am running into an issue where I cant get the correct output for how long an asset has been down. In SolarWinds you can see Trigger time in the Alert Status Overview. This shows the exact date and time the node went down.

I cannot find a field from the raw data between both sourcetypes that will give me that output. I want to use eval to show how much time has passed since the trigger.

Does anyone know how to achieve this?

Re: TriggerTime eval with SolarWinds Index

PickleRick — Thu, 12 Dec 2024 19:32:06 GMT

I'm not sure if I understand your question properly. Are you asking how to find a timestamp which is not included in the data you have? Well, if it's not there you need to make sure it's exported from the source somehow. It's more a Solarwinds question than a Splunk one.

Re: TriggerTime eval with SolarWinds Index

dural_yyz — Thu, 12 Dec 2024 20:39:03 GMT

@PickleRick is correct if the data is not in the logs you can't eval from nothing.

That said one way we have combated down time in the past is calculating the duration since the last anything log entry from the host. This is never really 100% because it could be a transport issue but the asset is still alive, or any number of other things where the asset is alive but not 'logging'. However, any calculation of x>acceptable duration of the last log is always a good thing to know. Pair that up with a good CMDB record to prevent tracking decommissioned assets.

There are many alternatives to lack of quality data, but they each come with pros and cons to be accounted for.

Re: TriggerTime eval with SolarWinds Index

PickleRick — Thu, 12 Dec 2024 20:45:23 GMT

Of course, checking when there are "missing" events is one of possible ways of checking uptime. But that's a completely different problem.