How to find the latest event (message) with a given key (field)?

[asingla]

I have components which are sending UDP messages to splunk. The message format is

key1=value1|key2=value2|....

Few keys represents some statistics about that component e.g. status which is only being sent when the status changes and rest of the key/value are sent every few seconds. I want to search for the last value of status send by the component. To find that I have to find the latest message which has the status key. I am able to write a query using joins but I don't know how far I should go in time to look for this message. If I say 'All Time', This query will not be efficient as I have to show this status value on the dashboard.

The query looks like this -

sourcetype='x' statuskey 
| join type=inner _time 
    [ search sourcetype='x' statuskey 
      | stats max(_time) as _time | fields _time 
    ] 
| table _time, statusKey

Is there any other way to solve this problem in splunk?

Generically, I am looking for latest message with a given set of keys.

[asingla]

Got it. Need to mention the table fields name in my saved search

index="summary" | dedup compName | table _time, compName, endSeqId

[asingla]

I scheduled the below search to run every minute for time period -2m@m to -1m@m and save the result in summary index.
sourcetype="x" | dedup compName

But when I run query on the summary index, it return nothing.

index="summary" | dedup compName

or

index="summary" compName='comp1'

I see that fields are not recognized in the events. Is there any way to extract the fields from events to these summarized events?

[dwaddle]

The technique in araitz's blog post demonstrates exactly that - keeping a lookup table with the consolidated state. It has some moving parts, but is not overly complex and is certainly possible.

[asingla]

I agree, but is there anyway I can keep updating the state of my compName when a message arrives. The one way I am thinking is to write a search and save the result. This search will consolidate the result based on the previous run and all the events which happened after that. And then another schedule search to delete all the consolidated data but the latest. My dashboard will always query the most recent consolidated state. To me this solution is looking complex if at all possible.

[gkanapathy]

Well, if you have no way to know what compNames exist (or at least how many there could be) it's impossible to do this query definitively except by searching "all time". Logically, If you don't know if you're done, then you have to keep going.

[asingla]

A series of most-recent values for several components. I have deleted my last post to avoid confusion. Messages from different components have same source/sourcetype but every message have fields (e.g. compName) which uniquely distinguish the component. Values of these compNames are not known in advance. I need to report the latest information(endSeqId in the below query) per component.

sourcetype="x" compName
| join type=inner _time compName
[
search sourcetype="x" compName
| stats max(_time) as _time by compName
| fields _time compName]
| table compName, endSeqId, _time

[dwaddle]

Just a general comment, you can always edit your original question to add more information to it as needed. Adding additional information in your own answer doesn't always make things the clearest.

But, about your question-in-general - are you expecting this search to return a single result value (the most recent for a single component) or a series of most-recent values for several components?

[gkanapathy]

Simplest and most efficient is simply this:

sourcetype=x statuskey | head 1

Run over "all time" and it will search till it finds the most recent event with the text "statuskey", return that one event, and stop. You can of course just limit the fields:

sourcetype=x statuskey | head 1 | fields _time, statuskey

You might do slightly better with:

 sourcetype=x statuskey statuskey=* | head 1 | fields _time, statuskey

depending on your data and field extractions.

[bhawkins1]

To be clear: the head of a query which isn't explicitly sorted will always be (a) the first n most recent events for fixed duration queries, or (b) the first n found events for live queries. The more interesting case is fixed duration queries. The results will be the most recent events by default because splunk returns events in time order, unless explicitly sorted. Please correct me if I'm wrong.

[dwaddle]

The simplest approach is to use the dedup search command. You can do something like:

sourcetype=x statuskey | dedup 1 statuskey sortby -_time

This effectively and efficiently filters down to only the most recent status. But, it requires that the "most recent status" is present within the search results. If something changes status once per 24 hours, then you'll need to search over 24 hours. Obviously, if you can't predict the status changes then your only option is 'All Time' which doesn't always end well performance-wise.

Splunk has advised in the past that the most search-efficient way of maintaining long-term state is with lookup tables. This is a pretty good application for that approach because you don't necessarily know how far back in time to search for a given event. The basic idea is that you use a combination of scheduled searches, inputlookup, and outputlookup to incrementally maintain an external lookup table that has the most-recent data. Then, in your dashboard, you use the data in the lookup for display purposes. Splunk's araitz provides a working example of how to do this in his blog post "Maintaining State of the Union"

[dwaddle]

The lookup-table approach is a mostly-reasonable workaround for when you don't know how many possible values you'll have. The lookup table maintains the long term state of "what is the last value I saw for this key (and when did I see it?)". Then you let your scheduled search update the lookup table occasionally by searching over a shorter interval than "all time".

[gkanapathy]

If you know how many statuskeys you need (see my comment on the question), then you can make this definitive with: sourcetype=x statuskey | dedup 1 statuskey | head 13, if there are 13 possible values. If the number is unknown, then (as said above), it's impossible to know when you've got them all without looking at all the data.