Find time of latest event type without streaming all data from indexer

[plaxos]

The time of the latest log record for a specific customer can be retrieved and used as the output from a subsearch with ...

..... [ search index=tlog | custid=1234 | head 1 | eval latest=_time | fields latest | format ] | ....

But depending on the volume of customer activity, this could result in up to xxx thousand records being streamed and then immediately dropped (by the head 1). Limiting the subsearch to a relative time period , e.g. earliest=-1d, is not possible.

Is there any way to have the indexer stream only one or a fixed number of records? i.e. the equivalent of doing head 1 on the indexer.

Using | metadata, it is easy to get the latest time for specific sourcetypes etc, but this does not help in this case.

[plaxos]

To elaborate ... I need to get the last 30 minutes of activity for a custid where the custid comes from an ad-hoc query through a dashboard. The activity for that custid may have been in the last few minutes or a few weeks ago. The total number of records for the customer could be in the hundreds or the tens of thousands. I am using this search with a subsearch to find the time of the last activity and put it in an "earliest" clause:

search index=tlog sourcetype=csrv custid=$custid$
   [ search index=tlog | custid=$custid$ | head 1 | eval earliest=relative_time(_time,"-30m") | fields earliest | format ]
| stats count by Env, Region

Perhaps there is a way to avoid doing two searches for this, I just can't think of it.

The reason that metadata doesn't help is that the "lastTime" field is for the index as a whole, and not by custid.

[Richfez]

Two options I can think off of the top of my head:

Perhaps you could build a simply summary index that would contain this (and possibly other) information? You can read up on this here, and/or watch a .conf 2013 session by Jesse Trucks on this sort of technique called "Automating Operational Intelligence: Stats and Summary Indexes". You'll have to search for his presentation in that page, I don't know how to link directly to it.

Another thought was to keep a lookup for this information, then just update it every few minutes. The docs for outputlookup will be a good starting place for this. This is probably slightly less work than the summary index, but less useful as well.

In both cases, you can find specific examples and help in Answers and Splunk blogs.

[dturnbull_splun]

Could you elaborate on your whole use case as it might help with suggesting approaches?

[MuS]

And tell us more why you cannot use metadata?