topic Index-time extraction of multiple timestamps fields within a single event in Getting Data In

Index-time extraction of multiple timestamps fields within a single event

robert_vincent — Thu, 08 Aug 2013 21:03:57 GMT

Need to set up searching and alerting for batch-job logging. Each log line will have the following format:

So I need to set up a search that starts with:

index="batch" | transaction fields=uuid

And then add logic that tests for:

linecount=1 | where ( timestamp + max-execution-time > current time )
Most recent "next-timestamp" for each appname is less than current time

Problem is, I don't know how to set up index-time field extraction for multiple timestamps within a single event. Help?

Re: Index-time extraction of multiple timestamps fields within a single event

linu1988 — Thu, 08 Aug 2013 21:43:51 GMT

Finally one timestamp will be recognized. Better to do leave it as it is, the first timestamp field will be recorded as _time field. Then you can go for the field extraction on the UI, which will be easier for you. it will add those entries to props.conf, then you can refer them to do the manual extraction entry yourself.

http://docs.splunk.com/Documentation/Splunk/5.0.4/admin/Propsconf

Re: Index-time extraction of multiple timestamps fields within a single event

lukejadamec — Fri, 09 Aug 2013 02:09:19 GMT

Is your data showing up as a file's worth of data in one event? Or, is each line showing up as a single event with it's own timestamp?

Re: Index-time extraction of multiple timestamps fields within a single event

kristian_kolb — Fri, 09 Aug 2013 08:16:30 GMT

You do not normally need to extract anything at index-time. You can make your comparisons with search-time extracted data.

Don't know what you really want to do, and what the transaction is used for, but if max-execution-time is in seconds, the logic/math will be rather simple. Current time (when the search starts) can be found via now().

...| eval XXX = _time + max-execution-time 
| eval YYY = if(XXX > now() AND next-timestamp < now(),"apple", "orange")

Perhaps you want to also look at the dedup command to let you only get the most recent event for some field.

See;

http://docs.splunk.com/Documentation/Splunk/5.0.4/SearchReference/CommonEvalFunctions

Some more explanation and a few sample events would let people here understand your problem better and be able to help you more.