Splunk Search

How to define a start time for a search based on log message

dowdag
Engager

Greetings,

Still confused with Splunk.

How do I specify start point to start searching from - for this application I do not start searching from the head of my Log files.
I would like to do something like:

pseudo code:

  index=* OR index=_* sourcetype=OneOfManyLogFiles* 
earliest = [ search index=_* sourcetype="terminal.log" | return  eval StartSearchFrom=strfTime(if(imatch(Info, "^ApplyPayment\(\) - ManagerId.+PaymentId"), _time),"%Y-%m-%d %H:%M:%S.%3N")   ]

This does idea not work.... how is this accomplished?

When using streamstats the log file that has the string "ApplyPayment" is not included in the final rendered table.

index=* OR index=_* sourcetype=OneOfManyLogFiles* 

| eval Action =case(match(Info, "^ApplyPayment\(\) - ManagerId.+PaymentId"),"StartTran" ,
    match(Info, "^Done Merchant Payment"),"EndTran", match(Info, "^Exiting"), "Exiting", 
    match(Info, ""), "Info", 1=1, Action) 
| eval EventTime=strfTime(if(in(Action, "StartTran", "EndTran"), _time,""),"%Y-%m-%d %H:%M:%S.%3N") 
| streamstats count(eval(match(Info, "^ApplyPayment\(\) - ManagerId.+PaymentId"))) AS TranCount BY host
| reverse 
| stats list(Info) As events BY host TranCount
| table TranCount sourcetype _time EventType action CheckNumber TransactionId PaymentId events

Thanks for any help/ideas on this.

0 Karma

memarshall63
Communicator

Well this works:

index=_internal sourcetype=scheduler earliest="07/01/2019:20:01:39"

So pulling the earliest time from a subsearch would be something like this example that I tested...

index=_internal sourcetype=scheduler 
    [| makeresults | eval start=relative_time(now(),"-60m") 
| eval earliest=strftime(start, "%m/%d/%Y:%H:%M:%S")
    | return  earliest]

I would think your search should work like this:

 index=* OR index=_* sourcetype=OneOfManyLogFiles*
  [ search index=_* sourcetype="terminal.log" 
  | eval StartSearchFrom=strfTime(if(imatch(Info, "^ApplyPayment\(\) - ManagerId.+PaymentId"), _time),"%m/%d/%Y:%H:%M:%S") 
  | rename StartSearchFrom AS earliest 
  | return earliest ]

Make sure you get the format string "%m/%d/%Y:%H:%M:%S" correct... and that StartSearchFrom holds a good timestamp, and you should be good.

Also watchout for that strfTime() in your search... that should be strftime() , It's probably just cut/paste issues, but that whole 'eval StartSearchFrom..." line could use some work.

Hope that helps...

0 Karma

dowdag
Engager

Marshall,
Thanks for your ideas. However this does not work.
Why does _time have the format of "%Y-%m-%d" and earliest is "%m/%d/%Y"

and if you add milliseconds to 'earliest' string splunk throws an error -- at least in the free version.

> not happy <

Will look at more examples and hope that i will find a solution.

0 Karma

memarshall63
Communicator

Like I said, that "eval StartSearchFrom clause needs work. I don't believe there's a function called "imatch", and your if() statement doesn't have a 2nd argument.

The search:
index=_internal sourcetype=scheduler earliest="07/01/2019:20:01:39"
works by using the earliest time modifier. The documentation says:

You can specify an exact time such as
earliest="10/5/2016:20:00:00", or a
relative time such as earliest=-h or
latest=@w6

So, that's the format for an exact time that the time modifier expects. If your time is not in that format, or has the seconds added to it, then it may not work.

So your success lies in getting your timestamp in that format and in the "earliest" field, before you execute a "| return earliest" from the subsearch.

This works, right?

index=_internal sourcetype=scheduler 
     [| makeresults | eval start=relative_time(now(),"-60m") 
 | eval earliest=strftime(start, "%m/%d/%Y:%H:%M:%S")
     | return  earliest]

you should just work to replace the stuff inside the strftime() function.

0 Karma

dowdag
Engager

I have yet to see this worked expected -- and yes imatch was a typo... have a good afternoon! 🙂

0 Karma

memarshall63
Communicator

What does your search look like and what results are you seeing? It's got to be pretty close.

0 Karma

adonio
Ultra Champion

can you share some sample data and desired output for search?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...