topic Re: Regex Extractions in Splunk Search

How to extract the time from the following two events (taken from the same log) and build a proper sourcetype?

John85 — Tue, 08 Mar 2022 04:03:55 GMT

Hello,

This is my very first post here and I need some advice because I've been trying for a couple of hours to extract the time from the following two events (taken from the same log) and build a proper sourcetype, but I couldn't find a solution:

ABIT Stack Job [DBS: ABITNET] ABIT_Outbound[extern] (not exclusive, scheduler) (818209397) 08:59:07,602 *** Threads: 2
ExportScheduler [Node http://127.0.0.1:8080/abitnet]-Thread-18727 08:59:07,622 [fmI9CashFlowArch]Export fmI9CashFlowArch wird ausgeführt...

Using regex101 I've gotten

.*(?:[^ \n]* )*\s(?<time>\d{2}\:\d{2}\:\d{2}\,\d{3})

but when I try to define a sourcetype, the parsing breaks with "Failed to parse timestamp". The problem is most likely the fact that the timestamp is at a different position in the two events.

Do you have any ideas?

Thank you.

Re: Regex Extractions

ITWhisperer — Sun, 06 Mar 2022 23:16:03 GMT

Your regex appear to work - what have you set for your timestamp lookahead?

Re: Regex Extractions

yuanliu — Mon, 07 Mar 2022 03:37:12 GMT

If you use Splunk Web to define sourcetype, this is the prompt about timestamp format: "A string in strptime() format that helps Splunk recognize timestamps." The optional "Timestamp prefix" is not for extracting timestamp, but to confine the time format to match a specific location in case that format is repeated in the event or when the format is so loose it may incidentally match other parts of the event.

In your case, I see that you are trying to interpret strings like "08:59:07,602" into timestamp. The question then is, what do each of 08, 59, 07, and 602 mean? The most common interpretation, when written as "08:59:07,602" would be hour, minute, second, and millisecond; in Splunk time format, that's %H:%M:%S,%3N. Is this what you are looking for? (I do not see anything in the sample data that corresponds to a date. As a result, Splunk will interpret your date as 1970-01-01, the Unix epoch 0.)

Re: Regex Extractions

John85 — Mon, 07 Mar 2022 08:01:57 GMT

Hello,

Thank you for the reply. I don't use Splunk Web to define the sourcetype, I create it manually. I also define (when I write the props manually) the timestamp format (in this case %H:%M:%S,%3N as you mentioned) and also the prefix (which in this case I consider it to be \s ). I also use the LOOKAHEAD param but without any success. But given the fact that my timestamp is at two different points in the two events which can be found in the same log, Splunk isn't able to extract properly the time, I suspect because the regex I use to identify what is in front of the timestamp, is not OK and it does not help Splunk. If I don't enter the time format it's able to identify the time but when I input the format, the parsing breaks with the mentioned error.

Re: Regex Extractions

John85 — Mon, 07 Mar 2022 08:45:54 GMT

Hello,

If I use MAX_TIMESTAMP_LOOKAHEAD=98, SPlunk only identifies the date from the "ExportScheduler...." line, normally since I consider the \s as the TIME_PREFIX, the lookahead shoud only be 12 ( the lenght of the time from the event ) but this doesn't work.

The full sourcetype is:

[<SOURCETYPE NAME>]
CHARSET=UTF-8
LINE_BREAKER=([\r\n]+).*(?:[^ \n]* )*\s(?<time>\d{2}\:\d{2}\:\d{2}\,\d{3})
MAX_TIMESTAMP_LOOKAHEAD=98
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true
TIME_PREFIX=\s

Re: Regex Extractions

ITWhisperer — Mon, 07 Mar 2022 09:36:46 GMT

Try setting your TIME_PREFIX to something like this

TIME_PREFIX=(\d+\)|Thread\-\d+)\s

Re: Regex Extractions

John85 — Mon, 07 Mar 2022 09:46:01 GMT

Thank you, that solution worked, it didn't cross my mind to use a conditional statement in the prefix.