topic Re: There are 2 timestamp formats in a log file in Splunk Search

There are 2 timestamp formats in a log file

lish123 — Wed, 13 Jan 2021 09:43:29 GMT

<Jan 10, 2021 6:58:06 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Jan 10, 2021 6:58:06,538 PM CST> <Notice> <Log Management> <BEA-170019> <The server log file weblogic.logging.FileStreamHandler instance=1128635794

Re: There are 2 timestamp formats in a log file

askkawalkar — Wed, 13 Jan 2021 11:15:29 GMT

What is your requirement ? are you looking for ingesting these 2 events in splunk with 2 time format ?
or you are looking to make constant time format for an all event when ingested to splunk .

I can try to provide solution once requirement are specified.

Meanwhile for both type of question you will get answer in example of below documentation.

Re: There are 2 timestamp formats in a log file

General_Talos — Wed, 13 Jan 2021 11:22:53 GMT

Check in props.conf and adjust timezone settings with "TZ" for your sourcetype

Example

[your sourcetype] TZ = GMT

https://community.splunk.com/t5/Archive/How-to-adjust-the-time-zone-for-an-logs-coming-into-splunk/m-p/306839

Re: There are 2 timestamp formats in a log file

lish123 — Thu, 14 Jan 2021 01:40:08 GMT

It’s not a time zone issue. One event contains milliseconds and the other does not contain milliseconds.

Re: There are 2 timestamp formats in a log file

lish123 — Thu, 14 Jan 2021 01:43:19 GMT

I want to extract these 2 events in a timestamp format.

Re: There are 2 timestamp formats in a log file

askkawalkar — Thu, 14 Jan 2021 06:17:34 GMT

Hi @lish123 ,

Please place below parameters in props.conf, replace SOURCETYPENAME with actual sourcetype. If you are aware about regex, you can define your own regex

[SOURCETYPENAME] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TIME_PREFIX = \<(?<Time>\w{3} \d{1,2}, \d{4} \d{1,2}:\d{2}:\d{2}[,\d{3}]* PM CST)\> TIME_FORMAT = %b %d, %Y %I:%M:%S,%Q %p %Z

Regards.

Re: There are 2 timestamp formats in a log file

lish123 — Thu, 14 Jan 2021 06:57:55 GMT

Thanks a lot @askkawalkar

However, this TIME_FORMAT

%B%d,%Y%I:%M:%S,%Q%p%Z
Cannot extract this type of timestamp correctly

Re: There are 2 timestamp formats in a log file

askkawalkar — Thu, 14 Jan 2021 07:07:49 GMT

Hi @lish123 ,

I have developed solution based on timestamp in sample data you provided.

Is there any more types of timeformat in the log ?

Regards.

Re: There are 2 timestamp formats in a log file

to4kawa — Thu, 14 Jan 2021 08:50:37 GMT

index=_internal | head 1 | fields _raw _time | eval _raw="<Jan 10, 2021 6:58:06 PM CST> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.> <Jan 10, 2021 6:58:06,538 PM CST> <Notice> <Log Management> <BEA-170019> <The server log file weblogic.logging.FileStreamHandler instance=1128635794 <CST 6:58:06 PM, January 10, 2021> <Notice> <Log Management> <BEA-170019> <The server log file weblogic.logging.FileStreamHandler instance=1128635794" | multikv noheader=t | fields - Col* | rex "<(?P<time>.*?)>" | eval _time=case(match(time,",\d{3}\s*[AP]M"),strptime(replace(time,"CST","-0600"),"%b %d, %Y %T,%3N %p %:z"), match(time,"^[A-Z][a-z]+\b"),strptime(replace(time,"CST","-0600"),"%b %d, %Y %T %p %:z"), match(time,"^CST+\b"),strptime(replace(time,"CST","-0600"),"%:z %T %p, %B %d, %Y"))

How about INGEST_EVAL?

https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/IngestEval

Re: There are 2 timestamp formats in a log file

lish123 — Thu, 14 Jan 2021 09:42:24 GMT

Thanks a lot. @to4kawa

I try to run your spl statement, and the result is only 3 events, but I have more than 3 events. Can I provide the configuration props.conf file to extract the timestamp when uploading new data?

Re: There are 2 timestamp formats in a log file

to4kawa — Thu, 14 Jan 2021 09:57:08 GMT

Set up transforms.conf as described in the reference.

First, extract the time field with REGEX and then set INGEST_EVAL.
For eval, you can use the same one as in SPL.