topic Re: How do I extract a timestamp from an event with bracket characters? in Getting Data In

How do I extract a timestamp from an event with bracket characters?

blaise — Tue, 29 Sep 2020 22:15:05 GMT

Hello

I am trying to extract a timestamp from this type of events. Here, 04 is the day of month and 12 is the month, Dec
on the search head, these events currently appear as 12th April
[04/12/2018 10:16:04] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: esysprod_NOA_5_min_box
[04/12/2018 10:26:03] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: esysprod_EX900 MACHINE:

It looks pretty straightforward, but I cannot figure out what I am doing wrong.

The source type for these events is called : "autosys_events_prod"

So, I created a props.conf as below, and located it in the app that gets distributed from my deployment server:
I also verify on the server where the log is created that the props.conf file is updated, and I also restart Splunk on the Universal Forwarder.

[splunk@msplunkutil01 local]$ cat props.conf

[autosys_events_prod]
TIME_PREFIX = ^[
TIME_FORMAT = %d/%m/%Y %H:%M:%S
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19

I have tried different time prefix(es) without success.

How do I know if my props.conf is actually being used?

Everything I have tried seems to have no effect so far.

What is the best way to troubleshoot this ?

Thank you for your help in advance.

it is the first time I am trying to extract a timestamp from an event, so I might be doing something wrong.

Blaise

Re: How do I extract a timestamp from an event with bracket characters?

richgalloway — Tue, 04 Dec 2018 03:59:12 GMT

TIME_PREFIX is a regular expression, but yours is not a valid regex. ^[ starts a character set, but doesn't finish it. Try `^[', which treats the bracket as a literal character.

Re: How do I extract a timestamp from an event with bracket characters?

blaise — Tue, 29 Sep 2020 22:15:52 GMT

Hi Rich
I have tried your suggestion and it still is showing events for the 12th April, instead of the 4th Dec
I tried those 2:
TIME_PREFIX = '^['
TIME_PREFIX = '['

to confirm my props setttings on the universal forwarder, I found this great command:
[splunk@bautoprod01 local]$ splunk cmd btool --app=autosys props list
[autosys_events_prod]
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = false
TIME_FORMAT = %d/%m/%Y %H:%M:%S
TIME_PREFIX = '['
[splunk@bautoprod01 local]$ pwd
/opt/splunkforwarder/etc/apps/autosys/local
[splunk@bautoprod01 local]$

So the above confirms that the settings are applied ("distributed"), but yet it still is not working
Thank you for your help anyway
Blaise

Re: How do I extract a timestamp from an event with bracket characters?

richgalloway — Tue, 04 Dec 2018 12:45:56 GMT

Change TIME_FORMAT to %m/%d/%Y %H:%M:%S.

Re: How do I extract a timestamp from an event with bracket characters?

prakash007 — Tue, 04 Dec 2018 19:44:38 GMT

This should work, give it a try....

[autosys_events_prod]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19

Re: How do I extract a timestamp from an event with bracket characters?

blaise — Tue, 04 Dec 2018 23:38:26 GMT

Hi Rich and Prakash
I have tried both suggestions and it still is not working

thank you both for your replies, you both suggested to use :
TIME_FORMAT to %m/%d/%Y %H:%M:%S
but my raw events timestamps shows as : [05/12/2018 10:32:03] text text ...
where 05 is the day of the month %d
and 12 is the month %m
so the correct TIME FORMAT should be : %d/%m/%Y %H:%M:%S
please explain why you suggested otherwise, I am getting really confused ...

I am also wondering why all my attempts are failing, is it possible that another definition or config somewhere could take precedence over the app's props.conf ?
Thank you again
Blaise

Re: How do I extract a timestamp from an event with bracket characters?

richgalloway — Wed, 05 Dec 2018 01:08:36 GMT

@blaise, I originally recommended %d/%m/%Y %H:%M:%S, but you said it was wrong so I suggested %m/%d/%Y %H:%M:%S.

Re: How do I extract a timestamp from an event with bracket characters?

prakash007 — Wed, 05 Dec 2018 03:57:59 GMT

@blaise: I tested it on my local with your sample data, it's working for me, except you need to make changes to TIME_FORMAT based on your requirements...

 ##this configs should be on indexers(data parsing happens on indexers)
 props.conf
 [autosys_events_prod]
 SHOULD_LINEMERGE = false
 TIME_PREFIX = ^\[
 TIME_FORMAT = %d/%m/%Y %H:%M:%S
 MAX_TIMESTAMP_LOOKAHEAD = 19

 try running this command to check all the props in that particular app...

 ./splunk btool props list --debug 
 ./splunk btool props list --debug --app=search

Re: How do I extract a timestamp from an event with bracket characters?

blaise — Wed, 05 Dec 2018 05:53:01 GMT

Hello
I have finally resolved the issue, the problem was I have a distributed environment ...
so like Prakash suggested, the props.conf needs to be on the indexers, where the timestamp extraction is done.
I have completely removed the props.conf from the universal forwarder server, where I only left the inputs.conf to define the inputs.
Thank you for all your help
Blaise

Re: How do I extract a timestamp from an event with bracket characters?

blaise — Wed, 05 Dec 2018 05:54:19 GMT

Hi Prakash,
thank you , you are correct and that was my mistake, the props.conf definition needs to be on the indexers.
Once I did that , it started working
Thank you heaps for your help, it is appreciated 🙂
Blaise

Re: How do I extract a timestamp from an event with bracket characters?

richgalloway — Wed, 05 Dec 2018 13:44:55 GMT

@blaise Please accept an answer to help future readers.