Getting Data In
Highlighted

How do I extract a timestamp from an event with bracket characters?

Explorer

Hello

I am trying to extract a timestamp from this type of events. Here, 04 is the day of month and 12 is the month, Dec
on the search head, these events currently appear as 12th April
[04/12/2018 10:16:04] CAUAJMI40245 EVENT: CHANGESTATUS STATUS: SUCCESS JOB: esysprodNOA5minbox
[04/12/2018 10:26:03] CAUAJM
I40245 EVENT: CHANGESTATUS STATUS: SUCCESS JOB: esysprod_EX900 MACHINE:

It looks pretty straightforward, but I cannot figure out what I am doing wrong.

The source type for these events is called : "autosyseventsprod"

So, I created a props.conf as below, and located it in the app that gets distributed from my deployment server:
I also verify on the server where the log is created that the props.conf file is updated, and I also restart Splunk on the Universal Forwarder.

[splunk@msplunkutil01 local]$ cat props.conf

[autosys_events_prod]
TIME_PREFIX = ^[
TIME_FORMAT = %d/%m/%Y %H:%M:%S
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19

I have tried different time prefix(es) without success.

How do I know if my props.conf is actually being used?

Everything I have tried seems to have no effect so far.

What is the best way to troubleshoot this ?

Thank you for your help in advance.

it is the first time I am trying to extract a timestamp from an event, so I might be doing something wrong.

Blaise

0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

SplunkTrust
SplunkTrust

TIME_PREFIX is a regular expression, but yours is not a valid regex. ^[ starts a character set, but doesn't finish it. Try `^[', which treats the bracket as a literal character.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

Explorer

Hi Rich
I have tried your suggestion and it still is showing events for the 12th April, instead of the 4th Dec
I tried those 2:
TIMEPREFIX = '^['
TIME
PREFIX = '['

to confirm my props setttings on the universal forwarder, I found this great command:
[splunk@bautoprod01 local]$ splunk cmd btool --app=autosys props list
[autosyseventsprod]
MAXTIMESTAMPLOOKAHEAD = 19
SHOULDLINEMERGE = false
TIME
FORMAT = %d/%m/%Y %H:%M:%S
TIME_PREFIX = '['
[splunk@bautoprod01 local]$ pwd
/opt/splunkforwarder/etc/apps/autosys/local
[splunk@bautoprod01 local]$

So the above confirms that the settings are applied ("distributed"), but yet it still is not working
Thank you for your help anyway
Blaise

0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

SplunkTrust
SplunkTrust

Change TIME_FORMAT to %m/%d/%Y %H:%M:%S.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

Builder

This should work, give it a try....

[autosys_events_prod]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

Explorer

Hi Rich and Prakash
I have tried both suggestions and it still is not working

thank you both for your replies, you both suggested to use :
TIME_FORMAT to %m/%d/%Y %H:%M:%S
but my raw events timestamps shows as : [05/12/2018 10:32:03] text text ...
where 05 is the day of the month %d
and 12 is the month %m
so the correct TIME FORMAT should be : %d/%m/%Y %H:%M:%S
please explain why you suggested otherwise, I am getting really confused ...

I am also wondering why all my attempts are failing, is it possible that another definition or config somewhere could take precedence over the app's props.conf ?
Thank you again
Blaise

0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

SplunkTrust
SplunkTrust

@blaise, I originally recommended %d/%m/%Y %H:%M:%S, but you said it was wrong so I suggested %m/%d/%Y %H:%M:%S.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

Builder

@blaise: I tested it on my local with your sample data, it's working for me, except you need to make changes to TIME_FORMAT based on your requirements...

 ##this configs should be on indexers(data parsing happens on indexers)
 props.conf
 [autosys_events_prod]
 SHOULD_LINEMERGE = false
 TIME_PREFIX = ^\[
 TIME_FORMAT = %d/%m/%Y %H:%M:%S
 MAX_TIMESTAMP_LOOKAHEAD = 19

 try running this command to check all the props in that particular app...

 ./splunk btool props list --debug 
 ./splunk btool props list --debug --app=search
0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

Explorer

Hi Prakash,
thank you , you are correct and that was my mistake, the props.conf definition needs to be on the indexers.
Once I did that , it started working
Thank you heaps for your help, it is appreciated 🙂
Blaise

0 Karma
Highlighted

Re: How do I extract a timestamp from an event with bracket characters?

Explorer

Hello
I have finally resolved the issue, the problem was I have a distributed environment ...
so like Prakash suggested, the props.conf needs to be on the indexers, where the timestamp extraction is done.
I have completely removed the props.conf from the universal forwarder server, where I only left the inputs.conf to define the inputs.
Thank you for all your help
Blaise

View solution in original post

0 Karma