Getting Data In

How do I extract a timestamp from an event with bracket characters?

blaise
Explorer

Hello

I am trying to extract a timestamp from this type of events. Here, 04 is the day of month and 12 is the month, Dec
on the search head, these events currently appear as 12th April
[04/12/2018 10:16:04] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: esysprod_NOA_5_min_box
[04/12/2018 10:26:03] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: esysprod_EX900 MACHINE:

It looks pretty straightforward, but I cannot figure out what I am doing wrong.

The source type for these events is called : "autosys_events_prod"

So, I created a props.conf as below, and located it in the app that gets distributed from my deployment server:
I also verify on the server where the log is created that the props.conf file is updated, and I also restart Splunk on the Universal Forwarder.

[splunk@msplunkutil01 local]$ cat props.conf

[autosys_events_prod]
TIME_PREFIX = ^[
TIME_FORMAT = %d/%m/%Y %H:%M:%S
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19

I have tried different time prefix(es) without success.

How do I know if my props.conf is actually being used?

Everything I have tried seems to have no effect so far.

What is the best way to troubleshoot this ?

Thank you for your help in advance.

it is the first time I am trying to extract a timestamp from an event, so I might be doing something wrong.

Blaise

0 Karma
1 Solution

blaise
Explorer

Hello
I have finally resolved the issue, the problem was I have a distributed environment ...
so like Prakash suggested, the props.conf needs to be on the indexers, where the timestamp extraction is done.
I have completely removed the props.conf from the universal forwarder server, where I only left the inputs.conf to define the inputs.
Thank you for all your help
Blaise

View solution in original post

0 Karma

blaise
Explorer

Hello
I have finally resolved the issue, the problem was I have a distributed environment ...
so like Prakash suggested, the props.conf needs to be on the indexers, where the timestamp extraction is done.
I have completely removed the props.conf from the universal forwarder server, where I only left the inputs.conf to define the inputs.
Thank you for all your help
Blaise

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@blaise Please accept an answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

blaise
Explorer

Hi Rich and Prakash
I have tried both suggestions and it still is not working

thank you both for your replies, you both suggested to use :
TIME_FORMAT to %m/%d/%Y %H:%M:%S
but my raw events timestamps shows as : [05/12/2018 10:32:03] text text ...
where 05 is the day of the month %d
and 12 is the month %m
so the correct TIME FORMAT should be : %d/%m/%Y %H:%M:%S
please explain why you suggested otherwise, I am getting really confused ...

I am also wondering why all my attempts are failing, is it possible that another definition or config somewhere could take precedence over the app's props.conf ?
Thank you again
Blaise

0 Karma

prakash007
Builder

@blaise: I tested it on my local with your sample data, it's working for me, except you need to make changes to TIME_FORMAT based on your requirements...

 ##this configs should be on indexers(data parsing happens on indexers)
 props.conf
 [autosys_events_prod]
 SHOULD_LINEMERGE = false
 TIME_PREFIX = ^\[
 TIME_FORMAT = %d/%m/%Y %H:%M:%S
 MAX_TIMESTAMP_LOOKAHEAD = 19

 try running this command to check all the props in that particular app...

 ./splunk btool props list --debug 
 ./splunk btool props list --debug --app=search
0 Karma

blaise
Explorer

Hi Prakash,
thank you , you are correct and that was my mistake, the props.conf definition needs to be on the indexers.
Once I did that , it started working
Thank you heaps for your help, it is appreciated 🙂
Blaise

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@blaise, I originally recommended %d/%m/%Y %H:%M:%S, but you said it was wrong so I suggested %m/%d/%Y %H:%M:%S.

---
If this reply helps you, Karma would be appreciated.
0 Karma

prakash007
Builder

This should work, give it a try....

[autosys_events_prod]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
0 Karma

richgalloway
SplunkTrust
SplunkTrust

TIME_PREFIX is a regular expression, but yours is not a valid regex. ^[ starts a character set, but doesn't finish it. Try `^[', which treats the bracket as a literal character.

---
If this reply helps you, Karma would be appreciated.
0 Karma

blaise
Explorer

Hi Rich
I have tried your suggestion and it still is showing events for the 12th April, instead of the 4th Dec
I tried those 2:
TIME_PREFIX = '^['
TIME_PREFIX = '['

to confirm my props setttings on the universal forwarder, I found this great command:
[splunk@bautoprod01 local]$ splunk cmd btool --app=autosys props list
[autosys_events_prod]
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = false
TIME_FORMAT = %d/%m/%Y %H:%M:%S
TIME_PREFIX = '['
[splunk@bautoprod01 local]$ pwd
/opt/splunkforwarder/etc/apps/autosys/local
[splunk@bautoprod01 local]$

So the above confirms that the settings are applied ("distributed"), but yet it still is not working
Thank you for your help anyway
Blaise

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Change TIME_FORMAT to %m/%d/%Y %H:%M:%S.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...